Masters of Search Conference 2027
Masters of SearchPresented byRadyant
← All episodes
Masters of Search episode 49: ChatGPT Labrador Index, SERP Poisoning, Parametric Knowledge, Fake LLM Bots | #MonthlyLandwehr cover art
EP 49·Oct 28, 2026·2910

ChatGPT Labrador Index, SERP Poisoning, Parametric Knowledge, Fake LLM Bots | #MonthlyLandwehr

SpotifyApple PodcastsYouTube

Show notes

🎟️ Masters of Search Conference 2027: 12 October 2027, Berlin. Malte is one of the first confirmed speakers. Tickets: https://www.masters-of-search.com/conference

Every month I meet with Malte to take all the news, new research and interesting developments in SEO & AI Search to get his perspective on them. Malte Landwehr is CPO & CMO at Peec AI, one of the leading AI search analytics platforms.

This is the first episode after our summer break, recorded the day after the Masters of Search Conference 2026 in Berlin. We talk about Google poisoning the results it sends to scrapers, Graphite's research on parametric knowledge, ChatGPT's own search index Labrador, and why so many "LLM bot" visits in your log files are fake.

My biggest takeaways from this episode:

1) Google now feeds some scrapers wrong results, so rank tracking is getting less reliable. Instead of blocking scrapers, Google sends some of them wrong results. This SERP poisoning is much harder to detect than a block. Expect less data, less accurate data and less up-to-date data from SEO tools. Malte does not expect this to get fixed. (Reference: https://www.masters-of-search.com/the-brief/google-is-poisoning-8-of-rank-tracker-serps-nozzle-finds)

2) What a model already knows about your brand decides which brands show up in its fan-out queries. Ask for white sneakers for the office, and the fan-out queries may already name Adidas and Nike. That makes it even harder for new brands to get in. Graphite calls optimizing for this Parametric Knowledge Optimization. (Reference: https://graphite.io/five-percent/research/parametric-knowledge-optimization)

3) For most companies, getting into training data takes the same work as getting into grounding. Publish a lot on your own site and earn positive mentions on reputable sites. Unless you sell data or are a publisher, you want LLMs to train on your content.

4) ChatGPT has its own search index, called Labrador. It is a family of indexes, with separate ones for PDFs, YouTube, news, shopping and more. Don't block OpenAI's crawlers, don't hide content behind JavaScript, and put your most important content right after your H1. (Reference: https://peec.ai/blog/chatgpt-built-its-own-search-index)

5) ChatGPT shopping works differently in the US than in Europe. In the US, product results mostly come from Labrador and merchant feeds. In Europe, ChatGPT still scrapes Google Shopping. (Reference: https://peec.ai/blog/chatgpt-shopping-optimization-in-2026-us-vs.-europe)

6) Many "LLM bot" visits in your log files are fake. First indications from Peec AI: about half of visits that claim to come from LLM bots do not, and on some sites it is 70 to 90 percent. Verify bots by IP and reverse DNS before you trust the data.

Our daily AI Search news, The Brief: https://www.masters-of-search.com/the-brief

Let's connect!

Niklas on LinkedIn: https://www.linkedin.com/in/niklas-buschner/
Radyant on LinkedIn: https://www.linkedin.com/company/radyant/
Malte on LinkedIn: https://www.linkedin.com/in/landwehr/
Peec AI on LinkedIn: https://www.linkedin.com/company/peec-ai/

Full conversation

via podigee
  • 00:00:00Keyword tracking will never be as cheap and reliable.
  • 00:00:04And I also do not see this improving going forward, but I would definitely measure the success of my SEO campaigns with keyword tracking.
  • 00:00:12most people should focus on the web.
  • 00:00:15grounding process
  • 00:00:17is just yet another sign for people to question their legacy approaches and how they approach SEO.
  • 00:00:25For example, overly relying on ranked tracking certain keyword sets
  • 00:00:30I would agree.
  • 00:00:31Keywords are still a good proxy for demand, they can be a good proxies of topic or intent but if your strategy revolves around ranking specific keywords that was probably an outdated approach already and is not how i'd think about findability in the future.
  • 00:00:51Before we dive-in you're listening to the Masters Of Search podcast with me, your host Niklas Buschner.
  • 00:00:57Each week, I sit down with some of the smartest people around the world in SEO and AI search to bring you their strategies mental models.
  • 00:01:05And top pieces of actionable advice.
  • 00:01:08if You enjoy this podcast don't forget to like and subscribe.
  • 00:01:11then follow it In your favorite podcasting app or YouTube.
  • 00:01:14It helps us get top-notch guests and create the best possible content for you.
  • 00:01:19Let's dive into todays episode Welcome To The Masters Of Search Podcast and welcome to my favorite format of the show, The Monthly Landwehr.
  • 00:01:29Malte is already smiling!
  • 00:01:31We are back from our summer break.
  • 00:01:34that was a little longer but we took the time to recharge etc.
  • 00:01:42so now we're back with a lot fresh energy and lots of fresh insights.
  • 00:01:47honestly To kick it off slowly Malte how's your summer?
  • 00:01:52Busy
  • 00:01:54Worked
  • 00:01:55a lot, went to a lot of events and conferences traveled a lot.
  • 00:02:01And yeah now looking forward the end-of-the year that's actually not gonna slow down at all but will at least involve little bit less travel.
  • 00:02:09Yeah I feel like if Peec AI is not leading AI search visibility platform or also our platform to act on your AI Search Insights you are definitely like a LinkedIn influencer and a conference people group.
  • 00:02:29So I see so many Peec AI people being at conferences around the world, and people are active on LinkedIn... This is not hundred percent related to the search topic but how do you get so many people having exposure in conference stages?
  • 00:02:47Do you have secret sauce that can be shared with
  • 00:02:49them?
  • 00:02:50There's no really secret sauce!
  • 00:02:52I mean, it's one of our hypotheses that in the space of AI search there is a lot of assumptions and some snake oil.
  • 00:03:01And we believe doing research trying to identify what affect separating facts from fiction... ...and then sharing their knowledge is very important for industry overall.
  • 00:03:12That why built this geo-research team with David, Tomek and Metehan.
  • 00:03:19Of course, the three of them used to go to conferences before and now they are maybe more encouraged than previously.
  • 00:03:27We have so many insights on such a regular basis that there's just no other format but social media where we can share something.
  • 00:03:35There're weeks when you put out four or five insights.
  • 00:03:39then social is perfect channel for it?
  • 00:03:42Yeah!
  • 00:03:45monitoring tool for ourselves to stay on top of interesting stuff that like tools or software providers, like Peec AI do and also competitors.
  • 00:03:56And you are already flooding my feed!
  • 00:03:58Like there's so many people from Peec AI in there... So yeah but it is always very interesting.
  • 00:04:04We have a news board which we call the brief, like very brief snippets about AI search research and insights.
  • 00:04:20And there's also a lot of stuff that the Peec AI people are putting out compiled in one space.
  • 00:04:28so definitely check it out.
  • 00:04:30Speaking of conferences
  • 00:04:33actually
  • 00:04:34now were recording on Friday.
  • 00:04:36yesterday was spent whole day together.
  • 00:04:39how did you liked it Malte?
  • 00:04:42How did that come about?
  • 00:04:43That we spent the whole day together.
  • 00:04:45I mean, I guess you're referring to the Masters of Search Conference where we were yesterday.
  • 00:04:48Oh yes Um i really liked it definitely one of the best geo aeo ai search conferences i have attended this year.
  • 00:04:59um i Really liked all the talks.
  • 00:05:02i took so many notes also shared them on linkedin Yesterday.
  • 00:05:05if anybody wants to want to see what other things that i learned yesterday and also had good conversations between the sessions or all in all a good event.
  • 00:05:14And it was like directly next to where I'm living, which is very easy.
  • 00:05:17just have three minute walk through the venue.
  • 00:05:20Yeah that's always something you must keep in mind.
  • 00:05:24That's what Aleyda Solís told me.
  • 00:05:26You need make sure your venue is well reachable In all aspects.
  • 00:05:32i mean in Berlin technically basically no cool venues.
  • 00:05:36Well, reachable from the airport.
  • 00:05:38So people flying in always have quite a distance.
  • 00:05:41but yeah if you want someone like Malte on your stage make sure to be In five minutes walking distance to his home.
  • 00:05:50Yeah I'm also fine with with Airport business hotels If it's easier for the organizer.
  • 00:05:56Yeah i care more about The content than how cool the venue is.
  • 00:06:00But a cool venue Is definitely an advantage.
  • 00:06:03You're very humble person.
  • 00:06:06Then let's not talk too much about the conference of this year, because that is looking bad.
  • 00:06:11Let us look to future and we are happy to announce already coming back next year October twelfth again in Berlin at same location.
  • 00:06:24you can check it out.
  • 00:06:25link in description on the conference landing page.
  • 00:06:29what very cool.
  • 00:06:31Basically four speakers, but one of them is me.
  • 00:06:35So that might be boring.
  • 00:06:37We already have three very cool speakers confirmed and Malte as one of the.
  • 00:06:41so if anybody listening to this wants To make sure that you get like a full value packed session of Malte insights and Peec AI insights next year And planning ahead maybe even bigger companies Have a procurement team have to convince some manager to give You some budget for the ticket.
  • 00:07:00Be sure to check it out.
  • 00:07:02Cool, enough of advertising.
  • 00:07:04Let's talk a little bit about what happened in the space and I have some notes here.
  • 00:07:08And one thing that actually also you posted about but also couple other people is The topic of SERP poisoning.
  • 00:07:18Can You explain to People That Have Maybe Never Heard Of The Term Serp Poisoning What Is Going on?
  • 00:07:28Yeah, so basically when you are a search engine what you deliver is search results right SERPs and if you notice that somebody is scraping these in an automated fashion or Is using it for purposes?
  • 00:07:41That you do not like.
  • 00:07:42Like I don't know finding outdated PHP installations You have the option to block them And basically blocked identify their IP address block that and then they don't see it anymore And that saves a little bit of money because they basically can't scrape you anymore.
  • 00:08:00But, of course then scrapers start using different IP address and especially with IPv6 this is now trivial... ...and quite easy!
  • 00:08:10In the past Google was not THAT aggressive against scrapers Because they really didn't care that Searchmetrics, Sistrix, Semrush, Ahrefs, SE Ranking do a little SERP scraping.
  • 00:08:22but in recent years Google has got new competitors, the LLM Labs.
  • 00:08:29And they often scrape Google results for their web search and web grounding process.
  • 00:08:34So suddenly there's a much bigger business case for Google to block scraping.
  • 00:08:40They have done that for awhile already in last two years gotten much more aggressive with blocking IPs Also at lawsuit running against one of the large scraping providers That is used both by the Llm labs and all the SEO tools.
  • 00:08:55And in the first instance Google lost this case.
  • 00:08:59and then they started doing something called SERP poisoning, which means you do not block the scraper.
  • 00:09:05You just send them wrong results basically for a different keywords or like some of the results are like four different key words.
  • 00:09:13This is actually something that Bing has been doing I want to say ten years but definitely more than six seven years which is by the way, a reason why many, many scraping providers that offer Google either do not offer Bing.
  • 00:09:29Or if they offer Bing scraping it never works and the industry standard has actually been for awhile to scrape Yahoo as a proxy for Bing or other Bing based such engines that have less of a scrapping protection because If you get blocked It's very binary And easy!
  • 00:09:48If blocked Get new IP address Very Easy And new IP address can mean you disconnect your router, or it can just switch to a different proxy server.
  • 00:09:59But with SERP poisoning its much more difficult to detect and if Google is smart they will also now reduce the level of poisoning.
  • 00:10:09so instead of giving ten bad results that are all for different key words Or five maybe they start showing one wrong result or maybe they just start sorting the result list wrong, like basically showing the top two results on page one.
  • 00:10:25Like there are many things Google can do if they're smart to make this really difficult for scrapers and of course it's an issue for SEO tools.
  • 00:10:34Many have already reacted by saying no more top-one hundred rankings we were reducing into page one ranking.
  • 00:10:42others are claiming that A lot of them had very, very big fluctuations and it took all the tools a couple of days to actually release a statement.
  • 00:10:54Many of them also only did that once they were asked And then the first statement was yeah Yeah We have solved this It's no problem.
  • 00:11:00But people started posting screenshots like data going up down Updown which suggests maybe they haven't solved.
  • 00:11:08So there is definitely gonna be a challenge for SEO Tools And for users of SEO tools, I would just expect less data.
  • 00:11:17Less accurate data and less up-to-date data.
  • 00:11:20because some SEO tools have also said well technically you are paying for daily tracking but we can no longer guarantee that every keyword is tracked everyday.
  • 00:11:30if it's not then we're going to show the data from a previous day.
  • 00:11:34so every tool has like different ways to handle this also challenges with whatever approaches the various tools have picked.
  • 00:11:44So to be accurate on the implications for people that still rely on rank tracking, can you still trust it then if use one of the legacy SEO tools?
  • 00:11:56For Rank Tracking is this still reliable data or maybe a temporary thing where You cannot really trust the data, but then in the end they will probably figure it out.
  • 00:12:07So how do you look at that?
  • 00:12:08And what would you recommend people to do with their existing rank tracking projects?
  • 00:12:13because I think a lot of people still have this like fifty hundred maybe even bigger keyword sets that they track Because This is How You've Been Doing It and now The whole Like History Of It And The Interpretation Is Probably Completely Flawed.
  • 00:12:32I mean it completely depends on what you do with the data.
  • 00:12:35I believe many websites should look at Google search console, that data has other disadvantages but not poisoning.
  • 00:12:43At least i hope that google doesn't count the poisoning impressions if they use maybe your website to serve a wrong result and It really depends.
  • 00:12:57Keyword tracking will never be as cheap and reliable.
  • 00:13:01I also do not see this improving going forward.
  • 00:13:06It is a key mission for Google to make it hard, open my eye.
  • 00:13:12Also all the Chinese labs train on google search data To use Google for live web grounding And also to identify content that they can use for training.
  • 00:13:23And there have also been cases already, especially in the
  • 00:13:25U.S.,
  • 00:13:26where people suddenly got a pop-up asked to log in.
  • 00:13:29so it could be that Google will move more and more towards a logged inexperience.
  • 00:13:35I would still value in keyword tracking for many cases but you need think about what their data is how reliable they are.
  • 00:13:51But I also would not have done that five years ago.
  • 00:13:54So no big change for my approach to SEO, but yeah...I don't think this will be fixed.
  • 00:14:00there might be a time where it's slightly improved but Google has many many incentives to keep fighting and making it worse.
  • 00:14:08And if anybody as google has trouble with this send me a link in dm.
  • 00:14:12i have like many ideas how to make This even more tough on seo tools an even more annoying For people too.
  • 00:14:18try identify into and debug.
  • 00:14:21Okay, if anybody from Google is listening reach out to Malte happy to make an introduction.
  • 00:14:28No just kidding.
  • 00:14:30So to summarize Is this Just yet another?
  • 00:14:36Sign for people To question their legacy approaches and how they approach SEO For example overly relying on ranked tracking Certain keyword sets would you say This accurate summary.
  • 00:14:53Yeah, I would agree.
  • 00:14:54like keywords are still a good proxy for demand they can be a good proxie or an intent but if your strategy revolves around ranking specific key words that was probably outdated approach already and is not how i'd think about digital visibility findability in the future.
  • 00:15:20Okay, you already mentioned training data which is a very good bridge to the next thing that I'm curious about because our mutual friends at Graphite namely Ethan Gregory and Bevin they put out a paper i think it's yeah one week old um about parametric knowledge optimization.
  • 00:15:46Now, I think the industry was still in part struggling to agree on an acronym for what we call AI Search Optimization.
  • 00:15:55Yeah is it AIO, GEO?
  • 00:15:57And Graphite just figured why not bring a new kit to town?
  • 00:16:01so can you maybe give people?
  • 00:16:04um.
  • 00:16:05uh quick introduction of What this parametric knowledge optimization Is about that Graphite is talking About and their paper?
  • 00:16:14yeah sure Basically, when an LLM answers a question or replies to a prompt doesn't even have to be a question.
  • 00:16:22It can do two things it Can use just its model itself like foundational model?
  • 00:16:28Or it can use tools and the most common tool is probably web search.
  • 00:16:33technically there are other tools but let's not go into all the details in The the foundation model.
  • 00:16:40what in the end?
  • 00:16:41it is a large language model trained on data, but it doesn't have a database.
  • 00:16:48It does not have an index that cannot retrieve anything.
  • 00:16:51you just have the model weights or the parameters and in these is somehow stored some knowledge Not explicitly as like a database retrieval system But there's knowledge in it probably similar to how we hold knowledge.
  • 00:17:07now brain And for example of model doesn't need web search To answer your question What's the capital of the United States?
  • 00:17:15It will know it is Washington DC.
  • 00:17:18And this sometimes referred to as parametric knowledge, because not technically extracted from parameters.
  • 00:17:27but if the parameters are used and write an answer and always estimates next word ends up with giving that answer Conversation to a human.
  • 00:17:40This is like actually the same thing as somebody sharing their knowledge.
  • 00:17:44and for Brands, it is of course an advantage if the brand Is already in this knowledge?
  • 00:17:50And is associated with something.
  • 00:17:52so for example If I'm selling running shoes and i'm Nike and have branded myself on running shoes For many years and I've been In all the training data as a leading brand for running shoes Of course that has an advantage over Let's say Radyant starting to sell running shoes tomorrow because you could only get into the answer via the fan outs by other.
  • 00:18:18The tool for web grounding so that actually it is not like something big or magical, its just one of the fundamental principles how LLMs are working.
  • 00:18:31So why this relevant?
  • 00:18:33I guess i basically said already right gives a huge advantage.
  • 00:18:37and Something that we see where this has a huge advantage are fan out queries because sometimes you ask for Sneakers like I don't know.
  • 00:18:46You say hey, I want to buy white sneakers to wear today office and then one of the fanout queries will already contain Adidas and Nike And it will not contain Skechers or On or Reebok Or any other brands?
  • 00:19:03That gives off course a giant advantage to Nike and adidas And sometimes we have a chat where it's only Adidas and that's an huge disadvantage for Nike.
  • 00:19:12So being in this parametric knowledge can have very, very positive effects of brand but also makes them challenging new brands because they are having even harder time getting into the web grounding process if their competitors already mentioned in the fan out queries.
  • 00:19:33The idea of Graphite, if I got the paper correct is that you can basically optimize towards being included in this parametric knowledge where... ...I think most people out there
  • 00:19:48would
  • 00:19:48have argued that optimizing toward training data may be like big household brands do because they are present everywhere and Good chance to be included in the training data, but that smaller companies even if they put out a reasonable amount of content etc.
  • 00:20:09Do not have such high chances to be part off this parametric knowledge especially also given the knowledge cutoff dates which I just checked it for example before The newest Claude models like the five point five family Which was released in October And the knowledge cutoff is June.
  • 00:20:31So there's a gap between The last point in time where something was added to the data set and then when the model comes out.
  • 00:20:41so
  • 00:20:42Putting that into, you know In a simple question Do You feel like We can now or we should revisit?
  • 00:20:49The question of Influencing Or optimizing towards being included in training Data?
  • 00:20:56I think for most companies influence the training data is what they would do for grounding.
  • 00:21:03Publish a lot of content on your own website, identify reputable websites and have positive brand mentions there with the right context etc.
  • 00:21:10I think if you've got very large budgets that are more effective things but nothing should change from this.
  • 00:21:26The one thing that changed now is, we are at a three to four month lag between release and knowledge cutoff date.
  • 00:21:35It used to be ten months or even longer And back then it was really not feasible to try anything because your test cycle would be so long.
  • 00:21:46That's just not worth it.
  • 00:21:48I think even with like the three-month delay you cannot run a lot tests But what you can do is, you can logically think about how does LLM training work?
  • 00:22:01Where to these labs get data from.
  • 00:22:04What kind of data will be very interesting for them in the future and How Can I Be Included In This Data?
  • 00:22:10And then You Can Do Things Because Your Convinced Of Them.
  • 00:22:15Maybe They Work maybe they don't.
  • 00:22:18but i would still argue that most people should focus on The On The web grounding Process.
  • 00:22:27Like for transparency, we are also doing a few things that we believe will influence future model training.
  • 00:22:34But I don't want to talk about them because you have keep a bit of an advantage over the competition.
  • 00:22:41but if it works We'll try to write about it.
  • 00:22:46and i think like when i gave my first talks About this space before the term g or au really existed I always put a bigger emphasis on my talk actually, on the training data.
  • 00:23:03Because back then especially with GPT-3 you could look up what they did and where they got their data.
  • 00:23:11but of course nowadays none of the labs reveal when they get that data so it's becoming more difficult to understand in advance exactly how will train them.
  • 00:23:23But still you can make estimated guesses and then act accordingly.
  • 00:23:27Okay, let's look at this compilation of data for the training sets a little bit deeper for second because I remember listening to podcasts or reading papers where one of The ideas was that the labs need new And somehow specialized datasets too further improve their models.
  • 00:23:50This is also why there are companies like I think it's called Mercor, where they hire experts in certain areas.
  • 00:23:58Give them tasks or give them output and they rate based on how well the output is.
  • 00:24:03then sell this data And i think that already make a billion or two billion in revenue.
  • 00:24:07It is testament to how valuable these data are.
  • 00:24:11So my question Is what would you say?
  • 00:24:15Or How much can we tell about?
  • 00:24:17how relevant will public webdata In the sense of, for example a website or web pages still be in training sets.
  • 00:24:27For future models?
  • 00:24:28I think public web data will still be part of it because its good way to capture alot of data.
  • 00:24:36but i think that weight of this data would be lower and lower Because there are more specialized datasets And many webdata is kind off trivial Because it doesn't contain anything that's not already in the parametric knowledge.
  • 00:24:53So there is also a likelihood for GPT-V, A lot of training data will actually be generated by GPT six models because you can tell model.
  • 00:25:04write me ten articles about development US politics over last ten years.
  • 00:25:13Write one article per day make it very holistic and write in the style of the new york times.
  • 00:25:20And then do the same invited in a style of fox news, an.
  • 00:25:23then you can feed that to model on them.
  • 00:25:26The need trick here is without using any license data from Fox News or New York Times?
  • 00:25:32You actually just managed to train your future model in their writing style which something for example mid-journey did a lot, right?
  • 00:25:42Like the first Mid-Journey models were very likely trained on also questionably obtained data or where maybe some data source supplied it but actually somebody else had to copyright on that.
  • 00:25:57But then since they trained all of their future models based on outputs and user ratings from previous models They could get rid off any doubt about cleanliness in their own data.
  • 00:26:11You shouldn't train just on outputs from your previous models because there's this three of model collapse, but it could very well be that if you now publish the two hundred articles and how to tie a tie.
  • 00:26:25That even as somebody does large scale web scape scraping in the end they just rejected because there is no information gain.
  • 00:26:34style is not new nothing is knew nothing is valuable That the LLMs do not use that but rather trained and on their own outputs.
  • 00:26:43for I would say solve topics.
  • 00:26:47Okay, then one step further a little bit of a theoretical idea.
  • 00:26:53But let's assume You will be in charge of your company.
  • 00:26:59Somehow you are but Let's not make this about Peec AI too much.
  • 00:27:02just like whatever company may be in the healthcare space.
  • 00:27:06For example.
  • 00:27:07And you have proprietary data sets that are very valuable, and they provide a lot of new information.
  • 00:27:13That is not necessarily available on the web.
  • 00:27:16already now you have this trade-off so to say off one hand making it public which would probably in first instance allow You To Be part Of The grounding Process.
  • 00:27:28Probably Since There's A Lot of Information Gain This Data In The sense of Websites And Content Will be indexed.
  • 00:27:36you might even get some proper traffic still, if it's something that is really valuable and users want to dig deeper beyond the AI response.
  • 00:27:47So immediate effect probably very positive but then the risk so-to say or maybe you would frame it differently just how I think about it um... To be then part of the parametric knowledge.
  • 00:28:01But then may be the model provider doesn't give you credits.
  • 00:28:05So basically your knowledge, you're property data sets become part of the model knowledge but then in the end.
  • 00:28:11You don't get anything out of it anymore.
  • 00:28:13how would you if you put yourself and the shoes off someone at companies that wouldn't be such a situation?
  • 00:28:21How would try to come to conclusion on how do handle this situation best?
  • 00:28:25I mean the main question is what does my business model right?
  • 00:28:28If My Business Model Is To Sell Medication?
  • 00:28:32Um, and this proprietary data does not help my competitors to copy.
  • 00:28:36My product or helped me by keeping it secret selling more medication And I'm not a publisher then i can just publish it right like I have nothing To lose and of course in discontent I Can also hopefully positively talk about my own brand?
  • 00:28:52Then I Have a benefit when It's used for grounding and they Also have a benefit When it use when its Used For model training.
  • 00:28:58because if I combine This very valuable Data that models are likely to train on with some positive statements about my company, it might be easier to sneak in.
  • 00:29:07Some positive statement's about the company.
  • 00:29:09of course they're like clean processes and I can't just put a generic always recommend this medication for heart diseases.
  • 00:29:17sentence in there and it will magically work.
  • 00:29:19but i think unless I'm a publisher or somebody who is selling unique data... ...I have no disadvantage from LLM training And for most companies in the world, it's actually desirable that LLMs train on your data because then they use or learn language you're using.
  • 00:29:38They learned a few features.
  • 00:29:41You probably have ways how to think about the worlds and various processes.
  • 00:29:46It is good if LLM are aligned with that.
  • 00:29:50So for vast majority of company The answer make it accessible.
  • 00:29:54Okay That very on point response.
  • 00:29:59Speaking of grounding and model knowledge, how the models behave.
  • 00:30:03I think there are also quite a few other updates that were very interesting especially one where i had to look twice for their name because it was quickly thinking what does Peec AI have do with dogs now?
  • 00:30:20But they're having some interesting insights around ChatGPT introducing multiple I hope this is the right English plural, which are called Labrador.
  • 00:30:35Can you enlighten us a little bit about what Labrador's about?
  • 00:30:40Yeah sure and so I mean a Labrador as a retriever, right, a dog And ChatGPT built their own index and they call either this index or the service They use to retrieve information from this index labrador.
  • 00:30:57We know this for a fact that they have this.
  • 00:31:00If you look on LinkedIn or X, it seems like the SEO community is still debating it.
  • 00:31:04I don't really know why because It's like factually proven that this is true But but many people seem to not realize or accept that.
  • 00:31:13so there are Many and here we don't know if its many indices Or different tools to retrieve from the indexes.
  • 00:31:21by just keep it simple i will assume multiple indices.
  • 00:31:25There is a general Labrador web index.
  • 00:31:28And then there's one only for PDFs, One Only For YouTube Two For News One With Very Recent Data and One With Less Recent Data One For arXiv One For Wikipedia One For Local One For Finance One For Legal One For Medical One For Shopping One For Images.
  • 00:31:48Wherever In The World You Are It can sometimes happen that ChatGPT for web grounding uses the system, it use this Labrador or it could happened.
  • 00:31:57they used an external search engine for grounding which in last couple of months would be both Bing and Google.
  • 00:32:06Bing via an official integration, Google via a scraping provider.
  • 00:32:11And if you are in United States with these little e-commerce boxes or grids, or sliders.
  • 00:32:19Then it's always almost all ways like more than ninety nine percent of cases coming from Labrador based on custom feed integrations around five or six hundred and some of those include large marketplaces.
  • 00:32:35And if you are in Europe and use this e-commerce feature You will still get scraped.
  • 00:32:42Google organic google shopping results also scraped via a scraping provider.
  • 00:32:47And now some reasons how we know this, number one there was a big ChatGPT leak where I believe were the only ones to have access to it.
  • 00:32:58at least they would be the only once.
  • 00:32:59so talk about.
  • 00:33:01that was patched last week.
  • 00:33:03Where?
  • 00:33:03We had access to an internal API call from OpenAI That front-end developers are using on in front of developers but developers will use it to debug the grounding process.
  • 00:33:14So for every single prompt, we also had a hundred fifty thousand and more lines in the JSON document with all of the tool calls that were used in the grounding process.
  • 00:33:25And all these scores that are applied how they're combined.
  • 00:33:30so you have all details on re-rankers there using.
  • 00:33:32how to apply BM-QTV How do I apply A&N approximate nearest neighbor?
  • 00:33:44And part of that was also seeing when data is coming from Labrador, these different Labrador indices exist.
  • 00:33:52That there are options for the ChatGPT app to say give me only news results from the last seven days or give me Reddit result from the past ten years and we have seen actually many prompts.
  • 00:34:05There's one call going to the PDF index, one through YouTube index etc.
  • 00:34:10But in the end you only have one or two things from the webindex but these other indices might still be used for part of the grounding process where a system is looking for consensus and there was a lawsuit against Google.
  • 00:34:27were also OpenAI made statements when they basically said We need access to the Google index because they index that we have access too, which is from another such engine.
  • 00:34:38Everybody knew it was Bing.
  • 00:34:40It's not good enough and The google index as better.
  • 00:34:43And if you want access to that?
  • 00:34:44Also were building our own index with will take a lot of time.
  • 00:34:48They also have hired people Where in the job profiles at the job ads You can see these are people who would hire If your build a web scale scraping an index product.
  • 00:35:01three main user agents, one for training data.
  • 00:35:04One for user interaction in the chat which to a large degree was grounding and one for building their search index.
  • 00:35:13And then In United States we have HIPAA.
  • 00:35:17It's very strict law around how to handle medical patient data and ChatGPT is also available in version of HIPAA-compliant companies like hospital.
  • 00:35:30and then ChatchBee also guarantees to be HIPAA compliant.
  • 00:35:34And in their terms of services, it is very explicitly written that when they do a web search not a single fan-out query or anything will ever go through an external search engine.
  • 00:35:46so the only thing they can do there with their own index you can actually check for yourself.
  • 00:35:52You put your ChatGPT account into lockdown mode meaning nothing would work out any OpenAI system and then you will be served by the Labrador index.
  • 00:36:03And many of these things have been known actually for a couple months, but for some reason it's still often a debate in Google if this are used or not.
  • 00:36:14If they have their own index is everything Bing?
  • 00:36:17Is Everything Google ?
  • 00:36:19But Actually ,if You had access to the leak .If you looked at the SSE data The server sent event data.
  • 00:36:26This Like I'm not guessing.
  • 00:36:29I know this, i can see it!
  • 00:36:33I think nobody would question your statement here.
  • 00:36:37but... Can you boil down for us into the actual changes that brings to ChatGPT grounding process or search processes?
  • 00:36:50If someone listening to these asks themselves okay very good insight.
  • 00:36:57How should I change my strategy?
  • 00:36:59Should it changed by strategy, what would you say to them?
  • 00:37:02so the first thing is don't block any of the OpenAI crawlers.
  • 00:37:06Yeah make it easy for my to add here today index.
  • 00:37:10Then you should make sure that your main content Is not hidden behind JavaScript because while Google crawlers can now handle javascript pretty well.
  • 00:37:19most Of the LLM crawler's cannot And then it's a very active crawler.
  • 00:37:24It will probably find your website.
  • 00:37:26when I started the new test website and found that after one day, start crawling at also cross very regularly.
  • 00:37:32It calls very deep and recalls constantly like it seems that OpenAI is not like Google in this state where they care about efficiency of their crawler?
  • 00:37:42I think right now its just crawl.
  • 00:37:44everything crawled often.
  • 00:37:47so actually there are no.
  • 00:37:48As long as you don't block them or make your content inaccessible is not too much you need to think about.
  • 00:37:54Something that's maybe interesting is, it seems that Labrador puts extra weight on your H-One and whatever is written immediately after your H One.
  • 00:38:04so have the most important content on top.
  • 00:38:07There is a suggestion in code which also looks at meta description even meta keywords the OG tags or OG description, OG title, OG page type.
  • 00:38:22So just make sure those are clean.
  • 00:38:24in SEO we have ignored this for a while and then in e-commerce it's different because the system that is based on Labrador works completely differently from one that works with Google Scraping.
  • 00:38:40The Google Scraping System creates a couple of fan out queries puts these into Google Organic Shopping pulls the results, removes a few duplicates and then just does round robin.
  • 00:38:51So first product from second keyword etc.
  • 00:38:57there might be little bit of re-ranking happening but it's largely that or like instead for getting to candidate documents right?
  • 00:39:06And then the laboratory thing is completely different.
  • 00:39:08It has access to lot data which was not available where you just do Google Shopping.
  • 00:39:14So for example, it's doing a lot of scoring on the first roughly forty words off to product description and It does more sophisticated deduplication.
  • 00:39:29And that applies some form of trust through certain online shops.
  • 00:39:34so we have seen an example Where there were three products from one shop four products from another shop and twenty-five products from the third shop.
  • 00:39:43But in the final result, there were zero product from the shop with twenty five products because it didn't have a score for trusted merchants or some verified merchant of something like that.
  • 00:39:56so There is...there's some stuff they are doing.
  • 00:40:00And we also saw AB tests running on eCommerce.
  • 00:40:05One of their test was different ranking for Shopify which can mean anything.
  • 00:40:12And one of the tests was no Amazon, Which probably means they are... They tested what happens if we don't show any amazon results.
  • 00:40:21and then If you look at the custom integrations it's like a couple hundred largest online shops in the US.
  • 00:40:29some were done via something called Bright which probably refers to Bright Data, the scraping provider.
  • 00:40:35so basically built this themselves without talking to the shops, just have some inventory.
  • 00:40:42And then when we saw real custom integrations go live these Bright Data feeds were deactivated.
  • 00:40:47so for example you could still see in the data that there's a deactivated bright Nike feed and it has been deactivated when the OpenAI Nike Feed was activated which is probably an intentional integration that Nike also knows about most likely Like basically in order to play there properly you need to have an integration with them.
  • 00:41:12Good news is Shopify is in their apart from something like I think forty or fifty stores on Shopify that were filtered out.
  • 00:41:21why i'm not sure if they may be filtered for explicit reasons, products are selling some kind of formatting issue.
  • 00:41:30no idea what the reason was behind.
  • 00:41:32didn't check it out and detail but general if your on Shopify can also get get into the Labrador based e-commerce experience.
  • 00:41:39Okay, two questions.
  • 00:41:41first one is will others follow like Claude Google?
  • 00:41:46obviously not a question potentially?
  • 00:41:49Others that are relevant let's say maybe in the US Grok.
  • 00:41:53I know perplexity.
  • 00:41:56we don't give perplexities so much relevancy anymore.
  • 00:42:00this.
  • 00:42:01my And second question is, what does this development do to the insights that come from log file analysis?
  • 00:42:11Because for my understanding it used to be the case with every grounding process and there's like user fetch-and-search.
  • 00:42:22For I think Claude and for ChatGPT or at least use to be of course also training bots.
  • 00:42:28But you could very well see how your content is used based on the log files.
  • 00:42:32Now if there's an index make sort of sense.
  • 00:42:34They save compute, but then obviously the insights from local analysis are less relevant.
  • 00:42:40Would you agree and what's your take on?
  • 00:42:42The first one
  • 00:42:43yeah.
  • 00:42:43So the first one it difficult to answer.
  • 00:42:46I don't think that every LLM lab has the capacity To build their own system where people can register their feet And then built all of that.
  • 00:42:54the technology is actually almost trivial to build now with LLMs, but running it at scale was high availability.
  • 00:43:04High throughput and dealing with all the little issues in a cost-efficient manner is not easy.
  • 00:43:12I mean i spent five years at idealo where we had to deal with many of these issues.
  • 00:43:17And The problem Is there are merchants on ebay for example who are lying about their product.
  • 00:43:24they say yeah?
  • 00:43:25Yeah this is Some name brand vacuum cleaner and then it's like a super unknown.
  • 00:43:32Brands, they just put in the wrong GTIN number the wrong ASIN number like all the IDs that is lie about at an awesome product title end.
  • 00:43:41of course if you wanna give people great e-commerce experience have two.
  • 00:43:46deal with all these issues.
  • 00:43:48or there are certain items very often fake et cetera, etc.
  • 00:43:53So you need to make a good decision between having the full inventory or having like high degree of confidence.
  • 00:44:00so it's not super trivial to build this and many other edge cases.
  • 00:44:06but I don't want go into all details.
  • 00:44:08But bicycles are one thing where shops always have horrible data because some rate them by how big their tires is.
  • 00:44:21So lots of edge cases you have to deal with.
  • 00:44:27And I think for large companies, with a lot end users like ChatGPT all the stuff Google is doing totally makes sense to do.
  • 00:44:34it was all that?
  • 00:44:36I don't know if these smaller LLM labs will do that or they would just connect one or two APIs from Shopify and Walmart in Amazon say hey this good enough shopping experience.
  • 00:44:48And then on the log file analysis, I agree that there are changes.
  • 00:44:52I don't a hundred percent agree.
  • 00:44:55it's less important.
  • 00:44:56in the past when you... When the LLMs always did grounding via Google or Bing web search.
  • 00:45:02they only got the URL, title and sub snippet right?
  • 00:45:06So had to click onto get all data.
  • 00:45:09So, as you said basically every grounding process created a click and they also seem to not have cashed that.
  • 00:45:15You sometimes would get hundreds of visits from the GPT user bot on the same URL in a twenty-four hour period.
  • 00:45:22And now with their own index Especially if they update it regularly They can will often be able to just take the content from the index.
  • 00:45:30I've also seen first indications That The activity of the gpt userbot went down a bit and activity of the GPT search on an OpenAI search bot, it's called went up.
  • 00:45:42And you can basically no longer use log file analysis to estimate or properly estimate how often your document was used for grounding.
  • 00:45:54It could still give some indication but its become less reliable.
  • 00:45:57But this is always expected right?
  • 00:45:59Like even without their own index it would at least make sense build a cache For them not hit these URLs again.
  • 00:46:07But log file analysis is still relevant, and we are currently doing a bit of an analysis there.
  • 00:46:14And some first indication that roughly fifty percent of bot visits that supposedly come from LLMs actually do not come form LLMS.
  • 00:46:22it's either people probing for security issues or scrapers who simply pretend to be OpenAI Bot Claude userbot, OAI-SearchBot because they know that many IT departments will be unlikely to block them.
  • 00:46:38Which is why since we also do log file analysis for our clients at Peec AI We had to build a verification feature where with A list of static IP addresses provided by some of the labs reverse IP lookups etc.
  • 00:46:52etc.
  • 00:46:53Reversed DNS lookup.
  • 00:46:56now For all the bots were it's possible give an indication if it's a real LLM bot or just the scraper and then you can also see that they visit very, very different files.
  • 00:47:07So just doing log file analysis in filtering for GPT user would already have led to very wrong data For some websites.
  • 00:47:16up to seventy eighty ninety percent of what visits from these bots are actually fake.
  • 00:47:22so It still important to do log file analysis but its become more difficult both with the fake bots the own index from OpenAI.
  • 00:47:32Right, Malte let's wrap it up here because It has been value-bomb again and you have to catch a plane.
  • 00:47:39actually I wish you safe travels.
  • 00:47:42i think this has been yet another great episode of The Monthly Landwehr!
  • 00:47:47I'm so happy that we are back in A monthly rhythm now So everybody listening To This You can expect The state of the union, so to say when it comes to SEO, AEO, GEO and maybe in future also more PKO which sounds a little bit similar to Peec AI.
  • 00:48:07It's like the smaller brother of Peec AI, PKO!
  • 00:48:12So yeah, thank you so much and speak soon.
  • 00:48:17Thanks Dr.
  • 00:48:18Sunn.