← All episodes
Masters of Search episode 45: AI Search Attribution, ChatGPT Clicks, Prompt Tracking, Google Search Agents | #MonthlyLandwehr cover art
EP 45·Jul 20, 2026

AI Search Attribution, ChatGPT Clicks, Prompt Tracking, Google Search Agents | #MonthlyLandwehr

Show notes

Every month I meet with Malte to take all the news, new research and interesting developments in SEO & AI Search to get his perspective on them. Malte Landwehr is CPO & CMO at Peec AI, one of the leading AI search analytics platforms.

My biggest takeaways from this episode:

1) Click-based attribution massively undercounts AI search. In a study from two Peec AI clients, click-based attribution credited AI search with 0.9% of leads. When they simply asked people "how did you find us?", it was 9%. That is 10x. When there is no link to click, people just type your brand into Google or the address bar, so the LLM recommendation gets credited to brand or direct instead. (Reference: https://www.linkedin.com/posts/ethanls_n8n-aeo-case-study-self-reported-vs-last-share-7475579627730690048-dMsn/))

2) Self-reported attribution is the best tool we have here, and it is almost too simple. Add one optional question at the end of your sign-up flow: "where did you hear about us?" Keep it optional so it doesn't hurt conversion, randomize the order, and drop in a fake option (a channel you don't use) to measure your own noise level. It is not either/or against your attribution software either. In his view, a truly complete tool would already include this.

3) ChatGPT started sending out a lot more clicks around May 7. It began adding far more links inside answers, especially list-style answers where every brand became a link to its homepage. Reported clicks at least doubled for most people, with some seeing +400% or +500%, and the effect looked stronger in the US than in Europe. His prediction: it will not grow from here and may already be reversing, since everyone wants to keep users in the app.

4) Prompt tracking works, but not off a single prompt. In their new paper, Peec AI re-ran Rand Fishkin's dataset of unique prompts many times over. Around 90% returned the same set of brands when run repeatedly; only 10% were truly unique. Most variation comes from LLM non-determinism, so you need many runs, not the exact wording your customers used. Small changes still matter: adding "list" gets more brands, framing the model as an expert gets fewer, and mid-funnel prompts are the most sensitive to rewording. (Reference: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6914539))

5) Google's search agents could be the first thing to pull Malte away from Claude. Google already has your email, calendar, and context. A search bar that handles quick questions, personal context, complex AI-mode chats, and scheduled agents running in the cloud is very powerful. He compared it to Duplex in 2018: huge if it works and gets adopted, but Google has tried and failed at things like this before.

6) The Peec AI team is spinning up a huge test to see how ChatGPT crawls. They noticed some sites keep getting thousands of ChatGPT crawls a day long after Google has given up on them, so they set out to find the limit and see how deep it goes, which other LLMs follow, and what it actually costs to host. It won't prove whether OpenAI runs its own index (Malte is convinced it does, as one of several grounding options), but it is a rare, fully public look at crawler behavior. (Reference: https://sandnjournal.com/analytics))

Let's connect!

Niklas on LinkedIn: https://www.linkedin.com/in/niklas-buschner/ Radyant on LinkedIn: https://www.linkedin.com/company/radyant/ Malte on LinkedIn: https://www.linkedin.com/in/landwehr/ Peec AI on LinkedIn: https://www.linkedin.com/company/peec-ai/

Transcript

Full conversation

via podigee
  • 00:00:00self-reported attribution is more or less the only tool, probably the best tool marketers have in their toolbox to understand how people came into that website.
  • 00:00:09So what you do at end of your sign up flow?
  • 00:00:11You simply ask people hey where did he hear about us?
  • 00:00:15then?
  • 00:00:15AI search should be one option and if want it next level.
  • 00:00:19but as a second question when somebody says yes it was AI search can asked them what prompt they use.
  • 00:00:26What do you think, what's your take on search agents and their potential relevancy for the future of Search?
  • 00:00:31I think if Google does them well they can become super relevant because Google already has all my contacts.
  • 00:00:39They have email, calendar.
  • 00:00:41that is very powerful!
  • 00:00:43That could be first thing.
  • 00:00:44makes me use something else in Claude to be honest a long time
  • 00:00:48Before we dive-in You're listening to The Masters Of Search podcast with me Your host Niklas Buschner.
  • 00:00:55Each week, I sit down with some of these modest people around the world in SEO and AI search to bring you their strategies mental models.
  • 00:01:03And top pieces of actionable advice.
  • 00:01:05if You enjoy this podcast don't forget to like and subscribe.
  • 00:01:09then follow it In your favorite podcasting app or YouTube?
  • 00:01:12It helps us get top-notch guests and create The best possible content for you.
  • 00:01:16let's dive into today's episode.
  • 00:01:18It is time again for my favorite format in this podcast, the Master's of Search Podcast.
  • 00:01:24And it is time for Monthly Landwehr.
  • 00:01:27Malte has already smiling because I tried to study this intro... ...because someone told me you should start with a same intro so that you have recognition effect.
  • 00:01:38But no honestly!
  • 00:01:39I'm always looking forward to this alot Because its MY private hour and as i also posted on LinkedIn I keep Malte's calendar link a well-known secret so that i can just book in and ask all the stuff, which is very interesting.
  • 00:01:59And get his knowledge on it!
  • 00:02:01Yeah, Malte thanks for joining again today.
  • 00:02:04how are you?
  • 00:02:04How has been your week so far ?
  • 00:02:06Thanks for having me !
  • 00:02:07It was great week actually.
  • 00:02:10yeah busy as always
  • 00:02:13Busy.
  • 00:02:13the weather is good, but not too busy.
  • 00:02:17So I'm I'm good
  • 00:02:19and The weather's not to good so you're not constantly thinking about going outside But your comfortable staying at your desk And Making if
  • 00:02:30i have an air-conditioned?
  • 00:02:31So i will Not leave the office If i can avoid it.
  • 00:02:34yeah i feel like this Is the ultimate hack To get people back to the office.
  • 00:02:39if You Have a Good Air Condition in Summer It.
  • 00:02:42it is much more uncomfortable if you work remotely, but people will come to the office more often.
  • 00:02:51We have a pet with couple of topics and I want start on an easy warm-up for your topic of attribution!
  • 00:03:01But i think also seen you starting the video series from PKI where you try to explain some basic concepts and ways of thinking about GEO and AI search.
  • 00:03:19And I think an important topic is attribution, and that i'm also constantly talking to our clients or potential clients about this topic... ...and I want to start like very easily.
  • 00:03:33why do YOU think attribution is something people should THINK about when it comes to AI Search?
  • 00:03:41In the past, if we did attribution on the web it was usually with clicks.
  • 00:03:45You either look at the HTTP referrer or you have a UTM parameter at end of your URL and especially when you come from performance marketing to use this way of attribution also when you're coming from SEO just used for attribution right?
  • 00:04:00It's what you know.
  • 00:04:01but AI search breaks because there are often no links to click on, right?
  • 00:04:07People get a recommendation about brand what service which company to go with.
  • 00:04:12And that was recently study from two of our clients and eight NN and graphite.
  • 00:04:18And together they found out that if you look at.
  • 00:04:22I think it wasn't leads that they looked at.
  • 00:04:24If you do it on click based attribution zero point.
  • 00:04:27nine percent come from AI search for an item.
  • 00:04:31but if your ask people hey How did you find us?
  • 00:04:35It's actually nine percent of people.
  • 00:04:37So in this so-called self reported attribution, it is ten times more than a click based attribution.
  • 00:04:44And that makes total sense, right?
  • 00:04:45Because if you get a recommendation from Gemini or Claude and there's no link.
  • 00:04:50What do you do?
  • 00:04:51You either type it into your Chrome address bar... ...or go to Google.
  • 00:04:55So is was be organic SEO or organic SEO brand Or maybe even paid?
  • 00:05:01because for branded searches often its a paid click How the person lands on your website.
  • 00:05:06It´s directly typed in which usually also attributed to Brand.
  • 00:05:09But at end LLM gave this recommendation.
  • 00:05:12So similar to word of mouth, recommendations from friends, recommendations for coworkers.
  • 00:05:19Self-reported attribution is more or less the only tool or probably the best tool marketers have in their toolbox to understand how people came to that website.
  • 00:05:29so what you do at your sign up flow?
  • 00:05:32You simply ask people hey where did he hear about us?
  • 00:05:36then AI search should be one option and if want it next level What you can do is as a second question.
  • 00:05:43when somebody says yes, it was I search.
  • 00:05:46You can ask them what prompted to use and then you store that in the database?
  • 00:05:51And then when you wanted one more next level which is what i'm right now doing um...I have a cloud automation That every day checks if people filled this out for pki with a prompt.
  • 00:06:04If yes, I grab all of the prompts over last couple days.
  • 00:06:06Check if they are unique and add them to a project via the PKI MCP so that i have a project where there only prompts in from real prompts.
  • 00:06:16uh... That people have entered who signed up to pki And it's a nice way to see prompt sets that generate.
  • 00:06:25was this prompt set based on People that found us?
  • 00:06:28Of course There is bias right because you Only get the prompts From people Who did find You.
  • 00:06:32But still, it's a very nice data set to have.
  • 00:06:35To understand how do people actually prompt?
  • 00:06:38And some of this is super weird stuff that you would not have expected.
  • 00:06:43and the important thing with self-reported attribution Is if You Have A Big Signup Form You Should Not Add Another Thingy Right That Will Kill Your Conversion Rate Just After Somebody Converted.
  • 00:06:55Ask Them This And Make It Optional.
  • 00:06:56So If They Skip It they Still Completed The sign up Flow but It Can Really Really Help With understanding better how people find you.
  • 00:07:05I personally would always add in something that you don't do like let's say, You Don't Do Snapchat.
  • 00:07:11just put in Snapchat as an option because if then five percent of people click on Snapchat... ...you can be pretty certain that your Five Percent uncertainty at least has a test from time to Time.
  • 00:07:21it is important.
  • 00:07:23also the options should Be In Random Order otherwise The Things That Are On Top will Always Click More Often.
  • 00:07:30Yeah, and then if you treat this as something that's not perfect but just directionally in like a directional indicator of truth it can be super useful.
  • 00:07:41And I mean just Google or ask GPD about self-reported attribution.
  • 00:07:46there are few more things to consider.
  • 00:07:49for example people typically mention the system that they use the most instead of a system, but I actually used to find you because they don't remember.
  • 00:07:58it's not that people intentionally lie.
  • 00:08:00They just don't know any better.
  • 00:08:01and then also small percentage will also lie intentionally etc.
  • 00:08:05etc.
  • 00:08:06But still its very good data.
  • 00:08:09otherwise we wouldn't get.
  • 00:08:11It sounds so simple almost too simple.
  • 00:08:15now i for example am confronted And I don't mean this negatively.
  • 00:08:22I think it's part of the job with people that have been told and also marketed by for example attribution software attribution providers Software, that provides a media mix modeling multi-touch attributions supposedly perfect technically Perfect solutions to tell you exactly Which touch coins?
  • 00:08:44A user had during their journey which mostly does not consist only off One channel or one touch point.
  • 00:08:51Now, how would you maybe also have the same conversations?
  • 00:08:55How would you try to convince a CMO ahead of marketing whoever in The end does that decision to change an attribution model Or at least allow for another legitimate way To look at attribution.
  • 00:09:10How Would You Try To Convince Them That This Supposedly Very Simple Fix Is Better Than I don't know, ten K-twenty K attributions of wear license they have been paying for.
  • 00:09:25I mean i'm not sure this fix is better than that software because the perfect attribution software would include this right?
  • 00:09:32Otherwise it's a performance marketing attribution softwares and overall attribution.
  • 00:09:38And if you look at how TV advertising radio out of home digital, out-of-home etc.
  • 00:09:45Etc.,
  • 00:09:45I attribute it.
  • 00:09:46there are just things that you cannot do perfectly with kick based attribution and as soon as we have like marketing mix basically the only thing That you can do is shutting off one channel and seeing The effect on other channels?
  • 00:09:59Then You always Have to problem that you Cannot really turn Off affiliate Marketing Unless you Really want a mess With your filled with Your publishing partners.
  • 00:10:08You cannot turn off SEO without causing major long term damage.
  • 00:10:12Um, you can not really turn of word-of-mouth et cetera, et cetera.
  • 00:10:17um and you cannot turn on AI search unless your willing to have Major Long Term Negative Effects.
  • 00:10:24So there you always had some way to deal with it.
  • 00:10:28And if somebody is convinced that they have the perfect solution I would simply ask them how they account for AI Search?
  • 00:10:34If They Have No Good Answer Then i am very sure that I should do the self-reported attribution in addition.
  • 00:11:06Neither is perfect and I need to find a model how-to combine them.
  • 00:11:11And if i pay A lot of money for a marketing attribution software, I expect that they can somehow model in AI search?
  • 00:11:19If They don't have a good answer...I still might Have to use Them To allocate my paid CIM Etc budget.
  • 00:11:27but I also Need to have the second reality That actually a significant part Is driven by AI Search.
  • 00:11:34it's similar For branded SEO traffic.
  • 00:11:37A lot of that should not actually be attributed to the SEO team, but rather to TV marketing brand
  • 00:11:43etc.,
  • 00:11:44etc.. But yeah it's.
  • 00:11:46there is no easy answer.
  • 00:11:47if you want perfect and exact numbers... ...but with self-reported attribution It's very easy to get.
  • 00:11:52a first indication
  • 00:11:54Sounds good.
  • 00:11:55I could relate especially With what you said That you have to accept contradicting realities in a way And its also something Pipp Kleppner once said on a podcast that this is basically part of the definition for someone who's smart and you can accept two contradicting realities living in your head without going crazy.
  • 00:12:17can you share, obviously adjusted to the public space here?
  • 00:12:24Although this podcast is a safe space.
  • 00:12:26Can you share how important self-reported attribution is for example for you and your marketing decisions at PKI?
  • 00:12:32You gave us already a glimpse into like next level ways of asking people for their prompts.
  • 00:12:43If you look at your marketing mix, that obviously also already consists of quite a few channels.
  • 00:12:48How much often importance has self-reported attribution in driving your decisions?
  • 00:12:55Data wise I would say it's one of the top two or three things we look at but of course We are still an early stage company like not even to years old and developing space so Not every decision can be pure database right.
  • 00:13:11there is also vision and anticipation And conviction to go in towards certain directions, but data-wise it's one of the top things we look at.
  • 00:13:22Okay I think that already tells a lot because One can obviously also see from the outside based on all the ad manager transparency centers That you're already do a little bit of LinkedIn advertising.
  • 00:13:34You do some Google paid?
  • 00:13:42with a high relevancy.
  • 00:13:43So thanks for sharing that.
  • 00:13:46Now the premise or I think so to say, the inception moment Or the renewed inception moment For self-reported attribution as you said has been The decline of importance Of click because AI search interfaces are not so Clickable and have not built To be so clickable A couple of weeks ago, I think.
  • 00:14:12I guess it was people started reporting it somewhere in June or end of May and they tie back to the beginning of may.
  • 00:14:23There was a sudden spike in clicks from chat GBT that a lot of people have reported.
  • 00:14:28i saw post form Adele Cahoe?
  • 00:14:33I hope its pronounced correctly.
  • 00:14:34Have no idea.
  • 00:14:35director product marketing at similar web.
  • 00:14:37She noted it, other people like Kevin Indyk also confirmed.
  • 00:14:43What has been happening there?
  • 00:14:45Has OpenAI, has Sam Oldman suddenly decided that they will be generous with us and give back a little bit of the clicks we lost but still mourn?
  • 00:14:58So what happened is around May seven chatGPT started adding significantly more links inside their answers And the primary thing that we notice is in, I would say list style answers where set of brands are mentioned.
  • 00:15:14Every brand mention was suddenly a link to the home page.
  • 00:15:18That's what you see in the chats that we observe and from clients also talking with various people who have access either log files or web analytics.
  • 00:15:31data seems to be that clicks at least doubled For most people, for some people up to plus four hundred percent plus five hundred percent clicks.
  • 00:15:46The effect seems to be stronger in the US than in Europe but I haven't looked into it too much.
  • 00:15:53if maybe English speaking was non-English speaking.
  • 00:15:56so there i'm not entirely sure what a differentiator is.
  • 00:15:59But in general us brands have reported bigger uplift then local European Brands and It was a very, very significant increase in May.
  • 00:16:10In June it stayed roughly on the same level and now in July I would say its TBD... ...I usually look at this data only on monthly cadence but to me it looks like there might have been a bit of reduction in July.. ..but i haven't looked fully into that so can only answer when month is really over.
  • 00:16:30What's decision behind us?
  • 00:16:32I've no idea could be a test just exploring, hey what would happen if we send out more traffic.
  • 00:16:40From what I'm seeing there's also another potential test running right now with this GPT-Five point six where they are chat GPT seems to be testing to be reliant much less on publishers who are usually the biggest drivers of these like legal pressure and public opinion pleasure that systems need to sent out more clicks.
  • 00:17:01so it might all be coincidence personally from observing what Google is doing very closely for a long time, with the also lawsuits that I was involved in when I said Idealo.
  • 00:17:14I have the impression that chat GPT or OpenAI is checking both directions like What happens if we send more traffic to the open web?
  • 00:17:22In case We are forced to do it but Also what happens If we Are less reliant on Wikipedia and publishers different mix of sources who might be less likely to stir up trouble for us.
  • 00:17:35And what the final result is, I would say it's still very open.
  • 00:17:40Okay if you had to make a bet or If this trend continues or Stabilizes Or if It will rather go back To The Level Before May seventh What Would Be Your Prediction?
  • 00:17:57i'm Pretty Sure.
  • 00:17:58Unless Something Significant Happens the effect is not going to increase, I'm not sure if it will stay the same or go down.
  • 00:18:08If i have to make a prediction ,I would say It Will Go Down.
  • 00:18:10So if we look at the July data there will be less linked mentions than in June and less clicks going out then In June.
  • 00:18:18but...it's A Prediction Like I've No Idea if Im Right Or Wrong.
  • 00:18:22Yeah And What Would Be Your Reason To Think That You Make This Prediction?
  • 00:18:27The Judgeability Wants To Keep People In The App.
  • 00:18:31I mean, everybody wants to keep people in their app.
  • 00:18:33That's part of building the super app and building this ecosystem And Google has found ways to do it for at least fifteen years.
  • 00:18:43All the LLMs are doing it.
  • 00:18:47Nobody is forcing them to do differently.
  • 00:18:51They have now licensing deals with a couple publishers Their promise to their users without the big reliance on publishers that they have right now.
  • 00:19:03So I think if push comes to shove, it will simply accept some publishers block them completely And it's more important to them that people stay in the app, have another chat.
  • 00:19:15Have a follow-up question share all of their personal information.
  • 00:19:18make all their purchase in the App whenever you lose someone going out off the app.
  • 00:19:22That is huge problem for them because they still need to monetize user base much better.
  • 00:19:31I saw one thing talk about monetization.
  • 00:19:36if your look at introduction of ads and the introduction of clickable rent mentions.
  • 00:19:42This is very closely tied together, And one theory also that they are trying to educate their users to click on things.
  • 00:19:50because why ads in Google work so well?
  • 00:19:52it's beacause you're used to click.
  • 00:19:54So if put something on top then click that In chat GBT your code not use to click Why would you click an app?
  • 00:20:00An ad like training people to click on brand mentions might also be part of making people click-on ads that usually recommend a brand or product.
  • 00:20:14Again, I can't prove it but its another theory which makes sense for me.
  • 00:20:19Do you know if ShedGBT has already introduced CPC as a way... Okay yeah they have introduced CosperClick bidding.
  • 00:20:33I just saw it in their blog post on May fifth.
  • 00:20:38That would be a close correlation to what you've said, because from my understanding initially, JGBT ads were like closed circle for couple of big advertisers in the US and based on cost per mill, cost per impression billing right?
  • 00:21:00And now obviously the model that made Google so successful is cost per click and not cost per impression.
  • 00:21:06You could argue Facebook meta, it's still running mostly on CPM.
  • 00:21:12but yeah okay... The hypothesis of making people learn how to click sounds almost a little bit stupid But Making them comfortable with this new pattern makes a lot sense.
  • 00:21:29what your overall Impression of, like?
  • 00:21:33do you have data or any insights on the adoption of ChatchBT ads?
  • 00:21:39I mean not global data.
  • 00:21:40but if you look at the US there are now industries where about one hundred percent of chats haven't had and There's a wide variety of advertisers.
  • 00:21:53I believe it still only open to advertisers from a small set of countries.
  • 00:21:57Like you cannot use a German credit card for example.
  • 00:22:00You need a us credit card probably UK and I don't have the list of other markets that are open right now.
  • 00:22:09So there is a lot of adoption, i saw message today on x where somebody said they stopped their advertising on chat gpt because the impression count is crazily high higher than what would expect possible for their topic.
  • 00:22:25so they assume that some irrelevant ads I would say questionable relevance, but i have not run a large-scale data analysis to make any kind of judgment on like how relevant the ads are.
  • 00:22:41Many are very relevant.
  • 00:22:42that is what I can... ...say from my observations.
  • 00:22:47Maybe chgbt doesn't wait as long as Google has waited to introduce stuff like broad match or performance max campaigns But they just start with with potentially showing irrelevant ads right away and not waiting like fifteen years.
  • 00:23:05so it's not random what they do definitely.
  • 00:23:09And I have a lot of targeting options where you basically describe but want, and then the ads are shown.
  • 00:23:17So i think they immediately starting was something like pmax campaigns or something going into that direction.
  • 00:23:23i think if you want to make the maximum money what should not do is allow people to bid different amounts of money on different kinds of keywords.
  • 00:23:31but he should convince people to give u. Their product feed or just a rough description like list off their landing pages and maybe little bit of text for that target audience, and then make everything automatically.
  • 00:23:44that's how you can onboard very inexperienced advertisers very quickly.
  • 00:23:49But this is also how you can milk the large advertisers, and get their most money out of them.
  • 00:23:56And to circumvent many advertising agencies if there's really just an option to upload something or describe it which I think a bit of trend right now to circumvents anybody who in between us as platform and customer but could be that they first will more agency friendly to get advertising agencies on board But I think long-term, if they want to monetize as good as possible and believe in their own ability to run ads that are relevant.
  • 00:24:29There should actually be no targeting where you bid different amounts of money for different keywords like there should been integration.
  • 00:24:34They showed have conversion events And then run all these ads.
  • 00:24:39i Have a bit of insights from the couple large companies that are similar to OpenAI, roughly.
  • 00:24:47And I know many of them consider these things like what kind of advertising options will they actually offer in the future?
  • 00:24:57Now based on your experience from Idealo especially around shopping placements and the introduction Google Shopping now we obviously also see an increasing relevance.
  • 00:25:12uh e-commerce and chat gbt, the product carousels being shown etc.
  • 00:25:17And you have done or your team i don't know if personally your team has done some great research on showing or proving that a lot of products come from google organic shopping results.
  • 00:25:32but do u see Open AI potentially going the same route as Google that they just sell these slots via their ad engine.
  • 00:25:45First question and second question, I know you cannot give legal advice but from your experience also with a google lawsuits etc.
  • 00:25:53do you think this would be legal or would then potentially backfire like legal action?
  • 00:26:04I don't know, whoever has an interest in that.
  • 00:26:09So in most countries of the world i would say it's legal just like having Google shopping ads on top of google and The only countries that ever did something against these google shopping ads As far as i know are the european union the uk turkey and russia And with Turkey and Russia i think having their most extreme measures And the UK basically copying the Digital Markets Act and from EU.
  • 00:26:38The EU introduced a DMA to regulate these large gatekeepers, but right now the DMA is not being enforced.
  • 00:26:46so... ...and the law that's been enforced isn't a law it just suggestion based on current actions of European Commission or US Administration.
  • 00:27:00I also don't think that the DNA will be and forced.
  • 00:27:03And even if it's being enforced, It might take years until OpenAI or chppt is classified as a. What does that cause?
  • 00:27:12VLOP very large online platform Or what there's like one stage above that in the DMA?
  • 00:27:18I forgot to name right now basically a gatekeeper because The class of answer engines doesn't even exist In the dma.
  • 00:27:25Like Is it a search engine?
  • 00:27:27If yes its not a dominant One Right.
  • 00:27:29so It will take time to regulate and usually regulation is always too slow, it happens shortly before there's a pivot anyways or disruption.
  • 00:27:40So I don't think that lawsuits against open AI introducing shopping ads are great tool that i would want to rely on.
  • 00:27:49I do believe at some point they have super prominent shopping ads but right now the ads are always in the bottom And I think they first need to train people, look at the organic shopping results and buy products before they can monetize it because you usually want to first form a habit.
  • 00:28:11Then make money out of that.
  • 00:28:15so we don't immediately see big shopping ads everywhere.
  • 00:28:21I
  • 00:28:22remember looking at a study from Cystrix, that is already couple of years old where they tried to assess the impact on Google
  • 00:28:31Ads.
  • 00:28:32On amount clicks still left for organic results and i think shopping ads placement or shopping carousel this type of SERP assembly had the strongest negative impact And with other things like featured snippet, etc.
  • 00:28:55It was a little bit less.
  • 00:28:57But I remember that roughly so not pointing to specific numbers here you can Google it or check chat GBT for Cystrix click-through rate study That it was roughly half half Like half of the clicks go to ads have off the clicks goto organic which obviously limits the potential for us SEOs organic marketer significantly.
  • 00:29:20Right now most of the AI search interfaces are still Most so Most strongly organic.
  • 00:29:28you only have limited at placements.
  • 00:29:30Do you think that we will in The future, have a similar setup?
  • 00:29:34That basically half of the clicks Are paid and only a half or not clicks?
  • 00:29:39It's it's hard even for me to say like what is there?
  • 00:29:42What does the currency then?
  • 00:29:42but you know where I'm going Like how well do organic potential over these platforms look like two three years down the road?
  • 00:29:51So that's the problem with the clicks.
  • 00:29:53is there are so few organic clicks.
  • 00:29:55Maybe ad clicks will be fifty percent, maybe they already are because their more paid clicks I can click than organic clicks.
  • 00:30:01for most prompts or most answers?
  • 00:30:03Um...I'm not sure what it would look like.
  • 00:30:07There'll definitely some form of monetization.
  • 00:30:12There also a question how much we actually prompt on our own and just use an agent.
  • 00:30:20because like especially in a B to B context, there are already small purchases that have been done without the person involved.
  • 00:30:28Like if my claw tells me I have to pay for a firecrawl or epi-fi i just do...I don't trust it enough yet to give out credit card.
  • 00:30:36so that's still what we're doing.
  • 00:30:38but ummm.. I was not actually involved with deciding on epi and fireclaw or for Cloudflare and GitHub Premium, all of these things that my cloud needs to do.
  • 00:30:51All the things I wanted to do.
  • 00:30:53And there is how will ads play a role?
  • 00:30:58Can there be ads that influence agents Or will agents immediately use different systems if they are any kind of ads where we just ignore them?
  • 00:31:08There's lot to be figured out because monetization is important While as a consumer on some way, I don't like ads.
  • 00:31:20Also have enough brain capacity to understand that ads are actually the thing That makes many things accessible to the general public.
  • 00:31:28Like watching TV would not be free if there were no ads right and There are many people who cannot afford premium TV cable subscription And they're dependent on ads.
  • 00:31:41same for radio Ads in sports pay for a lot of the salaries off athletes, etc.
  • 00:31:49So yes ads are good in a way because they make stuff accessible For non-rich people but adds also annoying and There somehow has to be away too.
  • 00:32:00deal with this indie in an economy where a lot Of decisions I'm made by agents I don't have.
  • 00:32:09I Don't have the clear answer or perfect vision for it.
  • 00:32:11I'm also thankful to the advertising industry for allowing me see Kilian Mbappé play and pay for his thirty one point two five million euro salary per year.
  • 00:32:25No, no just kidding.
  • 00:32:25i think your point makes a lot of sense.
  • 00:32:32quick transition from you not trusting your agent enough give it credit card.
  • 00:32:41You seem to trust your team enough.
  • 00:32:44To give them your credit card, at least for a Cloudflare account.
  • 00:32:47because I saw how did you like that bridge?
  • 00:32:52Great bridge!
  • 00:32:54Yeah so i saw post from your colleague Mittehan today where he says That you at PKI are now building one billion pages Something.
  • 00:33:08and you know, I'm just playing stupid here now.
  • 00:33:11I usually try to be a knowledgeable host But honestly, I don't know what are you doing there?
  • 00:33:16Why are you creating a billion pages?
  • 00:33:18What is this all about?
  • 00:33:19and why did you give me Tehan your Cloudflare account connected to your credit card?
  • 00:33:24Yeah so that credit card thing was a joke.
  • 00:33:26of course We some team members have websites that they use for experiments And we noticed that there are certain websites you can build where chat GPT will just crawl it.
  • 00:33:44Long after Google has given up, um... You still get thousands of ChatGPT crawls every single day and We just wanted to test what is the limitation of this?
  • 00:33:56So uh It was honestly not a big experiment That we planned for long time or anything.
  • 00:34:05Then MetaHan started it and the goal is to see how much that GPT will crawl this domain.
  • 00:34:14And which other LLMs would crawl, how search engines would crawl it?
  • 00:34:18How much cost did actually creates for a person hosting all of them?
  • 00:34:22because there's always this debate like hey LLM are crawling your website It's very expensive and doesn't matter for smaller websites.
  • 00:34:30you just put on Cloudflare and pay basically nothing.
  • 00:34:35But there are websites that are not on cloudflare.
  • 00:34:37There are websites That Dynamically have to render every request, they're still websites.
  • 00:34:43it don't use caching.
  • 00:34:44so It's just a little test how much chachapity is scrolling at website and how deep it will crawl And so on and so on.
  • 00:34:53Do you think he would only be able to answer?
  • 00:34:57Like if Chachapiti crawls the side when for example Google has as you said yourself already abandoned or just like doesn't see it as adding any additional value to its index.
  • 00:35:10Do you also think that you can answer questions like if chat GBT is building up its own index, et cetera?
  • 00:35:17How the crawlers are behaving?
  • 00:35:19so do you think you could generate other interesting insights from this project?
  • 00:35:27because I mean obviously everybody.
  • 00:35:34It's quite an interesting project that I have not seen anybody do in that transparency at least.
  • 00:35:43So from that project, i don't think we can make these conclusions?
  • 00:35:47Um...I'm very convinced that OpenAI is building their own index.
  • 00:35:51um..i mean they seem to have four different grounding services that they use and some of them seemed to be Google scraping.
  • 00:36:02Atleast one seems to be Bing either officially or also in a form of scraping, but scraping Bing is actually super difficult.
  • 00:36:09And most people who scrape Bing go via Yahoo because they have the same results and scrapping Bing it's difficult because there are strong scraper protection that even send you fake result instead just blocking you.
  • 00:36:26so like one these seems to be an own index.
  • 00:36:30There are a lot of people that are convinced this is an own index for high-trust, high priority domains.
  • 00:36:38I'm actually not sure if it's true like i've seen.
  • 00:36:41no proofs just theories and we're still exploring the topic.
  • 00:36:48some things have been published on LinkedIn aren't necessarily true or at least they are not provable.
  • 00:36:53They are just theories.
  • 00:36:55So yeah definitely Chatchapiti is building their own index.
  • 00:36:59How much it is used I cannot answer yet at this time.
  • 00:37:03Okay, but if It seems to be one of their grounding options
  • 00:37:08One of the topics i wanted To get your opinion on Is also something that Happened in May?
  • 00:37:15It was The Google IO Event Where they Also introduced Their.
  • 00:37:23I mean, they claimed also that Google is now AI search which i felt interesting.
  • 00:37:31Want to get your opinion on that obviously.
  • 00:37:33but the thing what was more interested in how you think about this new idea of Search Agents where as Im quoting google's blog post here We're entering the era of search agents where you can easily create, customize and manage multiple AI agents for many tasks right in search.
  • 00:37:54Starting with information agents operating in the background to twenty four seven.
  • 00:37:58these agents intelligence reason across information to find exactly what you need at exactly that moment.
  • 00:38:03so
  • 00:38:04this obviously seems
  • 00:38:06as
  • 00:38:07most off the times with corporate press releases and keynotes
  • 00:38:12seams
  • 00:38:13super big, the next big thing.
  • 00:38:15Big innovation?
  • 00:38:17What do you think?
  • 00:38:18what's your take on search agents and their potential relevancy for the future of Search?
  • 00:38:24I Think if Google does them well they can become super relevant Because google already has all my contacts right.
  • 00:38:32They have to have an email with a calendar.
  • 00:38:35So If i'm locked into My google account And after search bar that i Can use for quick information but also to search through my personal context.
  • 00:38:45And then, for complex chats with AI mode and also to start agents that are scheduled other than Claude who is never running on their own machine or always built to run in the cloud... That's very powerful!
  • 00:39:01So if it works?
  • 00:39:04This could be something else that makes me use Claude as an example in a long time.
  • 00:39:10I also was thinking of something that happened at twenty eighteen when Google announced duplex where they stood on stage and they called a hair salon to make an appointment in the restaurant, to make reservation.
  • 00:39:24And this would soon come everywhere!
  • 00:39:26And i think it was rolled out.
  • 00:39:28two pixel phones?
  • 00:39:30In the US...I don't know if it ever rolls globally..i dont'know what the adoption feature is but kind-of like They never again talked about it, like it went away.
  • 00:39:40It wasn't this big thing and if it was worked or adopted that would be a very-very big thing.
  • 00:39:46but I still believe with what Google is doing there they have the potential since most use search bar in the world to funnel more people into AI mode.
  • 00:39:59then also these agent experiences.
  • 00:40:02i mean they want you to wipe code directly from google search interface.
  • 00:40:07and a behavior that I noticed about myself, i used to use chat gpt app on my computer for like short questions instead of clod.
  • 00:40:19But now I've gone back to Google because AI overviews have actually become so good?
  • 00:40:25That is simple question.
  • 00:40:27it's quicker to open google and write.
  • 00:40:29then you click in clod on the simpler model.
  • 00:40:35If I look at some usage data, AI overviews are the number one LLM-based answer engine globally.
  • 00:40:44I'm very sure.
  • 00:40:45and then JetGPT is maybe half of their size.
  • 00:40:50Gemini is probably half the size of JetGPPT.
  • 00:40:54And there's AI mode, Claude, DeepSeq where especially Claude it super hard to measure because so many power users just using the app.
  • 00:41:04And after that it's then Grog Rufo's perplexity co-pilot.
  • 00:41:09I don't know how long Google will keep this AI overview, AI mode, Gemini separation?
  • 00:41:18It doesn't really make too much sense anymore.
  • 00:41:20if i now can also vibe code an app from my google search bar and like... at some point they need to merge Gemini and AI mode otherwise its just Not clear anymore.
  • 00:41:33What is the difference between the Gemini interface and AI mode?
  • 00:41:39But yeah, I think Google has the potential to make these agents very very relevant.
  • 00:41:46but many people who would be early adopters of this are already using Chatchi PTO Claude or Already.
  • 00:41:55beyond that it have their local deep-seek model running or something.
  • 00:41:59I don't know.
  • 00:42:00if Google will try to win over these early adopters and power users, which would be difficult.
  • 00:42:07Or they have to simplify it so much that... It's not even the late adopters but like in the middle of adopters.
  • 00:42:17So i'm not sure how they'll play that But I believe it can be super powerful.
  • 00:42:23When I looked at the little visualization that Google has on their side, they have this example where somebody types in keep me updated when any of my favorite athletes announced sneaker collabs or signature drops.
  • 00:42:36It's a bit like it is mixed with newsletters but then also social feed linked-in feed and maybe Instagram feed because in these examples sneaker drops wouldn't necessarily happen on LinkedIn.
  • 00:42:54So if you think about LinkedIn, for example or X
  • 00:42:58obviously
  • 00:42:59a lot of people say yeah it's becoming more AI slop.
  • 00:43:04everybody can post.
  • 00:43:05It is not the highest quality.
  • 00:43:06but they have also perfected this algorithm Gone above or beyond and try to perfect it.
  • 00:43:17where you passively give the platform a signal of interest like if You Like If You Stay Have A High Dwell Time.
  • 00:43:23If You Comment It's The Signal That Is Produced.
  • 00:43:26So, It Doesn't Need A Lot Of Effort For You To In An Ideal World Create A Curated Feed that You Like.
  • 00:43:34Now This Search Agents.
  • 00:43:35They Obviously Need Compared To That A Lot Off Effort You have to Type in.
  • 00:43:41What Do You Need When You Wanted Um, et cetera.
  • 00:43:45So do you feel like?
  • 00:43:48Long context easy question Do you feel that people might potentially be too lazy to use a feature like that?
  • 00:43:55so I have seen it less of a feat.
  • 00:43:58i envisioned more off Google alerts Like I set a real current task.
  • 00:44:06um That's not necessarily just alerting me right.
  • 00:44:08It can also be an agent that Does something regularly And I don't think that for everything, the feed is the right format.
  • 00:44:19Because there can also be agents.
  • 00:44:22just write something to some notebook or update something via an MCP on API of write-something in a Google Sheet and look at it every couple weeks.
  • 00:44:38so i'm not convinced by the feed as interface with all these agents.
  • 00:44:44I do believe that Google tried to build social networks multiple times and failed.
  • 00:44:54Maybe now this will be the one where they can actually win, And also OpenAI had this idea of a social network primarily based on Sora content.
  • 00:45:05i think That kind went nowhere.
  • 00:45:08So there probably is an argument LLM providers and social network feeds, and new personalized newsfeeds will merge into one product.
  • 00:45:21And also Google already ten years ago said it shouldn't be a search engine.
  • 00:45:26It should anticipate what you want and give it to You!
  • 00:45:31That would also be very much in line with What you just described.
  • 00:45:34before.
  • 00:45:35even thinking about the drop of a sneaker call-up Just by knowing that I by expensive limited edition sneakers.
  • 00:45:44And I like artists because they went to their concert, there could be something in my Gmail inbox that tells me hey this just dropped but why should it be free?
  • 00:45:55Why shouldn't it be an ad right?
  • 00:45:57Because Google also built a system where the people who make these sneakers can advertise to me.
  • 00:46:04so we will have to see how this looks like.
  • 00:46:08Hmm, I
  • 00:46:10just sometimes feel like last note when i look at this.
  • 00:46:14I Just question myself Our attention span Is somehow a zero sum game?
  • 00:46:24So we will not gain more Attention or We Will Not Be Able to Consume More and More Content Because The Days Are Not Getting Longer.
  • 00:46:32so Somehow Probably I mean with AI Search.
  • 00:46:35Now Somehow the cake has just gotten bigger, but most people also expected it
  • 00:46:41to be
  • 00:46:42a shift between usage like from Google to chat GPT From Chatcha BT to Claude and then just with this feature.
  • 00:46:48I feel Like okay will that beer?
  • 00:46:50Will People really keep using newsletters And the X feed and The LinkedIn feed and Then the search agents and then their Claude routines and then the chachi PT Whatever It's called there.
  • 00:47:02That gives you information.
  • 00:47:04This is Just Something Where i'm cautious if a feature like that has to win over some market shares from other forms of information consumption.
  • 00:47:19Last topic for today, you guys have
  • 00:47:23put
  • 00:47:23out another research paper again on SSRN which is titled Prompt Tracking Works.
  • 00:47:30It's a good start of the title.
  • 00:47:33Promptracking works but not as one prompt measurement system, prompt wording and brand visibility in AI answer engines.
  • 00:47:42now As you know people are lazy And most of them don't have that time Maybe also not willing to read the full paper.
  • 00:47:52Can You give us quick summary Of what you guys have analyzed there?
  • 00:47:57Why it is significant?
  • 00:47:59Yeah, so most of the work has been done by Jan Allen Spiel from my team and What we did is actually it's two separate studies that in the end.
  • 00:48:08We just decided to publish together.
  • 00:48:10You maybe remember that about a year ago ran fishkin did this experiment where he asked people Tell me how you would prompt if your looking for new pair of?
  • 00:48:22I Forgot was it was because we got some but i think It wasn't.
  • 00:48:25noise can chasing headphones And one other thing like buying software.
  • 00:48:29And then he had this data set and every single prompt was unique.
  • 00:48:33Then if you run these prompt once, You also get of course different answers for every prompt.
  • 00:48:38so everybody made a decision.
  • 00:48:39Ah!
  • 00:48:40Prompt tracking is so difficult because Every prompt is unique.
  • 00:48:43Every answer's unique.
  • 00:48:45So... He very graciously allowed us to use the status set?
  • 00:48:50We ran these prompts not once nor twice but many times.
  • 00:48:54Then we realized that ninety percent of these prompts actually give you the same answer.
  • 00:49:03So if you run them dozens of times, it's the same set of brands that you get for them and ten percent were like truly unique completely different brands he would get.
  • 00:49:12so that's a first interesting insight right?
  • 00:49:14That even though we use different words in the end If the intent into context is the same and DDD like funnel stage it's the same answer.
  • 00:49:28So that is very good news for people to do.
  • 00:49:31prompt tracking and a lot of variation just comes from the fact that LLMs are not deterministic, there always some randomness.
  • 00:49:37you need to run these many times.
  • 00:49:40And then we did second thing We took a set of prompts around fifty Then modified each of them in the tiniest little bit.
  • 00:49:51There was this thing called Cosine similarity which can be used to measure difference between prompts.
  • 00:49:57And we did this again and, until the intent shifted.
  • 00:50:02Because if you change a prompt too much at some point it is different search intent then you can't compare anymore?
  • 00:50:12Then also tracked all of these variations We basically could find what are the levels in similarity where suddenly different brands were being recommended.
  • 00:50:25And also there we found that, hey, prompts can actually be worded pretty differently but still the same brands are being returned in the answer.
  • 00:50:36Then re-analyzed it a bit more and what I can share with you is this style of a prompt matters.
  • 00:50:43so if you add a word like list or toplist You get more brands than the answer then shifts over all visibility.
  • 00:50:52If you write something you are an SEO expert, which SEO tool should I buy?
  • 00:51:02You will get fewer brands because this role-play export mode seems to trigger more information answers.
  • 00:51:10Also prompts that a very top of funnel or bottom of funnel is pretty robust for changes in wording.
  • 00:51:20But prompts in the mid of the funnel, they are the most sensitive to using slightly different wordings.
  • 00:51:26So there it makes probably the most sense to have more variation off your prompts.
  • 00:51:33and then The models also act differently if you put a lot of constraints on your prompt like You don't ask for their best CIM but say It has to be open source, it has to have single sign-on.
  • 00:51:49Has to be SOC II compliant and has to integrate with Microsoft.
  • 00:51:53JGPD and Perplexity will give you fewer brand suggestions but Gemini and AI Overviews will give more brand suggestions if we add these constraints.
  • 00:52:04I think that could relate how the systems use fanouts or how they pull in information.
  • 00:52:14But that was interesting.
  • 00:52:15and what really doesn't matter is the length.
  • 00:52:18like if I have a very short prompts, Like noise cancellation headphones.
  • 00:52:24And you will have...I am looking for Noise Cancellation Headphones or i'm Looking to buy noise cancellation head phones?
  • 00:52:32blah blah blah lots of filler content.
  • 00:52:34We get The same results in the answers If we got different Answers.
  • 00:52:38Of course if You run it our personal accounts because of the context but if You were To Run these two prompts in accounts that have no context or history, you get the same brands.
  • 00:52:49So all of this is very important for prompt tracking because it means yes, prompt tracking works!
  • 00:52:56It means you don't need to have the exact phrasing and wording your customers used but still relevant to use their general vocabulary that target audience uses.
  • 00:53:08Makes a lot sense.
  • 00:53:09I feel like this obviously great finding But that you have also gained enough trust over the years.
  • 00:53:17You personally, but also PKI.
  • 00:53:19now That we feel like this is maybe Also differently to some competitors may be and not some flawed research but as actually proper methodology And proper findings um that just prove your hypothesis Uh that you had in the first place.
  • 00:53:37Um Now I want to end Today's episode with something that I want to introduce also for a series in the future.
  • 00:53:44And which is, i wanted you ask beyond what we talked about today What was Something That You Saw In The Last Hours Days Weeks Since We Last Spoke That You Found Interesting Thought Provoking Exciting That Made You Happy Made You Cry Maybe not cry but Something where you want to point people towards and say hey, here's a thought.
  • 00:54:13Here is the content piece here as opposed that I think maybe some people should look at?
  • 00:54:18I wish you had prepared me for this spontaneous question because i'm drawing it blank And don't have good answer right now.
  • 00:54:28Okay no worries Unless i do a self promotional one which would be lame.
  • 00:54:33No You can do It!
  • 00:54:35Yeah,
  • 00:54:37people should follow David from my team.
  • 00:54:41He has a couple of really interesting insights on LinkedIn just in the last couple days especially about fan out sources if you want to learn something go there.
  • 00:54:55it's where I learned.
  • 00:54:57yeah and i mean this is great don't be shy because published something today about, so we're recording on July fifteenth.
  • 00:55:09About how do LLM secretly change tracking invisible shifts in query fanarts?
  • 00:55:14And I already congratulated him to this being reposted by Lily Ray and he also got reactions where i think he was pretty happy about from Glenn Gabe!
  • 00:55:25If you don't know these names it's like... and i think Malta you can confirm Well known people in the SEO space from the US and internationally.
  • 00:55:34So I think if you follow David, You can learn a lot.
  • 00:55:37so that's definitely That
  • 00:55:43is my recommendation for today.
  • 00:55:45Awesome!
  • 00:55:45You see we don't need preparation We just need to like share with People what on our mind.
  • 00:55:52anyways Cool Malte.
  • 00:55:55thanks so much For making time Today.
  • 00:55:57It was great as always Fun fact.
  • 00:55:59I had multiple people reach out to me over the last, i don't know one two weeks that randomly just said hey.
  • 00:56:08I Just wanted to say That monthly land where format is like these are always The episodes that I liked the most because there's so much value in So little time and they want it too.
  • 00:56:20pass this on to you.
  • 00:56:21yeah It's great And I feel Like today we lift up To This Promise again.
  • 00:56:27Obviously there are sometimes these long questions between your answers that might be a little disturbing, but sorry all of you.
  • 00:56:36That I have to somehow get Malte to share his thoughts.
  • 00:56:39But yeah period.
  • 00:56:41thanks so much Malte and already looking forward to the next one
  • 00:56:45Thanks for having me and looking forward it
  • 00:56:47Bye bye.