Your Analytics are Wrong!

In this episode of "Full Tech Ahead," host Amanda Razani interviews Josh Koenig, co-founder and SVP of Marketing at Pantheon. 

They discuss how the rapid adoption of AI agents and LLMs is fundamentally changing web traffic and search behavior. As more people "ask" AI instead of searching Google, organic website traffic is dropping, but the visitors who do click through are highly "primed" and ready to act. 

Koenig advises companies against turning their websites into chatbots; instead, they should focus on AI Engine Optimization (AEO) by ensuring lightning-fast load times, properly structured content, and strong third-party reviews. 

He also warns against the flood of bland, AI-generated content ("AI slop"), emphasizing that a unique brand voice is essential to stand out. Finally, he notes that as privacy changes make tools like Google Analytics less reliable, the future of metrics lies in server-side, full-clickstream tracking.

Key Quotes

"The majority of people are no longer searching. They're asking and they're having an AI, an agent, an LLM, sort of start their research."
"Trying to beat ChatGPT or Claude at its own game on your website is probably not smart."
"Fundamentals are what matter more than ever... having your website be fast, having your content be good, having a team that can move quickly without breaking things."

Takeaways

Traffic is Down, but Intent is Up: AI answers simple queries directly, reducing overall organic website traffic and increasing cost-per-click. However, visitors who bypass the AI to reach your site are much further along in their journey and highly motivated.

Optimize for AI Crawlers: To be cited in AI overviews, your website needs three things: high speed (AI bots won't wait for slow pages), well-structured content (using Q&A formats and clear H1/H2 tags), and a strong reputation on third-party review sites.

Avoid "AI Slop": The internet is being flooded with cheap, mediocre, AI-generated content. To succeed in marketing, you must educate and entertain by starting with a strong point of view and a distinct human voice that AI cannot replicate.

The Future of Analytics: Increased privacy settings and opt-outs are making traditional tools like Google Analytics less reliable. Businesses will need to shift to server-side (clickstream) tracking to accurately measure true human engagement and track how often AI crawlers are querying their sites.

Find Amanda Razani on LinkedIn.  https://www.linkedin.com/in/amanda-razani-990a7233/

Follow the FTA LinkedIn Page: https://www.linkedin.com/company/full-tech-ahead/

Visit the FTA website: https://fulltechahead.com/

Check out the Substack Channel: https://fulltechahead.substack.com/

Transcript

SPEAKER_00 0:19

Hello and welcome to Full Tech Ahead. I'm your host, Amanda Rosani, and today I'm excited to have Josh Koenig. He is the co-founder and senior vice president of marketing at Pantheon. How are you doing today?

SPEAKER_01 0:32

Um, great. Thanks for having me here, Amanda.

SPEAKER_00 0:34

Happy to have you on the show. Can you first share with our audience what does Pantheon do?

SPEAKER_01 0:39

Uh of course. So Pantheon is a website operations platform. And what that means is we run people's websites and we really focus on professional teams and sites that matter. So we're used by thousands of people who build the web as a profession every day, and they support tens of thousands, hundreds of thousands of websites that are collectively reaching well over a billion unique visitors every month. Today, in today's era where there's higher stakes than ever for what's going on with the internet and what you actually do with the web has a pretty big influence on whether your business or cause or uh organization will be successful or not, we help teams that need to really professionalize their practices, whether that's uh trying to work with uh teams so that you can have projects working in parallel, um, figuring out how to make releases to your website without putting them at risk, uh, or incorporating AI into your practices. Like for instance, um, we support a lot of people running WordPress websites. And as they're trying to introduce an AI as a companion to their team or as part of their development workflow, it's really important that they have version control on every change that gets made so they understand what change we made, what change was suggested by the LLM. They have a process for reviewing every change that's suggested before it gets released to the site. This is for the code of the site itself, not just for the articles or or uh or posts. And Pantheon actually does all this kind of automatically. Every website on Pantheon is under version control, and you can just connect your GitHub account and deploy directly to our platform. You can see uh a proposed change on in a preview environment automatically. Um it's all set up for you rather than having to have kind of like a DevOps guru on your team that can put all that together. Um, so that's a really exciting place to be. And, you know, in terms of everything that's going on with the web, you know, we have a statistically significant uh sample of traffic. Like we can tell what's what trends are actually happening and uh sort of what's going on with the weather on the internet. And uh, I don't know, it's a pretty interesting and exciting place to be, you know, as a as a marketing professional, being able to kind of see how the sausage gets made.

SPEAKER_00 2:45

Great. Well, so then you'll have a lot of insight on the impact of AI as far as that traffic goes. What from your experience, what are some critical things that you're noticing right now?

SPEAKER_01 3:00

Well, I think broadly, um, what what is being noticed overall is the the pattern of behavior for people to encounter information on the web has changed and is still changing, but there's been a sort of a fundamental shift in the last year. You know, the majority of people are no longer searching, they're asking. And there's uh, you know, they're having uh an AI, an agent, an LLM sort of start their research and do a bunch of work for them that they would previously have gone to Google and started sifting through all those results themselves. And so in a certain cases where it's like it's a simple answer to a simple question, you know, they they're not gonna go to anybody's website at all. They're just probably gonna get the answer and move on with their lives. Uh, if it's a more complicated uh type of process or a higher stakes question that they're trying to figure out, then you know, they're gonna still encounter the the websites that they're uh gonna be researching, but it'll be a different stage in their journey. And practically what that means in the data is you can just see, you know, kind of traffic to websites as a direct result of a search, a referral, an organic visit, uh uh, you know, what we call it, those are down, and they're down a lot, um, depending on what site you're you're looking at. I mean, you know, this has really impacted, I think, clickbait more than anything else, RIP clickbait, but uh that's not that's not the worst thing in the world. But it is it is really difficult for people who you know use organic uh SEO uh as a source of their demand. Um this has also impacted um uh the the cost uh per click on most paid platforms as well, because there's just fewer clicks to go around, which means the cost is going up uh quite a lot. And um, but you know, the other thing that's true is these visits that are coming in, these are people that are already, you know, they're already midway through their journey, perhaps, uh, before they're encountering, you know, your site, your property, which means they're primed, they're they're probably ready to go for whatever it is they're trying to figure out. And despite all the hype, OpenAI has quietly walked back their whole like let Chat GPT buy it for you thing, because it frankly didn't work, you know, despite um what people what may be possible, who can say what we where we'll be five years from now. But the reality is today, when it comes to meaningful action, people still take that action themselves. They're not, they're not, we're not at a point where you can just let an agent do everything for you. And so when those visitors are reaching your site, you know, they are ready, they're they're further along, they're ready to go, they're ready to buy, they have a question that that the agent couldn't answer for them. They're looking to get into the details to potentially do some more uh comparison or verification. And so those experiences that you can deliver when that person does arrive are actually more critical than ever before. And it kind of you know raises the stakes or or deepens the challenge of digital marketing because it's not just like throw up a flashy landing page and have the right CTA to get someone to like view a second page or so forth. It's like when someone shows up, they have a purpose and you really have to meet them where they are and and and carry that journey forward from there. So I again like this is all kind of turn things upside down in in the biz. Uh, and we're certainly not done with the with change yet, but uh we're starting to see some some pretty solid patterns emerge in terms of what works and what doesn't.

SPEAKER_00 6:20

Yeah, absolutely. I have so many questions. You shared a lot there. So let's rewind. First of all, so the customer expectation, they're expecting that aid from an AI source. So should companies take note of this and be integrating some sort of AI tools onto their website if they don't already have one?

SPEAKER_01 6:44

You know, that's a that's a a a really interesting question. And I have a little bit of a contrarian take on this, which is that trying to beat Chat GPT or Claude at its own game on your website is probably not smart, right? The the the chatbot experience, now now there is validity in having like if you have customer service or support and figuring out how to scale your team and be able to answer more questions quickly, um, like you can do a lot with those tools, and and that's very valid. But I think there's this notion that like the whole experience, just because chat GPT is here, you know, people want to chat with every website. That's not being borne out in reality. And the and the the fact is that the the leading AI labs, you know, anthropic, openai, chat, uh chat GPT, Claude, and and Google, Gemini, um, they've got massive teams that are devoted to making that product experience delightful. And that's part of why they're so popular, is because they are really well done. And replicating that quality of experience um on your own um is hard. And so I don't think like turning your website into a chat experience uh primarily is going to work very well for people. The the uh the places where I think you can leverage this technology to do better, again, customer service, this is this is becoming pretty well uh well trod. Um people who have overdone it a bit by like trying not to have people take do customer service and only have AI, and that doesn't work. Uh, but but figuring out how to give quicker responses that are more on target um and and have humans in the loop to oversee uh customer service and support clearly works. Um and the other is if someone is doing content search on your site, um, rethinking that experience um a little bit uh because the uh the audience out there is being primed by what they're getting from Google in terms of an AI overview. And that, if done right, now this is tricky because you got to put some work into thinking about designing that experience and what content you have to bring to bear, but an AI overview in search on your site uh is also something that is showing a lot of promise. The other thing is, you know, again, rather than trying to like turn your site into a chat bot, the other thing to think about though is just what the overall expectations around user experience, navigation, layout. Again, like a lot of the stuff that we do with uh with websites is based on patterns that are, you know, well over 10 years old in terms of how we think about menus and navigation and other stuff. So I think that there's a lot of room for experimentation with design. I just don't think it's a good idea to try to make your site its own, you know, chat GPT experience.

SPEAKER_00 9:22

Okay. So, and then something else you said, people are used to focusing on SEO. So that's kind of it seems like it's going out the door in favor of more AI optimization. So, what tips or advice do you have for business leaders as far as how do they make sure that they're reaching those AI optimization key points?

SPEAKER_01 9:46

Yeah, yeah, that's that is probably one of the most critical questions right now. And um, you know, disclaimer up front, TLDR, we don't really know for sure because the models all work differently and they're all changing pretty rapidly. However, people have been working on this pretty hard for a couple of years now, and again, some core underlying patterns are emerging. So the good news is, you know, we've we've had uh decades of you know having a machine audience for websites with SEO, like that's always been a part of the practice. Um, and uh a lot of the same a lot of the core underlying things that you should be doing um for SEO uh also at work and they're that's good for AI as an odd, it it's just another crawler on some level. So having your website be well structured uh and so forth is a good idea. I think things that are emerging as perhaps more important than ever is the sidebar. These are also things that are good for human visitors too. So it's not like you do like bizarre, weird things to appease the artificial intelligence, and like that's not helping you out with your regular visitors. More than more important than ever, uh the speed of the website really does appear to matter. Like highly cited sources uh in AI overviews on average are 30 to 40% faster in delivering responses than ones that aren't cited. And this is both because uh that sends us a trust signal in, just like it did with Google, having a faster website would help you rank more highly. But also a lot of times when the user experience user types in a question and the little bot is like thinking, it's actually searching the web and and and trying to scan for like current information because the model's sort of training data is inevitably out of date, right? It's from before. And so if there's a any kind of query that involves up-to-date information, the the AI is gonna try to refresh that in real time. And as like I said before, there's legions of engineers and product designers trying to make that experience for the end user as delightful and sticky as possible. They don't want to make that user wait. And so if your website is slow, like you know, Claude just basically will give up and not wait on the response. Just like humans will, though. Like, you know, if your website is slow, they're whoops, wipe, I'm done. I don't I don't that moving on. That didn't uh uh answer my question. Um, so speed is more important than ever. Uh and then the other thing that's more important than ever is structured high-quality content. Um, and this is actually where I mean, let's be honest, when people are doing SEO content, very often they're not producing the highest quality, most insightful, uh differentiated breakthrough stuff. They're trying to be relevant in the eyes of Google on a topic in general. And the truth is that is the kind of content that an uh LLM can just spit out now. Uh it's not differentiating, uh, it's not providing a distinct point of view, um, and and often you know it's just like overly long and wordy because it wants to mention a keyword as many times as possible. That type of content doesn't really help you with uh with an AI. What does help you is um structuring your content um as uh as more like a question and answer, because that does appear to have some influence in in how things get cited, uh, in just making the text really clear, like having your headings be used properly, you know, H1, H2, H3, and you know, uh building uh and if your your documents follow kind of a clear outline, that will help it be more likely to be digested. The other thing that really helps, and this is not about your site, but it has been a cornerstone of digital marketing, like I mentioned question and answer before. Um, these tools take uh review sites and what people say about stuff online in forums, like famously Reddit, but not just Reddit, they take those things, it takes those things pretty seriously. Um, and so you know, having a good presence, making sure you're looking after um how you show up in reviews, how you're discussed online in third-party sources. Also really important. If anybody asks about your company by name, you can bet that the bots will be filtering a bunch of those QA's into the results. So, you know, um that those are three things that we know for sure. There's a lot of other stuff that's out there that, you know, may matter more, may matter less. Like the whole um, you know, LLMs.txt we're up in the air as to whether that really helps or doesn't. Um, there's a bunch of stuff that like the internet giants are playing around with, but it's kind of inconclusive. But we can say for sure that having a speedy website with well-structured content and good reviews online will help you out tremendously in AI. And guess what? You need those things anyway.

SPEAKER_00 14:21

Yes. Yeah, and I think an another part of this is I'm seeing more and more people say that they're very tired of what they call AI slop. It seems to be more and more noticeable, dry, bland, different phrases that are used over and over again. Uh people are getting tired of that.

SPEAKER_01 14:40

Oh, yeah. I mean, I think the the what this technology can do with very little effort is produce replacement level, median, middle of the road copy. And, you know, uh again, like under time pressure and volume pressure, historically, great firms and wonderful people have had to crank out replacement level, medium quality uh copy, but now the cost of it has gone down to almost zero and it can be done automatically. And so what you have as a result of that is like um one of the the sources I was I'm working on uh on the uh a whole data study around this based on what we can see with our traffic and a few other things from uh specifically looking into AI optimization. And one of the sources is a study a guy did uh just to see how quickly things got picked up. But his uh his case for his study was well, I had to make a website to do this, so I had I don't remember which one he used, but he had a um uh an AI build like a 60,000 article website uh with a bunch of like tips and tricks for like how to you know do better digital marketing, I think. And uh he said, you know, it ended up costing about like $13 in token consumption, and it took about two hours. And it's just and if you look at the you know, you look at the site that he made, it's like, oh, this is really bland, it's super forgettable, but it worked as a test of how quickly do things crawl a new website. Um and and so the end at the end of the day, like bland stuff is just now ubiquitous. Like we're we're just flooded, the zone has been flooded with forgettable content. And and the thing about that is, you know, it doesn't hook anyone's attention, it doesn't have a point of view. If you're in the game of marketing, you're you have to do two things, ideally both at the same time, which is to educate and entertain, put those things together, and you've got great marketing. And you can certainly use AI to help with that. It's not saying that these should never be, this should never a tool that shouldn't be used. Obviously, this is changing, you know, many any practice that uses language is being changed by this. But you have to be very tasteful about how you use it, how you leverage it as a in your content production process. And if you don't start with a point of view, an opinion, and a voice, then what's gonna come out on the other side is just gonna be more of that noise. And so that's like that's really the challenge is like how do you create how do you create something that's worth spending time on? And then, you know, secondarily or primarily, you gotta have something that's worth spending time on, and then secondarily you've got to get people's eyeballs to it. But again, same as it ever was, right?

SPEAKER_00 17:11

Yes, absolutely. So my last question would be knowing all of this, how what tips or advice do you have for business leaders as far as how can they track metrics? You said they're dwindling now, there's less people actually coming to their sites, and people are used to they ask for this data for their media kits to show their value. And if that's dwindling, how can they track, where can they make that up?

SPEAKER_01 17:38

Well, so I think there's um there's a couple of different things that are happening. There's a third factor in the the stats that I didn't mention before because it's actually been going on for longer, but just like the pendulum is swinging back towards people having privacy online, which as an end user of the internet, I actually think is good. Uh, but the flip side of it is Google Analytics no longer reliable, right? Um, it's not gonna show you people who opt out of your consent framework. It's not gonna show you people who have their browser settings turned up or who have a plug-in for privacy. Like there's just a lot of people that are not gonna be measured uh the way that that tool measures because it requires them to opt in and consent to a rather invasive uh uh use of the data. And so I think we're gonna see it's this is not really usable yet, but you're starting to see it with like Cloudflare's put up like their first stab at an analytics framework. We're actually working on something very similar at Pantheon, and I think others are as well, where when you have part of your tech stack that can actually measure just the full click stream, like every single interaction with the site, um, and then can give you data from there, that's then a useful sort of source. And that's kind of what we look at. We can tell how much is happening with like the AI crawlers, how much is happening with stuff that like it doesn't identify as a crawler, but doesn't really look authentic either. And then how much of this traffic really does look like it's you know valid human traffic, whether or not they're letting themselves be cookied or not. Um, and so I think you know, we're that is uh kind of a like live active development work that's happening on several fronts right now. But I believe that these are gonna be the tools people will need, particularly if they're just trying to demonstrate like high-level value. Like, yes, we are actually getting people engaged with our with our content. In fact, like you're gonna be able to measure how often your web uh site was uh live queried by uh by AI. And that's a valuable metric. That means that like people are asking questions and your site is part of the response. Um, now it's not it's not the same as getting them to your site itself, but it's still that is uh a good signal, right? So all those things are just are just starting to come together now, and I think people that are that are eager to have the answers to those questions will will have more and more, you know, later this year, honestly, as these tools really do start to come together.

SPEAKER_00 19:55

Great. Well, if there was one key takeaway you could leave our audience with today, what would that be?

SPEAKER_01 20:00

I think the key takeaway is fundamentals are what matter more than ever. Uh it's easy to get spun up in the middle of this giant hype cycle that we're in and think that everything is changing and forget what you knew and so on and so forth. And it and look, everything is changing, but some things, you know, don't, right? Having your website be fast, having your content be good, having a team that can move quickly without breaking things, like that's how you keep up with wrapping in a rapidly changing environment. Um, and uh that's what we exist to help people do. So it's I'm happy to just talk about it. It's a it's a great story.

SPEAKER_00 20:34

Wonderful. Well, thank you so much for coming on the show and sharing your insights with us today.

SPEAKER_01 20:39

Yeah, thank you for having me, Amanda. This is a pleasure.

SPEAKER_00 20:41

And thank you to our audience. If you have any questions or comments, make sure to leave those and I'll try to respond as quickly as possible. Have a wonderful day.