Last week, I noticed Google’s AI overview appearing on both mobile and desktop search results pages.
As I’ve said before, I’m not a huge fan of this feature, and to be honest, it was pretty funny when it pulled data from Reddit scumbags, as it gave some pretty ridiculous answers.
Now that Google AI Overviews (AIO) has been launched in the UK, have the results improved and what impact does it have on SEO?
Here’s what the AI summary currently looks like:
On desktop, clicking “Show details” will expand the answer box and display referenced sources to the right. You can also click the link icon next to each section of the answer to see where the information cited comes from.
We decided to look at which queries were showing AI summaries and which sources were being cited. Was it relevant? Was it factual? And was Google favoring sites that already ranked highly?
test
Because we don’t have advanced software that can pull data from the SERPs, including whether an AI summary is shown, we had to do this by manually searching random queries to see if they offered an AI summary and checking each source, which is why we were only able to look at 50 different search queries.
As part of our testing, we also noted when the AI summary did not appear at all or frequently, and what types of prompts did and did not show the AI summary. While conducting this research, we actually performed hundreds of searches, but we did not have the time or resources to write them all down.
We found some interesting patterns across different verticals and queries.
- Recipes/Food – Very few AI summaries were shown. No AI summaries were shown for “Top 10 Vegetarian Recipes”, “Lamb Mince Recipe”, or “How to Make Pizza Dough”. It did teach me how to boil eggs, which surprised me.
- X places to visit/things to do/popular attractions – Limited AI Summary. This was also a bit of a surprise, as I expected the AI Summary to collect and summarize data for these types of queries.
- General knowledge about people – The AI summary shown for a specific person is not very extensive. It will provide information about the tallest man in the world, but not “how tall is Taylor Swift,” “how many kids does Elon Musk have,” “who is Ben Affleck married to,” or “how old was Michael Jackson when he died.” It will show up for queries like “highest earning pop star” or “first man on the moon.”
- Home/Garden/Crafts – There are a ton of AI summaries out there. There are a ton. If you’ve created high ranking content, you may start to see a significant drop in traffic.
- Health/Fitness – There are a lot of AI summaries, which is a bit unsettling to me given that Google offers a disclaimer: “This is for informational purposes only. Consult a professional for medical advice or diagnosis. Generative AI is experimental.” This could easily be overlooked. On the plus side, no AI summaries appeared when I searched for things that could be considered harmful, such as “fastest way to lose 20kg without exercise” or “how to remove toenails.”
- Finance – this was limited. It showed up for queries like “how to open a savings account” and “ISA vs bonds”, but nothing for “best bank account” or “best mortgage rates in the UK”. But these figures are constantly changing so are likely to be inaccurate, so it’s good that it didn’t show up.
- Cars – somewhat complicated. It didn’t show up for repair-related searches but did show up for other searches (‘how to change a tyre’ doesn’t show up, ‘fix a cracked windscreen’ and ‘fix a scratched alloy’ do), and it also shows up for ‘top 10 most reliable cars’ but not ‘best electric car’.
- Retail – AI summaries were scarce and ads were prominent in the majority of searches. The following queries did not show an AI summary even though they were not commercial (best washing machine under £300, best portable turntable, affordable wedding dresses, most comfortable Dr. Martens, best bikinis of 2024).
From these searches, we found that AI overviews appeared less frequently when searching queries that included words like “top” and “best.”
Next, I looked at the search queries that had AI summaries popping up. I wanted to look at three things:
- Was the data relevant?
- Was the data factually accurate?
- Are the cited sources already ranked on page one?
Here are the results:
result
Data Relevance
The sources cited were largely relevant, with some caveats.
For some queries, AIOs cited transactional sources rather than informational sources, for example citing links to product pages even though the query was clearly positioned as a question seeking an answer.
In another example, the brand name “Vets Now” was quoted in the response.
For the same query (dog choking), one of the sources cited is about the color of vomit, not choking. While certainly relevant, vomit is not mentioned anywhere in the AIO, and it doesn’t even appear on page 1, so I’m not sure why they decided to include it as a citation.
There were also concerns about the lack of reliable and authoritative sources cited, particularly when it came to health-related enquiries.
For the query “how to reduce calorie intake,” there were no citations to the NHS, CDC, or WebMD, even though these sites all appeared on page 1. Instead, AIO chose to cite fitness apparel and supplement sites.
We also found that AIOs may have some Google bias, for example, YouTube videos frequently surfaced for queries like “how should redheads shape their eyebrows” – the content of the videos wasn’t particularly helpful and some were just product reviews without any real tips or advice.
A search for “add multiple services to Google Business Profile” returned the top four results from Google, with only the first one being relevant: There were far superior sources on page one that were not cited, but they were not included.
Data Accuracy
The accuracy of the data is significantly improved from what we saw a few months ago.
The only mistake I found was in the query “how to clean vinyl records” which suggested using sandpaper for a smooth finish, quoted from a YouTube comment: “I like to clean them with sandpaper and polish, it gives a really smooth finish.”
Some of the information was also out of date: The music artist with the most monthly listeners on Spotify as of August 2024 was Billie Eilish, not The Weeknd, as AIO stated.
There was a quote on “How to Kill a Wasp” that I found a little unsettling.
We’re not doubting its effectiveness, but at first glance, using gasoline seems like a way to kill our angry striped buddies. However, if you visit the cited websites, the posts advise against ever trying to kill a hornet by blowing it up with gasoline. Killing hornets with gasoline is not mentioned anywhere in the top organic search results, and reputable sites like Gardener’s World and Good Housekeeping are not cited by the AIO. Thankfully, Reddit is not cited either. Some of the organic search results shown have a feel like an accident waiting to happen.
I spend a lot of time searching for the query “who would win in xxx”, mainly because I love winning pub arguments. I expected the query “who would win in a lion or a tiger” to give me some pretty random results, but I didn’t expect to see a leadership talk cited as a source.
Source SERP Position
In most cases, the sources cited were from sites that ranked highly on the search results page. There were a few exceptions, such as for the query “what denim styles are trending right now,” but AIO did not cite Vogue, which appeared twice on page one, or Glamour, which was number one.
Citations for the search “what to drink to detox your body overnight” were also a bit odd. They were mostly from India-based domains, such as the Times of India (twice), an Indian supplement website, and another Indian domain displaying slideshow content. In this case, very few citations were from page 1 results.
A search for “how to grow eyebrows” returned even fewer results on the first page, with just one from Healthline. The rest were beauty and holistic health sites recommending applying onion juice to your eyebrows, a first for me.
One search query in particular generated some surprising citations with only one site citing it on page 1: “How to rank #1 on Google.”
Most of the big SEO companies were on the first page of organic search results: Ahrefs, Backlinko, WordStream, and even Google. However, only one site on page one was referred to by an AIO: Hobo Web.
One of the quotes was a 2019 LinkedIn Pulse article that was quite full of grammatical errors, typos, and inaccurate information, plus it was about TV advertising and social media marketing.
Additionally, AIO has AI companies quoted, which makes you wonder if the content is 100% AI generated.
Conclusion
After completing this study myself, I found a larger and better study by seoClarity that looked at 36,000 keywords, but focused on “money” keywords. My searches were long-tail and sometimes more obscure, but the results were pretty much the same as the conclusions I came to, even if the sample size was pretty small.
Finding 1: High-ranking sites are more likely to be included in AI summaries
While there are a few outliers (possibly due to the long tail or complexity of some of the queries I tested), the majority of the AI overview is dominated by sites that are already highly ranked. This tapers off the further down the page you go in the SERPs.
The totals are as follows:
Position 1: 31
Position 2: 29
Position 3: 22
Position 4: 24
Position 5: 23
Position 6: 15
Position 7: 15
Position 8: 16
Position 9: 14
Position 10: 9
Finding 2: Cited sources generally provide factually correct information
The feature was plagued with issues when it first came out of beta in the U.S. Many of the results were taken from UGC sources like Reddit and Quora, and offered some pretty cool results, like putting glue in cheese so it sticks better to pizza, or drinking urine to pass kidney stones.
Finding 3: Small sites may get attention, but they don’t necessarily get the traffic.
For lesser known searches, smaller sites that can answer them better might have an advantage over more authoritative sites, but that depends a lot on whether someone will bother to click on the link, which let’s be honest, they probably won’t.
Finding 4: AI overviews can still lead to reduced traffic
If the AI summary answers the query sufficiently, with or without the citation, users are less likely to click on the link. Because this feature is relatively new, you should monitor the impact AIO has on your traffic and conversions, especially for sites that rank highly in organic search.
Discovery 5: Don’t stop doing what you’re already doing
Don’t worry about the “optimize for AI” talk: if you look at the results, for most queries, the AI synopsis cites a source that appears on page 1. Not all, but as seoClarity highlighted in their research, this happens 99.5% of the time.
If you want to see for yourself, here are my findings.
