What we found auditing 1.5k+ websites.
We crawled and scored 1.5k+ real business websites with our own audit tool between 13 August 2026 and 20 September 2026. No site is named. None were clients — most never heard from us.
Research and analysis by Rushil A. Bajpai, founder · Data collected to
The practical version, one problem at a time: the 21 SEO mistakes I see most often, and the order to fix them in.
Most of these businesses have a perfectly decent website and are nearly invisible to AI. They average 83.4 out of 100 on conventional SEO but only 68.6 on AI visibility — and out of all 1.5k+ sites, exactly 1 scored an A.
What this means if you run one of these businesses
Two things happened at the same time. People started asking AI who to use. Almost no business got ready for it. That gap is the opportunity — and it is closing.
of shoppers used AI to find a local business last year
The year before it was 6%. Count Google's own AI answers and it reaches 76%. Nobody announced this. Your customers just quietly started doing it.
BrightLocal, Local Consumer Review Survey 2026of the businesses we checked were actually ready
One site out of 1.5k+ scored an A for AI. Those same sites score 83.4 out of 100 on normal Google search. So these are not bad websites. They are good websites, built for a world that has moved on.
Our own audits, 2026-08-13 to 2026-09-20So what is actually going wrong?
have nobody but themselves saying they are good
No news story, no write-up, nothing outside their own website. Ask AI “who is good at this?” and it goes looking for other people saying so. It finds none, and names someone else instead.
never linked their own social pages to their website
Their Instagram and LinkedIn pages exist. Their website just never points at them. So AI cannot be sure it is all the same business — and when it is not sure, it stays quiet about you.
never answer a single question on their site
AI answers by copying out a short, clear sentence. Pages full of long flowery paragraphs give it nothing to copy — even when that business really is the best answer.
Is it different in your industry?
Tap one to see what we found there. These are the industries we have checked so far. If yours is not here, the honest answer is that we have not tested it yet.
Dental clinics · United States1,147 sites audited
Dental clinics · France225 sites audited
Dental clinics · Dubai78 sites audited
Dental clinics · Abu Dhabi41 sites audited
Who they shut out
AI companies send small programs to read your website. You can let them in or turn them away. Most sites turn away the ones that only collect text, and let in the ones that actually answer people — which is the sensible way round.
Shut these out and ChatGPT, Claude or Perplexity simply cannot mention you when someone asks. Most of this is security software, not a decision anyone made.
These gather text to build future AI. Saying no is a fair choice, and it costs you nothing today.
Almost nobody shuts these out on purpose, and it would be a bad idea. Counted strictly: only sites whose own robots.txt says no.
What was wrong, starting with the most common
How many of the 1.5k+ sites had each problem. Every bar points the same way, so a longer bar is always worse.
›8 more checks, and the three things nearly everyone gets right
Nearly everyone gets these right
- 100% have a secure padlock on their site
- 83% give search engines a list of their pages
- 82.6% label a few basics for computers to read
| Crawler | Told no in robots.txt | Refused by the server |
|---|---|---|
| GPTBot | 3.5% | 10.9% |
| CCBot | 3.5% | 10.9% |
| ClaudeBot | 3.5% | 9.7% |
| Bingbot * | 0.1% | 9.7% |
| Googlebot * | 0.1% | 9% |
| PerplexityBot | 2.3% | 4.9% |
| Claude-User | 2.3% | 3.8% |
| OAI-SearchBot | 0.1% | 5% |
| ChatGPT-User | 0.3% | 4.7% |
| Claude-SearchBot | 0.1% | 4.7% |
| Perplexity-User | 0.1% | 3.9% |
Google and Bing check their own crawlers by address, not by name. So a site refusing something that merely says it is Google is behaving sensibly, and that right-hand number tells you nothing about whether the real Googlebot gets in. For those two, only the robots.txt column counts — and across all 1.5k+ sites, at most 1 of them say no there. We report the other number because we measured it, not because it means anything for those two rows.
How we measured it
- We opened a handful of pages on each site — the same tool we run for paying clients, no different version.
- We asked each site eleven times, once pretending to be each AI and search company's reader, to see which ones it lets in and which it turns away.
- Before any of that, we visited each site as a normal person would. If it did not load, we threw the result away. A site that was simply down never gets counted as blocking anyone.
- Where a single request timed out or errored, we count it as neither let in nor turned away. It tells us nothing either way, so it is left out of both columns rather than quietly counted as a block.
- Each website counts once, using its most recent check.
What this does not show
- These are not random websites. They all came from lists of businesses we were thinking of contacting — dental clinics in the UAE and France, and local businesses in US cities. Nearly all of them are small. So these numbers are true of businesses like these. They are not a fact about the whole internet.
- We added a couple of checks partway through, so they only cover some of the sites. We count those against the smaller group, not all 1.5k+ — pictures with no description (1,470 sites), titleClash (1,473 sites), extra business checks (300 sites).
- Part of the AI score depends on what other websites say about a business. You cannot see that by reading the business's own site, so we leave it blank instead of guessing. That means the real picture is probably a little worse than the numbers above, not better.
Find out what AI says about you.
Send us your web address. We run our own tool on it, free, and send you what it says. The whole thing — not a taster with the useful bits taken out.
Whether AI can even read your site
ChatGPT, Claude, Perplexity, Google and Bing all send a small program to read your pages. We knock on your door as each one and see who gets let in. If they are being turned away, nothing else matters until that is fixed.
Two scores out of 100 — one for Google, one for AI
Plus what is behind each number. Where we could only make a fair guess, we say so instead of dressing it up as a measurement.
A list of fixes, best one first
Ordered by what gives you the most for the least work, so the top of the list is the quickest win. It is yours either way — use it with us or without us.
We ran it on our own site too. It told us what we were getting wrong, and we published that. A tool you will not point at yourself is not worth much.
One thing the free check does not do: actually ask ChatGPT, Perplexity and Gemini for businesses like yours, dozens of times, and count how often your name comes up. That costs us real money each time we run it, so it comes with paid work.