How to check in five minutes whether AI assistants can see your site

The check takes three to five minutes and runs as a single prompt in Claude Desktop, in the Code tab. You paste your site address, Claude runs the commands itself and returns a report on seven points: robot access, sitemap, the llms.txt file, machine-readable markup, whether your text survives without JavaScript, titles and descriptions, and the technical basics. Nothing on your site changes: the prompt only reads what is already public.

The most valuable thing you learn in those five minutes sits in point one. Your robots.txt file may contain a single line that blocks the crawlers responsible for showing your pages inside ChatGPT, Claude and Perplexity answers and in search results. Site owners usually have no idea that line exists, because a contractor or a website builder template put it there. The consequence is simple: you will not be shown anywhere, no matter how many articles you write.

The prompt below is the working one, in full. Copy it, replace the address in the SITE line and send it. The rest of the article explains how to read the report and which numbers in it count as normal.

Why run this in the Code tab instead of a regular chat?

Because the Code tab runs commands on your machine and returns facts, while a chat can only reason about them. Ask a chat "is my site open to crawlers" and you get a paragraph containing the word "probably". The Code tab downloads your robots.txt with curl, opens it and quotes the lines that are actually in there.

For the same reason the prompt specifically says to count title and description length with python3. A byte count inflates the number for any non-Latin alphabet, and you end up rewriting titles that were fine. Do not delete that line.

What exactly does the prompt check?

Seven points, in plain language.

1. Robot access. robots.txt is a text file at the root of your site listing which crawlers may read which pages. The prompt checks whether the file exists at all, whether the whole site is blocked, and how search and AI crawlers are written into it.
2. Sitemap. sitemap.xml is a list of every address on your site so a crawler does not have to guess. The prompt counts the addresses and checks whether pages carry a last-modified date.
3. A calling card for AI models. The llms.txt and llms-full.txt files: a short description of what your product is and who it is for, in a format a model reads easily. Missing files are not a fault, most sites do not have them.
4. Machine-readable markup. Schema.org is a set of labels in your page code, invisible to a human, saying "this is an organization", "this is an article", "this is a price". Minimum for the home page: Organization and WebSite. For an article page: Article or BlogPosting plus BreadcrumbList.
5. Text without JavaScript. The prompt strips the scripts and shows the first 350 characters of what is left. Empty means your page is drawn by scripts, and some crawlers will see nothing instead of your text.
6. Titles and descriptions. title, description, canonical, og tags and exactly one h1, on the home page and three to five inner pages.
7. Technical basics. HTTPS, whether the http version redirects to https, how fast the site responds, whether meta viewport is present, and whether a non-existent address returns 404 rather than 200.

What prompt do I paste?

Open Claude Desktop, Code tab. Paste the text in full and replace the address in the SITE line. Nothing else needs changing.

```text Check this site for search readiness and for being cited by AI assistants.

SITE: https://put-your-address-here.com

Change nothing. Back every conclusion with a real command, never write "probably". If something fails, say so: "could not check, here is the reason".

LENGTH: count title and description in characters with python3, not with wc -c and not with ${#variable}. Byte counting inflates length for any non-Latin alphabet.

1. ROBOT ACCESS curl -s ADDRESS/robots.txt Does the file exist, is the whole site blocked, is there a sitemap link. Then the AI crawlers:

GROUP A, they show the site in answers, never block: OAI-SearchBot, Claude-SearchBot, PerplexityBot, YandexBot GROUP B, they train on content, owner's decision: GPTBot, ClaudeBot, Google-Extended GROUP C, they arrive on a human request: ChatGPT-User, Claude-User, Perplexity-User ChatGPT-User and Perplexity-User may ignore robots.txt, Claude-User follows the rules. OAI-AdsBot also checks pages submitted as ads in ChatGPT.

State it plainly: is anyone from group A blocked (that is an emergency); are crawlers listed by name or covered by a blanket allow.

2. SITEMAP curl -s ADDRESS/sitemap.xml | grep -c "<loc>" How many addresses, do they all carry a modified date, is hreflang present if the site has several languages.

3. CALLING CARD FOR AI MODELS llms.txt and llms-full.txt. If present, show the first 15 lines: is it clear what the product is and who needs it.

4. MACHINE-READABLE MARKUP curl -s ADDRESS | grep -oE '"@type":"[^"]*"' | sort | uniq -c Home page and an article page. Minimum: Organization, WebSite. Desirable: Course or Product with Offer, FAQPage. On an article: Article or BlogPosting and BreadcrumbList.

5. TEXT WITHOUT JAVASCRIPT Strip script and style, remove tags, show the first 350 characters of text. Empty means the site is drawn by scripts and crawlers will see nothing.

6. TITLES AND DESCRIPTIONS Home page and 3-5 inner pages: title up to 60 characters, description up to 160, canonical, og:title, og:description, og:image, exactly one h1. Count length with python3.

7. TECHNICAL BASICS HTTPS, does http redirect to https, how fast the site responds (curl -s -o /dev/null -w "%{time_total}" ADDRESS), is meta viewport present, does a non-existent address return 404 rather than 200.

REPORT First line: how many of the seven points are fine and what to fix first. For each point: verdict, finding with numbers, one sentence on why it matters, what to do. At the end, a fix list by priority, starting with whatever keeps the site invisible. Plain language, as for a business owner, no jargon. No praise. If you could not check something, say so. ```

You do not type any of these commands, Claude runs them. No site of your own yet? Take one from your niche: the exercise still works, and you learn to read the report on a live example.

Which three lines of the report do I read first?

The report is long, but the decision comes down to three places.

Point 1, group A. The crawlers that show you in answers and in search: OAI-SearchBot, Claude-SearchBot, PerplexityBot, YandexBot. If even one of them is blocked, the rest of the report can wait. A blocked GPTBot, ClaudeBot or Google-Extended does not belong in that bucket: those collect text for model training, and they get blocked on purpose. Who is who, what breaks and what a working file looks like is covered separately: https://academy.agineai.com/blog/robots-txt-dlya-neyrosetey

Point 5, the first 350 characters. Empty means the page is assembled by scripts in the browser, and some crawlers will see blank space instead of your text. Not an evening's work to fix, but worth knowing before you write more articles.

Point 7, the code for a non-existent address. It should be 404. If it comes back 200, your site answers "this page exists" to any junk arriving through broken links, and search engines dutifully drag that junk into the index.

The crawler names in the prompt were checked against OpenAI, Anthropic and Perplexity documentation on 10 August 2026. Companies rename them quietly, so re-read the list once a quarter.

Which numbers in the report count as normal?

• title up to 60 characters, description up to 160. A practical guideline at which the line usually fits in full. Google publishes no official character limit: the line is truncated by pixel width, so it gets cut earlier on a phone than on a wide screen.
• Exactly one h1 per page. Zero or three means the template needs rewriting.
• A sitemap up to 50,000 addresses and up to 50 MB uncompressed. Those are the limits in Google's sitemap documentation: a larger file has to be split into several. Google separately notes that at roughly 500 pages or fewer a sitemap is not required at all, as long as the pages are linked from inside the site. For scale: our own sitemap on academy.agineai.com holds 184 addresses, blog included.
• A dash next to llms.txt. Normal, not an error.
• Hreflang. Only needed if the site has two languages or more.

What do I do with the report the next day?

You do not fix everything at once. The report is a map: it shows exactly where the crawler trips. The order goes like this.

Group A in robots.txt first, because nothing else matters until that is open. Then empty text without JavaScript, if point 5 came back blank. Then titles and descriptions. Then sitemap and markup. Technical details last.

Make the edits on a local copy, not on the live site. A local copy is the same site running on your own machine, usually at localhost:3000. Visitors keep seeing the old version until you deploy it yourself. If your site sits on a website builder such as Wix or Squarespace, you will not get the files: edit titles, descriptions, robots.txt and the sitemap in the SEO section of the settings. The remaining points are unavailable there, and that is fine.

When do the fixes start showing results?

There is no fast result, and the honest timings are these.

Google's documentation says crawling a new or updated page "can take anywhere from a few days to a few weeks". The same page states that requesting a recrawl multiple times for the same URL will not get it crawled any faster, while your submission quota is spent anyway. Yandex publishes no fixed figure: a page enters search after the next crawl, and Webmaster data refreshes with a delay of several days.

First mentions inside AI answers show up after roughly one and a half to two months of regular publishing. That comes from our own experience running the Academy, not from a document, so treat it as a ballpark.

A month later, run the same prompt again and compare the two reports. It shows immediately what the crawler has picked up, and what you fixed in a place that was never broken.

A detailed walkthrough with ready-made prompts lives in the Echo block of the Academy, three lessons are open for free: https://academy.agineai.com/block/b12-eho