Three files decide most of it. None of them is your homepage.
When someone asks an assistant about your category — "best orthodontist in Jumeirah", "who fixes commercial fridges in Dubai" — the assistant does not read the internet. It retrieves a small number of chunks of text and writes an answer out of what it retrieved. Whether your business is in that answer is mostly decided before any prose is read: by three files that tell the machine what you are and what it is allowed to take.
Before anything is fetched, crawlers read /robots.txt. If that file blocks a crawler, the
conversation ends there: the assistant will never see the page, however good it is. This is the most common
self-inflicted wound we see, and it is almost always accidental — a copied file from a template, a security
plugin that blocks anything with "bot" in the name, or a wildcard someone added during a bad week.
The AI crawlers that matter now have names: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent — plus the ones that fetch live pages when a user asks a question, which run from ordinary-looking infrastructure and cannot be blocked by name.
Check it in 60 seconds: our AI crawler access checker reads your real robots.txt and shows the exact rule that allowed or blocked each crawler.
An assistant crawl has a budget. It will not read thirty pages of your site; it will read a few and move
on. /llms.txt is a plain-text file, proposed as a convention, that tells it which pages are
worth its attention and what your business is in one place.
It is not a standard and no engine is obliged to read it. But it costs nothing, it is unambiguous, and on a site with dozens of pages it is the difference between a model guessing which page matters and being told.
Typical shape — one line per thing that matters, plain markdown:
Check it: the llms.txt checker tells you whether yours exists and whether the scanner can read it. The free scan generates one for you from your own pages.
Prose is ambiguous. A page that says "we have been looking after Dubai's smiles since 2009" reads well to
a human and tells a machine almost nothing it can attach to an entity. JSON-LD is a small block of
key-value data that states it directly: this is an Organization, this is its name, this is the
address, this is the phone number, these are its opening hours, this is the same business as that social
profile.
This matters for one specific reason: retrieval works on chunks. A page gets split into pieces, and the piece that survives retrieval may be a paragraph with no business name in it. An explicit entity block is how a model connects that paragraph back to you rather than to a competitor, or to nobody.
sameAs links do quiet heavy lifting here — they connect your site to your other verified
profiles, which is how a model decides that the "Al Noor Dental" in one chunk is the same Al Noor Dental it
has seen somewhere else.
/robots.txt and confirm the AI crawlers are not blocked. Fix the rule, not the tool.llms.txt naming your most important pages and stating what the business is.sameAs.Run the free scan: it reads these three files the way a crawler does and grades all six pillars that decide whether you get recommended. 60 seconds, no card.