How Llms.txt And Robots.txt Affect AI Crawlers

Get the Basics Right Before Anything Clever Once access is confirmed, check that content actually exists for a crawler to read. Load your important pages with JavaScript disabled. If your specifications, pricing, service areas or contact details vanish, they are effectively absent from this channel regardless of how permissive your robots file is.

The output is a spreadsheet and it is the most important document in the project. It tells you whether you are named, whether what is said about you is true, who is named instead, and which pages your category's answers are actually built from.

The defensible position is to spend an hour on it if you like, and to spend the rest of the week on the things every system already reads: accessible pages, accurate Organization markup, consistent identity and content a machine can quote.

Run each one across the assistants your customers use, and write down the answers verbatim. Do this from a signed out session so your own history does not colour the result. What you want at the end is a simple table: which prompts named you, which named competitors, and which sources got cited.

Start by Finding Out Where You Stand Before changing anything, establish what assistants currently say. Write out the questions a buyer would actually ask, in their words rather than yours. Include the category question, the problem question, the comparison question and the question that names your competitors directly.

It is also worth recording the reason for every rule you keep. A disallow line with no explanation gets preserved indefinitely through migrations and redesigns because nobody dares remove something they do not understand. A one line comment saying who added it and why turns a permanent mystery into a decision that can be revisited.

What llms.txt Proposes It is a proposed convention: a file at your root offering a curated, plain text guide to your site for language model consumers, pointing at the documents you consider authoritative.

None of them are harmful. They just consume implementation and maintenance time that would achieve more if spent making the Organization markup accurate everywhere, or correcting the directory listing that has your old address on it.

Then Measure Again, and Keep Measuring A single snapshot tells you very little. Assistants vary their answers between sessions, between accounts and between model versions, so one run is a sample and not a verdict. Re-run the same prompt set on a fixed schedule and watch the trend rather than any individual answer.

The idea is reasonable and adoption is inconsistent. Support varies by provider and no major system currently treats it as required. Treat it as a cheap and speculative addition rather than a deliverable worth paying much for.

If you must change the prompt set, add new prompts as a separate cohort and keep the original series running unchanged. Editing the instrument retrospectively destroys the comparison you have been building.

What to Do About llms.txt and Similar Files Proposals for machine readable files aimed specifically at language model consumers appear periodically. Adoption is inconsistent and support varies by provider, so treat these as low cost and speculative rather than as a requirement.

Track three things over time: how often you are named, which sources get cited when you are, and which competitors appear alongside you. Movement in the second of those usually predicts movement in the first.

The Baseline Is Worth More the Earlier You Take It A baseline taken today lets you attribute change later. Without one, when something moves you will be reduced to guessing whether it was the assistants, a search update, a competitor's campaign, seasonality or your own site changes.

Log the conditions with every run, including which assistant, which mode, whether web access was enabled and the date. When a result moves sharply, the conditions log is usually what tells you whether the world changed or your setup did.

Treat markup as something with a maintenance cost rather than a one off implementation. Prices change, people leave, products are discontinued, and structured data quietly keeps asserting the old version long after the visible page has been updated. Adding a schema review to whatever process already updates your pages costs minutes and prevents the most damaging failure mode, which is confidently stating something that is no longer true.

Make Sure the Crawlers Can Actually Read You A surprising number of brands are invisible for the dullest possible reason. Their robots.txt blocks the crawlers that feed AI systems, or llm seo their content only appears after JavaScript executes, or their key pages sit behind a form.

Get Represented Accurately Off Your Own Site This is the step most brands underestimate. Assistants frequently cite review platforms, industry directories, forum threads and journalism rather than the brand itself, because independent sources read as less self interested.