I Audited the Site I Thought Was AI-Friendly

I built this site to be AI-friendly. Semantic HTML, JSON-LD on every page, an llms.txt, two languages wired together with hreflang. The checklist was complete.

Then I sat down and verified it, item by item. Three of the four claims were partly false.

These are the notes from that audit: what was broken, how I noticed, and why I now trust a check that fails the build rather than a list.

Broken: category pages had no hreflang

The site is bilingual — Turkish at the root, English under /en/. Posts, the home page, the about page: all of them carried reciprocal hreflang.

The category pages did not. Ten of them.

The cause is simple and quiet: category slugs differ between the languages. yapay-zeka maps to ai, urun-gelistirme maps to product-building. Posts were paired through a key in their frontmatter, but no mapping had ever been defined for categories. The code never asked the question: a category page looked at its own language and never at the other one.

To a search engine, /blog/kategori/yapay-zeka/ and /en/blog/category/ai/ were not two languages of the same thing. They were two unrelated pages.

The fix was not code, it was a record: one source holding each category's slug, display name and description in both languages. hreflang is built from it now. A side benefit — category names had been rendering as raw slugs ("urun-gelistirme posts") — went away with the same change.

Broken: half the JSON-LD was missing

JSON-LD was "there". Looking inside:

• The Organization block had no sameAs. Nothing on the site connected JUKIPER to its GitHub, GitLab or X accounts, so a search engine had no reason to treat them as one entity.
• The BlogPosting block had no image, even though an OG card is generated for every post at build time. The field Google wants for article rich results sat empty.
• No page emitted a BreadcrumbList at all.

All three are the "forgot to add it" kind of bug. Nothing errors. Nothing looks broken. They are just quietly absent.

Broken: llms.txt described half the site

llms.txt is a plain-text map that introduces your site to AI agents. Mine existed and was written carefully — but it listed neither the home page, nor the categories, nor the learning series.

The map I was handing agents left out the site's spine. Categories are the backbone of this blog; a map that skips them is an incomplete map.

The surprise: the live robots.txt was not the file I wrote

Lighthouse told me this one. The SEO score was stuck at 92, and its only complaint was "robots.txt is not valid: unknown directive".

The robots.txt in my repository is four lines. The one being served was longer than thirty.

Cloudflare was the difference. Its "Managed robots.txt" feature injects a block of its own, containing a content signals policy and Disallow: / lines for crawlers like GPTBot, ClaudeBot, CCBot and Google-Extended.

The real lesson is underneath that: a training crawler and a search crawler are not the same thing. GPTBot is OpenAI's training crawler; the one that lets ChatGPT cite you is OAI-SearchBot. The same split holds for ClaudeBot and Claude-SearchBot, and for Applebot-Extended and Applebot. Only the training side was in the blocked list. The search and citation side was open, and Google's own documentation says Google-Extended does not affect inclusion in Search.

The dashboard's own numbers agreed: Googlebot and BingBot requests had gone through, while ClaudeBot and ChatGPT-User sat at zero.

Set deliberately, that is a defensible position. The problem was that I had not set it. It was on by default, and I did not know what my own site was saying.

The real point: checklists rot

All four of those had been marked done at some point. hreflang was done — then categories were added later. JSON-LD was done — then OG images arrived, and nobody went back to add the image field. llms.txt was written — then categories and a series came into existence.

A checklist is a photograph of one moment. The site keeps moving. The gap widens a little with every commit, and nobody notices, because none of it throws an error.

So I dropped the list and put a check in the build.

The check that fails the build

A script runs after the site is generated and verifies:

• every generated page is in the sitemap, and every sitemap entry was actually generated
• every page's canonical points at its own URL
• hreflang is reciprocal — if page A points to B, B points back to A — and x-default points at the right language
• if a page has no counterpart in the other language, is that deliberate? There is an explicit list in the script, and the build fails until the page is added to it
• every JSON-LD block parses, carries the required fields for its type, and has breadcrumb positions in order
• every own-origin URL inside that JSON-LD actually exists in the output — this is what catches a broken OG image path
• internal links in llms.txt and robots.txt resolve to pages that exist

If one of those fails, the build is red. Nothing deploys.

You cannot know a check works until you have watched it fail. So I broke the output on purpose: stripped hreflang from a page, deleted a page that was in the sitemap, corrupted a link in llms.txt, removed a required JSON-LD field, scrambled a breadcrumb order. It caught all six, with the right message each time.

What is left

Today: every generated page is in the sitemap, every sitemap entry is a real page, every JSON-LD block is valid. Lighthouse scores 100 across all four categories on desktop.

But that is not the real output. The real output is this: being AI-friendly is not markup you add, it is an invariant you hold. Adding it is a day's work. Holding it is every build's work.

And one more: check what your site says in production. Do not trust the file in your repository.