findable-audit report

Verifies that /robots.txt exists and is served as text/plain rather than an HTML fallback page. It is the first file search and AI crawlers request to learn what they may access, so a missing or misserved one leaves them without an explicit crawl policy or a pointer to your sitemap.

Checks that robots.txt is syntactically well-formed — plain text, only recognized directives (User-agent, Allow, Disallow, Sitemap…), and no Allow/Disallow before the first User-agent line. A malformed file is interpreted unpredictably by different crawlers, silently changing what they fetch.

Verifies that robots.txt does not block the traditional search crawlers (Googlebot, Bingbot, and the * default) from the root. A site-wide disallow on these agents removes the site from classic search, which most AI answer engines still lean on.

Checks that robots.txt does not block AI crawlers, separating citation-time fetchers (a hard fail) from training-only bots (a softer warning). If a citation-time fetcher is disallowed, that assistant simply cannot read or cite your site in its live answers — the single highest-weighted check. Transparency note: robots.txt is honor-based — it only works if the crawler chooses to respect it. Neither this check nor any audit without server log access can confirm that a bot claiming to be "GPTBot" or "ClaudeBot" is authentic; only a server-side reverse-DNS lookup can do that. A green result here means the declared policy is correct, not that impersonation is impossible.

Verifies the root URL responds with HTTP 200. If the homepage errors or is unreachable, crawlers have no entry point to index and assistants have nothing to cite.

Checks the homepage for blocking robots directives (noindex/noai) in the X-Robots-Tag header or the meta robots tag. Such a directive on the homepage tells search and AI crawlers to skip the page entirely, making it invisible in results and AI answers.

Refetches the homepage and a couple of sampled pages with AI-crawler user-agents (GPTBot, ClaudeBot) and a mobile browser, then compares HTTP status, body size, title and main content against the default fetch. It surfaces cloaking or edge bot-blocking — when a CDN/WAF hands AI crawlers a blocked, redirected or stripped-down document instead of the page a browser sees.

Checks whether pages allow full text, image and video previews (max-snippet, max-image-preview, max-video-preview) instead of restricting them with nosnippet or zero limits. Generous preview directives let search results and AI answers show rich excerpts of your content, improving click-through and citation.

Checks that sampled pages are not blocked from indexing by noindex/none in the meta robots tag or the X-Robots-Tag header (and flags header/meta conflicts and nofollow). A noindexed page is invisible to both search engines and AI crawlers, so it can never be found or cited.

Checks that /llms.txt exists, is served as plain text, and is well-structured (an H1 title, a one-line summary, a "## section", and at least 5 descriptive same-origin links). This file is a curated map that helps AI answer engines discover and understand your most important pages. Honest caveat: llms.txt is a signal of unproven value — large 2025-26 studies (Ahrefs 137K sites, SE Ranking 300K, Otterly 62K, Trakkr 37.9K) measured no citation gain, adoption sits around 3.2%, and Google says it has zero ranking impact. It keeps a modest weight as a cheap bet; the value lies in the combined check set, not in this file alone. Since 2026, Chrome Lighthouse audits this file under its own "Agentic Browsing" category, and its 2026 sample found 96.8% of sites still have none — so a passing check here is a real, if modest, edge over most of the web.

Checks that /llms-full.txt exists as plain text with substantial content (about 2000+ words under at least 2 headings). It gives AI crawlers the full text of your key pages in one clean file so they can quote and cite you accurately. Like llms.txt, it is a low-cost signal of unproven value — no large study has measured a citation gain from it — hence its small weight here.

Verifies each sampled page exposes at least 200 characters of visible text without running JavaScript. AI crawlers generally don't execute JS, so content that only appears after client-side rendering is invisible to them.

Inspects the raw HTML of each sampled page for an empty client-side mount root (#root, #__next, #app, <app-root>…) paired with almost no server-rendered text. Such pages only fill in once JavaScript runs, so AI crawlers — which generally don't execute JS — see a blank shell. Genuine SSR/SSG output that merely also ships framework markup is not penalized.

Checks that each page's main content meets a word-count threshold (≥300 words for articles, ≥150 for other pages). Thin pages give search and AI engines little substance to extract or cite.

Checks that each page opens with a concise direct answer or definition (≈40–320 characters), or a TL;DR block, right after the H1. Answer engines favor content that states the key point up front rather than burying it.

On long pages (≥300 words), checks that at least one H2/H3 is phrased as a question (starts with what/how/why… or ends with "?"). Question-style headings mirror how people query AI assistants and make passages easy to lift as answers.

On substantial pages (≥150 words), checks for lists (<ul>/<ol>) or data tables with <th> headers in the main content. Structured content is far easier for AI engines to parse and reuse than an unbroken wall of prose.

On substantial pages, checks for at least one external, non-social outbound link in the main content. Citing authoritative primary sources builds the credibility that AI answer engines reward when choosing what to cite.

Compares the main text of sampled pages using shingle/Jaccard similarity to flag near-duplicate bodies (≥0.8 overlap). Duplicate content dilutes ranking and confuses AI engines about which URL to trust and cite.

Checks that reachable /about and /contact pages exist and that a contact method (tel, email, or ContactPoint) is exposed. These trust signals help users and AI engines confirm that a real, accountable organization stands behind the site.

Checks whether the site publishes a /.well-known/ai.json discovery manifest — an emerging convention letting AI agents find machine-readable facts about the site (name, description, contact, policies). Advisory: absence only warns, and a 200 answer must be a real JSON object rather than an HTML SPA fallback.

Cross-checks the three freshness signals a page can emit — the HTTP Last-Modified header, the JSON-LD dateModified (or article:modified_time), and the sitemap <lastmod> — and flags when they contradict each other or claim a future date. Contradictory or fake freshness makes engines distrust and ignore the signal entirely.

On substantial pages (≥150 words), counts hedging phrases (maybe, it seems, peut-être, il semble…) in the first two paragraphs. Generative engines preferentially quote confident, committed statements, so an evasive lead lowers the odds of being cited.

On pillar pages (≥300 words), checks for at least one "answer unit": a short, self-contained statement (8–40 words) carrying a number, date or named entity, opening without an anaphora and hedging nothing. These are the passages a generative engine can lift and quote verbatim.

On substantial pages (≥150 words), flags DOM structures that fall apart when a page is split into retrieval chunks: long tables without header cells, FAQ answers detached from their question by decorative markup, and lists orphaned from any title. A retrieval chunk loses the context outside it, so these boundaries strand meaning.

Cuts pillar pages (≥300 words) into ~512-token windows the way a retrieval pipeline would, then measures how many of those windows still stand on their own: each needs a topic anchor (a number or a named entity, counting the heading trail a retriever prepends) and an opening that does not point back at the previous window. An engine is handed one window, never the whole page.

Looks for text an assistant ingests but a visitor never sees: copy hidden through inline styles or the hidden attribute, model-directed instructions concealed inside it, and user-contributed links that claim no rel="ugc". Hidden instructions are a prompt-injection payload, and unattributed user links let third parties speak in the site's voice.

Asks whether an agent can ACT on the site, not just read it: every form must have a submit control that is not disabled, an action that is not JavaScript-only, and a name on every field, and the site must expose at least one machine-readable way to reach a human (mailto:, tel:, a submittable form, or email/telephone/contactPoint in the JSON-LD). An assistant acting on a visitor's behalf cannot run click handlers.

Checks that the homepage contains at least one valid JSON-LD block. Structured data lets search engines and AI answer engines reliably identify your business, content, and key facts instead of guessing from raw text.

Confirms the JSON-LD declares a meaningful entity (Organization, LocalBusiness, or Article) and, for businesses, includes name, address, and telephone. A clear entity is what AI assistants cite when they name and describe your site.

Checks how many sampled pages include valid JSON-LD structured data. Structured data helps search engines and AI assistants understand what a page is about and makes it eligible for rich results and citations.

Checks that the homepage exposes complete Open Graph tags, including an absolute https image. These control how the site’s links look when shared or cited on social platforms and by AI assistants, boosting click-through and recognition.

Verifies the page declares a Twitter Card (or a complete Open Graph fallback with title, description, and an absolute HTTPS image). This controls how links to your page render as rich previews when shared, improving click-through and how AI and social tools summarize the page.

Validates that every JSON-LD block parses without errors and has a schema.org @context plus an explicit @type on each node. Malformed structured data is silently ignored by crawlers, so a single syntax slip can waste all your markup.

Checks that an Organization or LocalBusiness node carries name, url, an absolute HTTPS logo, and sameAs profile links. These fields feed knowledge panels and let AI engines attach a verified identity and logo to your brand.

Detects FAQ-style content on the page (schema question/answer pairs, expandable details blocks, or question-shaped headings) and flags it when it isn't backed by FAQPage or QAPage structured data. Marking FAQs as schema makes each question and answer machine-readable, so search and AI assistants can surface direct answers.

On interior (non-homepage) pages, checks for a valid BreadcrumbList with contiguously ordered positions and names, or a visible breadcrumb navigation element. Breadcrumbs express your site's hierarchy, improving rich results and helping search and AI engines understand where each page sits.

Checks that every @id reference in the JSON-LD graph points to a node that exists and that no @id is declared twice. Broken or duplicated references confuse parsers and can make them drop entities from your structured-data graph.

Compares key JSON-LD values (name, headline, price, rating) against the visible page text to catch markup that describes hidden or absent content. Structured data that contradicts the page can be treated as spam and get your rich results demoted or removed.

Builds the JSON-LD entity graph across the sampled pages and checks it is coherent: every @id referenced is defined (no dangling references) and the site's core identity entities (Organization, WebSite, Person, LocalBusiness) are linked into one connected graph. A clean, connected entity graph is how AI engines resolve who you are and tie your pages, brand and authors together.

Checks that the site publishes a valid XML sitemap (discovered via robots.txt Sitemap lines or common paths) with a urlset/sitemapindex root and at least one <loc>, and that it is referenced in robots.txt. A sitemap lets search crawlers and AI answer engines discover every important page instead of only the ones they reach by following links.

Verifies each sampled page declares a single, absolute canonical URL pointing to itself on the same origin, rather than a blanket canonical to the homepage. This tells search engines and AI crawlers which URL is authoritative and prevents duplicate-content dilution.

Follows every internal link found on the sampled pages and flags any that return an error status (>= 400) or fail to load. Dead links waste crawler budget and stop search and AI engines from reaching parts of your site.

For each declared hreflang alternate, verifies that it loads (200) and links reciprocally back to the page that referenced it. Correct, reciprocal hreflang tells search and answer engines which language or region version to serve, preventing the wrong variant from surfacing.

Fetches every declared <link rel="canonical"> target and verifies it returns HTTP 200, is indexable (not noindex), and does not itself redirect. A canonical pointing at a broken, noindexed, or redirecting page confuses search engines and AI crawlers about which URL to index.

Checks that only one host variant (www or apex) serves the site with a 200 while the other returns a single permanent 301 to it, with no redirect chains or loops. If both hosts serve content, search engines and AI crawlers see duplicate sites and split their ranking signals.

For each sampled path, fetches the opposite trailing-slash form and flags it when that variant also returns 200 (duplicate content at two URLs). Serving the same page at both /page and /page/ splits ranking signals between the two URLs.

Follows the redirect chain of the homepage and sampled URLs, flagging multi-hop chains, loops, or temporary redirects used for permanent moves. Extra hops slow crawlers, waste crawl budget, and can leave a page unindexed.

Requests a random non-existent path and checks the server returns a real 404/410 status instead of a 200 or a redirect to the homepage (a soft 404). Search engines rely on the HTTP status to drop missing pages; a soft 404 lets empty routes get indexed as real content.

Fetches a missing route and checks the returned 404 page offers a way back — internal links, a nav, or a search box — rather than being a dead end. A helpful 404 keeps visitors on the site and gives crawlers links to follow.

Inspects sampled and linked URLs for readability issues: excessive length, uppercase letters, underscores, deep nesting, or session/tracking query parameters. Clean, predictable URLs are easier for users and crawlers to parse, share, and de-duplicate.

Scans pages for <meta http-equiv="refresh"> client-side redirects. These are slow and poorly understood by crawlers, which may not pass ranking signals or follow them at all — a server-side 301 is the reliable equivalent.

On pages that use hreflang, validates the BCP-47 language codes and checks for an x-default alternate, a self-referencing entry, and absolute URLs. Correct hreflang tells search engines which language or region version to serve; gaps make the wrong version surface or the whole set be ignored.

Builds an internal link graph across sampled pages and flags pages with no outgoing internal links, orphans (nothing links to them), or pages more than 3 clicks from the homepage. Orphaned or deeply buried pages are rarely discovered, crawled, or ranked.

Builds the internal link graph over the sampled pages, then computes each page's in-degree and a sample-scoped PageRank to name the top-ranked pages plus any orphans (nothing links to them) or dead-ends (no internal outlinks). Where internal-linking flags depth and underlinking, this maps how link equity is actually distributed, exposing the winners and losers.

Counts navigation links and flags pages where too many require JavaScript to work (href="#", javascript:, or no href). AI answer-engine crawlers (GPTBot, ClaudeBot, PerplexityBot) and Google's first pass do not run JavaScript, so JS-only navigation hides those pages from them.

Checks that sitemap entries carry real, valid ISO-8601, non-future and varied <lastmod> dates rather than none, garbage, or one shared build date. Accurate last-modified dates tell crawlers which pages actually changed so they recrawl updated content promptly.

Samples sitemap URLs and confirms each is same-origin, HTTPS, returns 200 without redirecting, is not noindexed, and is self-canonical. Listing only clean, final, indexable URLs stops crawlers from wasting budget on redirects, dead links, or pages that should not be indexed.

Cross-references the sitemap against the site's internal link graph, flagging sitemap URLs that nothing links to and internally-linked pages absent from the sitemap. Key pages should be both listed and internally linked so crawlers and AI engines discover and trust them.

Checks that the homepage has a `<title>` and a meta description of the right length. These are the headline and summary that search engines and AI answer engines show and rely on to understand what the page is about.

Verifies that every sampled page has a distinct <title> and meta description. Duplicate titles or descriptions make pages indistinguishable in search results and AI citations, which weakens relevance.

Verifies each sampled page has a <title> of 10-70 characters and a meta description of 50-160 characters. These are what search engines and AI answer engines display and use to understand a page, so well-sized, unique ones improve click-through and how accurately your page is summarized.

Checks that the homepage <title> leads with the topic and places the brand after a separator. A topic-first title tells search engines and AI assistants what the page is about before the brand name, improving relevance in results.

Checks that the homepage <title> and <h1> share meaningful words once the brand is excluded. When the title and main heading agree on the subject, search engines and AI models receive a consistent signal about the page topic.

Verifies each sampled page has exactly one <h1> and never skips a heading level. A clean heading hierarchy is how crawlers and AI models parse page structure and pull out sections to cite.

Measures the share of internal links that use generic text like "click here", "read more", or a bare URL. Descriptive anchor text tells search engines and AI what the linked page is about, helping both navigation and understanding.

Checks that the homepage declares a character encoding, ideally UTF-8. Without a declared charset, browsers and crawlers can misread accented or non-Latin characters and corrupt your content.

Checks for a favicon and an apple-touch-icon on the homepage. These icons make the site recognizable in browser tabs, bookmarks, and on mobile home screens, reinforcing brand trust.

Estimates the reading grade level of the homepage main text using a Flesch-Kincaid formula. Simpler, shorter sentences are easier for readers and for AI models to summarize and quote accurately.

Measures the size of the HTML document, flagging anything over 100KB. Bloated HTML slows the first render and costs search and AI crawlers more to fetch and parse.

Counts scripts in the <head> that block rendering on each sampled page. Render-blocking scripts delay when the page becomes visible, hurting perceived speed and Core Web Vitals.

Counts render-blocking stylesheets in the <head>. Each one blocks the first paint until it downloads, delaying how quickly visitors and crawlers see content.

Checks what share of images across sampled pages declare an explicit width/height (or a CSS aspect-ratio). Missing dimensions make the layout jump as images load, worsening the CLS metric.

Checks that the likely-LCP hero image loads eagerly while below-the-fold images are lazy-loaded. Lazy-loading the hero delays the largest paint, while eager-loading off-screen images wastes bandwidth.

Measures what share of raster images are offered in next-gen formats like WebP or AVIF. These formats are far smaller than JPEG/PNG, so pages load faster and cost crawlers less to fetch.

Checks whether cross-origin scripts and stylesheets have a preconnect or dns-prefetch hint. These hints let the browser set up third-party connections early, shaving time off the load.

Counts the number of DOM elements and the maximum nesting depth of the homepage. A large or deeply nested DOM slows rendering and makes the page harder for browsers and crawlers to parse.

Checks whether the HTML is served with Brotli, zstd, or gzip compression. Compressing text responses cuts transfer size dramatically, speeding delivery to visitors and AI crawlers alike.

Measures the volume of inline <style> and <script> in the <head>, flagging over 14KB. Excess inline code bloats every page load and cannot be cached separately.

Measures the Lighthouse lab performance score from PageSpeed Insights, a synthetic overall grade of page speed. A fast-loading page keeps human visitors engaged and helps AI crawlers fetch and index your content reliably.

Checks Largest Contentful Paint — how long until the page's main content renders — from real-user (CrUX) field data, falling back to lab data. Slow main content hurts search rankings and can make crawlers time out before they see it.

Checks Time To First Byte — how fast your server starts responding — from real-user field data with a lab fallback. A slow server delays everything else and can make AI crawlers give up before fetching the page.

Measures Total Blocking Time from lab data — how long the main thread stays blocked and unable to respond to input — as a stand-in for interactivity when no field data exists. High blocking time means a janky, unresponsive page.

Checks First Contentful Paint (cross-checked with lab LCP) from PageSpeed Insights lab data — how quickly the first content appears — as the speed proxy when no real-user data is available. Fast first paint reassures visitors and crawlers that the page is loading.

Measures the share of <img> elements that carry an alt attribute across the sampled pages. Alt text lets screen readers and AI models understand images, improving accessibility and the semantic richness engines can extract from the page.

Checks for a mobile viewport meta tag on the homepage. Without it, pages render at desktop width on phones, hurting mobile usability, which search engines factor into ranking.

Verifies that every sampled page declares a valid BCP-47 language code in `<html lang>`. Screen readers rely on it to pronounce content correctly, and search and AI answer engines use it to serve the page to the right-language audience.

Verifies each page uses semantic landmarks — a single `<main>` plus at least two of `<header>`/`<nav>`/`<footer>`. These regions let assistive tech and AI crawlers tell the main content apart from navigation and boilerplate.

Checks that every input, select, and textarea (excluding hidden and button-type inputs) has an accessible name from a `<label>`, `aria-label`, or `aria-labelledby`. Without one, screen-reader users can't tell what a field is for and the form becomes unusable.

Verifies that links have an accessible name from their text, an `aria-label`, or a child image's alt. Nameless icon or image links are unusable with a screen reader and give search and AI crawlers no clue about where they lead.

Checks that the viewport meta doesn't block pinch-zoom via `user-scalable=no` or a `maximum-scale` below 2 (WCAG 1.4.4). Disabling zoom locks out low-vision users who need to enlarge text to read the page.

Confirms the site is served over HTTPS after following redirects. Secure transport is a baseline ranking and trust signal, and many crawlers and AI agents skip or downrank pages that are not on HTTPS.

Checks that the http:// version of the site permanently redirects (301) to https:// by inspecting the very first hop of the redirect chain. A permanent redirect consolidates ranking signals on one secure canonical origin and keeps crawlers off an insecure or duplicate version.

Checks that an HTTPS page does not load subresources (scripts, stylesheets, iframes, images or media) over insecure http://. Mixed content triggers browser security warnings and can silently block resources, weakening the trust signals search and AI answer engines rely on.

Checks for a Strict-Transport-Security header with a max-age of at least 180 days, which forces browsers to reach the site only over HTTPS. Enforced HTTPS protects visitors from downgrade attacks and reinforces the security posture that builds search and AI-engine trust.

Confirms the site sends `X-Content-Type-Options: nosniff`, which stops browsers from guessing (MIME-sniffing) a file's type instead of trusting the declared one. Blocking MIME-sniffing prevents a class of injection attacks and is a baseline security signal that trustworthy sites are expected to send.

Checks for a Content-Security-Policy that restricts which scripts, styles and connections the page may load, and warns when it still allows 'unsafe-inline' or wildcard sources. A strong CSP limits cross-site scripting and signals a well-secured, trustworthy site to search and AI engines.

Verifies the page defends against being embedded in a hostile frame, via an X-Frame-Options header or a CSP frame-ancestors directive delivered over HTTP. Anti-clickjacking protection stops attackers from tricking users through an invisible overlay of your site and is a standard security signal of a trustworthy page.

Checks for a valid, non-leaky Referrer-Policy header that controls how much of your URL is shared when visitors follow links to other sites. A privacy-preserving policy avoids leaking sensitive URLs and reflects the security hygiene that reinforces overall site trust.

Checks for a Permissions-Policy (or legacy Feature-Policy) header that disables powerful browser features such as camera, microphone and geolocation by default. Locking down unused features shrinks the attack surface and is a modern security signal of a well-maintained, trustworthy site.

Checks for /.well-known/security.txt (RFC 9116), the machine-readable address a researcher uses to report a vulnerability, and verifies that its required Contact field is present and its Expires date still in the future. An expired or contactless file signals an unattended site — a trust signal search engines and security scanners both read.