Technical SEO best practices: what has to be true before a page can rank or be quoted

A robots.txt file should allow every crawler by default and disallow a path only where somebody has written down why. Disallow controls fetching, not indexing: a blocked URL can still be indexed from an external link, and because the crawler never fetches it, a noindex tag on that page is never seen.

Disallow and noindex are therefore alternatives, not belt and braces: a page that must stay out of the index stays crawlable and carries noindex, while a disallowed path may still surface as a bare listing. Our own robots.txt disallows nothing — the legal pages carry noindex, and a disallow rule we once had would have stopped the tag ever being read. The confusion with llms.txt has its own piece, llms.txt is not robots.txt.

Content that exists only after client-side JavaScript runs is content some fetchers never see. Googlebot renders JavaScript later and with limits; AI retrieval crawlers largely do not render it at all. Anything that must be indexed or quoted — copy, headings, structured data, canonical tags — belongs in the HTML the server sends.

Static generation or server rendering is the plain fix. The test is cheap: fetch a page with a tool that does not execute scripts and compare it with what a browser shows; wherever the two differ on something that matters, that content is at the mercy of a render queue. Client-side rendering remains fine for a scheduler or a calculator that nobody needs to index. This site is built that way, every page generated at build time.

Every page should have one canonical URL that returns 200, declares itself as canonical, and is the only address the site links to. Choosing www or the apex domain, and HTTPS over HTTP, is a decision made once and enforced with a permanent redirect. Where a site has several languages, each language gets its own URL.

A canonical tag that points at a URL which redirects is a contradiction: the page says the true copy lives elsewhere, and elsewhere says it has moved. The same goes for trailing slashes, letter case and query strings — pick a form, redirect the rest, and make internal links, sitemap and canonical agree. Hreflang annotations must be reciprocal and self-referencing, and redirecting visitors by IP address hides the alternatives from crawlers, which arrive from one country.

Large sites mint their own thin and duplicate pages — facets, sort orders, session parameters, near-identical variants — and each address dilutes one intent. Canonicalise, noindex or stop generating them, and point internal links at the owner rather than the variant. Crawl budget is the wrong frame for most of this: Google's own guidance on it is addressed to very large sites, and at a few thousand pages what a log-file review turns up is almost always duplication rather than budget.

An XML sitemap should list only URLs that exist, return 200, and are meant to be indexed — no redirects, no noindexed pages, no planned pages that are not yet live. The reliable way to guarantee that is to generate the sitemap from the same source that generates the pages, so the two cannot drift apart.

A hand-maintained sitemap is a list of good intentions: entries lag additions and outlive removals, and every crawler that trusted the file gets a 404. This site's sitemap is built from the content registries that build the pages — a service page is listed because, and only because, its data exists — and the noindexed legal pages are left out on the same basis. That is a practice we describe, not a result we claim.

A redirect should resolve in a single hop to a URL that returns 200, and a migration should be mapped URL by URL before anything is switched. Crawlers follow a chain only so far before giving up, and a migration without a map sends old pages to the homepage or a 404, discarding what they had earned.

Redirecting an entire old site to the new homepage is the failure we see most often, and engines treat it much as a soft 404: the destination has nothing to do with the source, so nothing is passed along. The map is tedious and there is no shortcut. Chains accumulate over successive redesigns, each layer redirecting to the last, so part of any migration is flattening the previous ones.

Core Web Vitals should be read from field data — measurements collected from real visitors — and assessed per template against Google's published thresholds. They are a genuine ranking signal and a weak one, deciding between otherwise comparable pages. The work is worth doing as a conversion project, with the ranking effect treated as a side benefit.

Search Console groups URLs that share a failing pattern, so ask which template fails on which metric, not which page. A lab run on a fast machine will pass a page that real visitors on cheaper devices experience as broken; where the two disagree, the field data counts. The heaviest cost is usually third-party scripts nobody owns as a total, so start with an inventory of every external tag. Field data accumulates slowly; put the verification date in the diary.

Structured data should describe what the visible page says and nothing more. An Offer node with no visible offer, a Review node with no reviews, a rating you cannot substantiate, or a LocalBusiness node for a city with no office are claims made to machines that the page does not make to people, and they invite manual action.

Validate the markup whenever it changes and watch the enhancement reports in Search Console afterwards, because one template edit can silently drop a property from every page at once. Link the nodes into one graph with stable identifiers — the organisation, the website, the page — rather than fragments that each describe the company differently. Our own site emits a LocalBusiness node only for the one city where it has a transcribed address, and nothing for the cities it serves remotely.

Technical SEO needs monitoring because its failures are silent: a robots.txt edit, a canonical change or a dropped rendering step produces no error, only a decline that surfaces later. Server logs show what crawlers actually fetch, index coverage shows what an engine kept, and a change log ties any drop to the deploy that caused it.

Logs are the only record of what a crawler did: which user agents arrive, where they spend their requests, and whether they reach the pages that matter or a parameter swamp. Index coverage is the other half — what was kept, what was excluded, and the stated reason. The change log is the cheapest: every deploy, template change, redirect rule and robots.txt edit gets a dated line, so that when traffic moves, what shipped that week is a lookup rather than a guess.