Answer in brief
Indexing directives are generated by build code, which means they regress like build code. A staging Disallow or a noindex left in a shared layout ships without an error and surfaces as a traffic curve six weeks later.
Three files decide whether a page exists
Before ranking is even a question, every page has to pass three gates that have nothing to do with how well it is written. The robots.txt file says whether a crawler may request the URL at all. A meta robots tag, or an X-Robots-Tag header, says whether the fetched response may be kept in the index. A sitemap says which URLs the site itself considers canonical and worth discovering quickly. A page can be excellent and fail all three.
The part teams skip is where those three things physically live. They are not settings inside a marketing dashboard. On any modern stack they are generated at build time by the same code that renders the page: a route exports its metadata object, a sitemap comes out of a function that queries the database, robots.txt is emitted differently per environment, canonical URLs are assembled from a base-URL constant. Whatever produces them can be edited, refactored or broken by a developer who has never heard the phrase crawl budget.
Which means indexing rules obey the rules of code. They have versions, branches and merge conflicts. They behave differently depending on which environment variable happened to be set. They can be changed by a dependency upgrade that alters a default, or by a helper function being moved into a shared layout. And, like any code without tests, they drift steadily toward whatever state the last hurried commit left them in.
The catalogue names this exact scope as a service rather than as advice. Sitemap, robots.txt and indexing protection in CI sits in the Indexing category at Beginner difficulty, ⏱ 2–5 hours, $80 – $300 or 5 000 – 20 000 ₽. The difficulty rating is honest and slightly uncomfortable: none of this is hard. It is simply never anybody's job in particular until the traffic is already gone.
How a section vanishes without anyone touching SEO
The first mechanism is leaked staging protection. A team blocks the staging environment with a blanket Disallow so it cannot compete with production for the same queries. That robots.txt is generated from a template, the environment is read from a variable, and one day a deploy runs with the wrong variable set or with the variable missing entirely and falling back to the safe-looking default. Production now serves a site-wide block. No error, no failed build, a perfectly valid file.
The second is a template-level noindex that escapes its cage. A developer adds noindex to one page type during development — a new faceted filter, a paginated archive, an unfinished checkout step. Months later the condition gets refactored into a shared layout component because three routes needed the same wrapper. The directive now applies to every child route under that layout. The section stays live, stays linked, stays beautiful, and is quietly excluded from the index.
The third is a sitemap that collapses. Sitemap generation queries a data source at build time and filters out drafts. Somebody renames a status field, or adds a new column that defaults to null, and the filter that was meant to exclude unpublished rows now excludes almost everything. The build succeeds. The output is well-formed XML that validates against the schema. It contains eleven URLs where it used to contain four thousand, and nothing in the pipeline considers that remarkable.
The fourth is a canonical tag pointing at the wrong host. The base URL used to build canonicals falls back to a staging domain when its environment variable is absent. Every canonical on the site now points at a host the crawler either cannot reach or, worse, can. None of these four are exotic edge cases dreamt up to make a point. All four are ordinary refactors that ship on an ordinary Tuesday afternoon and pass code review.
The failure that arrives six weeks late
A failing unit test fails now, in front of the person who caused it, with a line number. A broken indexing rule fails on the crawler's schedule instead of yours. The bot has to come back, re-read robots.txt, re-fetch the affected pages, decide the directive looks stable rather than transient, and only then drop URLs from the index. Rankings decay rather than vanish, and traffic falls in a curve that is easy to mistake for seasonality or a competitor's campaign.
By the time the loss is visible in a dashboard, the causal thread has been cut. Dozens of commits have landed since. Nobody remembers editing a robots template, because from the developer's side it was a one-line environment change in a much larger pull request. The investigation starts in analytics, moves to rank tracking, and only reaches the file that caused it if somebody thinks to diff the production robots.txt against what it looked like last quarter.
That asymmetry is the whole argument for putting the check in the pipeline. A test that runs on every pull request converts a six-week forensic exercise into a red build with a filename attached. The person who introduced the change is still holding it in their hands, still has the context loaded, and the cost of resolution at that moment is roughly one minute. The same defect discovered from a traffic report costs a quarter.
This is deliberately not a monitoring argument. Monitoring tells you the site is already deindexed and asks you to react. A CI assertion tells you the deploy would have deindexed the site, before anybody outside the team can observe it. It is the same information delivered on the other side of the release boundary, which is the only side where it is cheap.
What an indexing check asserts before a deploy is allowed through
The checks that actually work are boring and extremely specific. The robots.txt produced by the production build must not contain a blanket disallow. It must reference the sitemap URL. Most usefully, it must match a snapshot file committed in the repository, so that any change to it becomes a reviewed diff rather than an invisible side effect of something else. Deliberate changes update the snapshot and pass; accidental changes have no matching update and fail.
The sitemap gets counted and sampled rather than merely validated. Assert a minimum URL count relative to the previous successful build, so a collapse from four thousand entries to eleven becomes a failure instead of a deploy. Assert that a random sample of the listed URLs actually returns a 200 response. Assert every entry uses the canonical host and an absolute URL. Assert the file parses and stays under the size and entry limits before it is ever submitted.
Templates get scanned for directives they should not carry. Assert that no page type renders noindex unless that page type appears on an allowlist stored in the repository — checkout steps, account pages, internal search results, whatever the team actually decided. The allowlist is the real deliverable here. It converts an invisible convention into a written, reviewable statement of intent, so the question why is this page noindexed always has an answer somebody wrote down on purpose.
The catalogue scope for this work is stated plainly: ⏱ 2–5 hours, $80 – $300 or 5 000 – 20 000 ₽, Beginner difficulty, Indexing category. Its description promises the outcome in the same register — search engines see exactly the pages they need, find new ones quickly, and do not waste crawl budget on rubbish, while a site accidentally closed by a release no longer leaves the index unnoticed.
What Schema.org markup actually tells a machine
HTML describes where things sit on a page. It does not describe what they are. A price, a phone number and a product code are all just text nodes inside divs as far as the document tree is concerned, and any meaning a machine extracts from them is inference from layout that was designed for human eyes. Schema.org markup, usually written as JSON-LD inside a script tag, removes the inference step by stating the types outright.
That explicitness has two distinct consumers, and they use it differently. Search engines read it to decide whether a result is eligible for an enriched presentation — a star rating, a price with availability, breadcrumbs, an event date, a FAQ accordion under the blue link. Language models, which consume pages as text, get an unambiguous statement of which entity is being described and how it relates to the others, instead of reconstructing that from visual hierarchy they cannot see.
The catalogue description says precisely this and no more: search engines and AI models begin to clearly understand what kind of product, company, author and event you have, and rich snippets in search results along with the chance of being quoted in AI responses grow. The operative verb is grow. Not guarantee, not unlock, not rank. That distinction between eligibility and outcome is the entire content of the next section.
The scope is Schema.org micro-markup with verification in CI, Structured data category, Experienced difficulty, ⏱ 4–8 hours for 5–7 templates, $120 – $500 or 8 000 – 35 000 ₽. Note that the unit is templates, not pages. You mark up the product template once and every product on the site inherits it, which is also why a single template bug propagates across the entire catalogue in one deploy.
What markup will not do, no matter how much of it you write
It will not rank a page. Markup is a description format, not a quality signal, and adding more of it does not add more of anything else. A page carrying flawless Organization, Product and BreadcrumbList markup with nothing worth reading underneath stays exactly where it was. Teams that treat markup as a lever are genuinely surprised when the graph does not move, and that surprise is a failure of expectation-setting during the sale, not a failure of the work.
It will not fix indexing, and the ordering matters. If robots.txt blocks the URL, no crawler reads the JSON-LD at all, because the response is never requested. If a shared layout carries noindex, the markup is discarded along with the page it described. This is exactly why the cheap indexing service goes first in any sane sequence: markup on a page that cannot be indexed is a well-formed file that nobody will ever open.
It will not create eligibility that the content has not earned. Rich results carry content requirements — a review needs a real review, a recipe needs real ingredients and steps, an FAQ block needs questions genuinely answered in the visible page. Markup describing things a visitor cannot find on screen is not an optimisation. It is a mismatch between the machine-readable and human-readable versions of the same document, and mismatch is the one thing in this whole area that carries manual penalty risk.
And it will not override the page. Where the markup and the visible content disagree, the visible content is treated as ground truth and the markup is the part that gets ignored — or, if the disagreement is systematic, distrusted for the whole domain. The useful mental model is a caption on a photograph. A good caption tells you what you are looking at. A caption describing a different photograph is worse than no caption at all.
Markup rots because it is a copy of data that keeps changing
JSON-LD is a projection of your data, not a description of your page. The product template reads price, currency, availability, SKU and rating count out of the same objects the HTML uses, then re-emits them in a second format that sits in a script tag nobody looks at. Two representations of identical facts, produced by two code paths, and only one of them is inspected by a human being every single day. It is not difficult to guess which one silently stops matching.
The failure modes are dull and relentless. A currency field starts arriving as a symbol instead of an ISO code after a localisation change. An availability enum stops being InStock and becomes a translated string. A date loses its timezone offset during a serialisation refactor. An author object becomes a plain name string after a CMS migration. Every one of these produces markup that still parses perfectly as JSON and no longer validates as Schema.org, which is the worst possible combination.
This is why the catalogue puts verification into the service name rather than offering it as an optional extra afterwards. The CI step renders each marked-up template with real data, extracts the JSON-LD from the rendered output, and validates it against the required and recommended properties for its declared type. A missing required property fails the build. The check runs on every pull request for as long as the repository exists, not once during handover week.
Five to seven templates is a realistic count for most sites: home, product or service, category or listing, article, and an organization or contact page. At ⏱ 4–8 hours for 5–7 templates, the arithmetic any reader can repeat — offered as an example of how the band divides, not as a measurement of anything — lands somewhere between roughly half an hour and roughly an hour and a half per template. That is about what writing one type correctly and wiring one assertion actually takes.
AI crawlers read a different site than your browser does
There are now two audiences asking two separate access questions. Classic search bots have been negotiated with for decades and their behaviour is exhaustively documented. AI crawlers are newer and more varied: some fetch pages in bulk for training corpora, some retrieve a handful of documents at answer time, some fetch a single page because a user asked a question about that specific company. They arrive under their own user-agent strings, respect robots.txt to varying degrees, and mostly do not execute JavaScript.
That last clause is the expensive one and it is routinely discovered too late. A site rendered entirely on the client returns a shell document with an empty root div to anything that does not run a full browser engine. A human sees a complete page. A fetcher sees a heading, a loading skeleton and nothing else. No amount of markup discipline or sitemap hygiene compensates for this, because the content was never present in the response that was actually served.
The access decision itself is a business decision, and there is no single correct answer to hand every owner. Some want maximum reach into AI answers and open everything deliberately. Some want documentation readable while pricing pages stay out of bulk corpora. Some want to refuse training crawlers entirely while allowing the retrieval agent that fetches a page precisely because a customer asked about them. robots.txt can express all three positions, per user agent, and most sites currently express none of them.
llms.txt is the emerging convention on the other side of the same problem: a plain-text map at the site root that states what the site is and which page carries the canonical explanation of each topic, so a model-driven reader does not have to guess from navigation. The catalogue scope is Preparing a website for AI crawlers: robots.txt, llms.txt, rendering — AI infrastructure category, Beginner difficulty, ⏱ 4–8 hours, $150 – $450 or 10 000 – 35 000 ₽. Its description adds the part owners react to most strongly: for the first time they see who is taking what.
Doing the three in the order that stops the bleeding first
Order by dependency, not by ambition and not by price. Indexing protection comes first because everything downstream is conditional on it. Markup, AI access, content investment and internal linking all evaluate to zero on a URL that is disallowed or carries noindex. It also happens to be the cheapest of the three at $80 – $300 or 5 000 – 20 000 ₽, and the shortest at ⏱ 2–5 hours, which removes the usual excuse for postponing it.
Crawler preparation is second, because it is the other half of the same access question and it touches the same files. You are already editing robots.txt; adding per-agent rules, confirming that the templates that matter render server-side, and publishing llms.txt belongs in that same pass. The catalogue prices it at ⏱ 4–8 hours, $150 – $450 or 10 000 – 35 000 ₽. Doing it separately means opening the same file twice with two different mental models, which is how contradictory rules appear.
Markup is third and it is the only one of the three rated Experienced rather than Beginner. ⏱ 4–8 hours for 5–7 templates, $120 – $500 or 8 000 – 35 000 ₽. It is third because it is meaning work rather than access work: it presumes the pages are reachable, presumes the content genuinely exists, and presumes somebody in the room can say what each page type is supposed to be an instance of. That last presumption is the one that usually fails.
Added together — arithmetic any reader can repeat from the figures above, offered as an example and not as a quoted package price — the three bands span $350 to $1 250, or 23 000 to 90 000 ₽, across ⏱ 10 to 21 hours of work. That is a site's entire machine-readability layer expressed as one number a buyer can weigh against their own budget before a conversation starts, rather than three quotes arriving separately over three weeks.
Keeping the checks alive after the invoice is paid
A CI check that emits a warning is a check that will be ignored by the fourth week. The assertion has to fail the build and block the merge, and that is a social decision far more than a technical one. It should be made explicitly at handover with the person who owns the pipeline, because the first time a build goes red over a robots.txt diff somebody will ask to downgrade it to a warning, and the answer needs to already exist.
The committed-snapshot pattern is what makes a blocking check survivable. Rather than a rule saying robots.txt must never change, the repository holds a file describing exactly what robots.txt should be. A deliberate change updates that file in the same pull request and gets reviewed like any other diff. An accidental change arrives without the matching update and fails immediately. The check does not forbid change; it forbids unnoticed change, which is the only kind that hurts.
Ownership is the other half of survival. These files sit in the gap between marketing and engineering and belong to neither department by default, which is precisely why they rot. Naming one reviewer for changes to robots.txt, sitemap generation, canonical logic and the noindex allowlist costs nothing at all and removes the ambiguity that lets a directive ship unexamined because both sides assumed the other was watching it.
Revisit the markup when the data model changes rather than on a calendar. A new product attribute, a CMS migration, a new page type, a change to how authors are stored — each of those is a real reason to re-run validation, and each of them is invisible from a quarterly review schedule. If the validation already lives in CI, every one of those changes re-runs it automatically. That is the difference between a project that was delivered and a property the site now permanently has.
Questions and answers
How much does it cost to add Schema.org markup to a website?
The VIT MARKET catalogue lists Schema.org micro-markup with verification in CI at $120 – $500, or 8 000 – 35 000 ₽, with a stated duration of ⏱ 4–8 hours for 5–7 templates. It is rated Experienced difficulty and filed under Structured data. The unit is templates rather than pages, so one product template covers every product on the site.
Can a website deploy accidentally remove pages from Google?
Yes, and it normally happens with no error anywhere. A staging-wide disallow shipped to production, a noindex left inside a shared layout, or a sitemap query that silently returns eleven URLs instead of four thousand will all pass a build. The catalogue service Sitemap, robots.txt and indexing protection in CI exists for exactly this and is priced at $80 – $300 or 5 000 – 20 000 ₽ over ⏱ 2–5 hours, at Beginner difficulty.
Does structured data improve rankings?
No. Markup makes a page eligible for enriched presentations and states its entities unambiguously for AI models, but it is not a ranking signal and it cannot rescue a URL that robots.txt blocks. The catalogue frames the benefit as growth in rich snippets and in the chance of being quoted in AI answers — a chance, not a guarantee — for ⏱ 4–8 hours across 5–7 templates.
What is llms.txt and does my site need one?
llms.txt is a plain-text file at the site root telling model-driven readers what the site is and which page is canonical for each topic. It belongs to the catalogue service Preparing a website for AI crawlers: robots.txt, llms.txt, rendering — AI infrastructure, Beginner difficulty, ⏱ 4–8 hours, $150 – $450 or 10 000 – 35 000 ₽. It helps least on a client-rendered site, which returns nothing to a fetcher that does not run JavaScript.