Why pages are not indexed: an audit of our own 3310 addresses

The number this started with

The page indexing report in Google Search Console puts two figures side by side. Ours read: 738 pages indexed, 2572 not indexed. Google knows about 3310 addresses on this site. Twenty-two per cent of them made it into search.

A gap on its own is not a verdict. A young domain is almost always indexed partially — a search engine has no reason to trust a site it met six months ago the way it trusts one with a decade of history. Some pages are closed to indexing deliberately, some redirect, some are administrative.

But a four-and-a-half-fold gap is no longer noise. When fewer than a quarter of three thousand addresses reach search, you open the breakdown by reason and read it line by line instead of guessing from general advice.

What follows is that breakdown, with our own figures, the mistake we made along the way, and the fix that worked. Nothing theoretical: all of it happened on this site inside one week.

Ten reasons, and they are not equal

The report sorts unindexed pages into categories. We had ten, and the first thing to understand is that they differ enormously in weight. Half can be safely ignored.

Page with redirect — 27 addresses. Normal site behaviour: an old address points at a new one, the crawler notes it and moves on. Nothing to fix.

Alternate page with proper canonical tag — 16 addresses. Also normal: several versions of one page exist, you named the primary one, and the crawler heard you.

Excluded by noindex tag — 132 addresses. This can be normal if you closed those pages on purpose. Ours are closed by the journal's editorial filter: pieces that fall short on depth or sourcing do not enter search. It would be a problem only if something landed there that we never meant to close.

Blocked by robots.txt — 5 addresses. Administrative sections. Intended.

The remaining three categories are where the real conversation starts.

What crawled but not indexed actually means

The phrase means exactly this: the crawler arrived, read the page in full, and decided against it. Not could not, not did not find — decided.

We had 383 pages here, and thirty-two more appeared within a day. We found the cause quickly, because we had been working on product pages shortly before.

The Russian product description was assembled on the fly from the English name and a translated category. The result read like a considered design in such-and-such category, quiet silhouette, quality materials. The same sentence across all sixteen products, with two words changing.

To a search engine that is sixteen near-identical pages. It reads the first, the second, the third, and stops seeing the point of holding the rest. Exactly the same happens with descriptions copied from a supplier: if one text sits on a hundred sites, none of them gains an advantage.

There is one cure — unique text on every page. Not rewritten with synonyms, but written about the specific item: what it is, what it is made of, what it goes with, how it differs from the piece next to it in the catalogue.

Why we close some pages to search ourselves

A short detour, because 132 closed addresses look like an oversight and are in fact a rule we wrote deliberately.

The journal runs an editorial filter: roughly twenty-five conditions a piece must satisfy before it enters search. A minimum length, set separately per language. At least two sources on two independent domains. No repeated headings inside the text. Primary keyword density no higher than a little over two per cent.

A piece that fails the filter is still published and still readable, but carries a noindex tag. A thin note stays out of search and does not drag the section's reputation with it.

The logic is simple. A search engine judges not only the individual page but the section as a whole. A hundred strong pieces and three hundred weak ones is a site where three quarters of the journal is not worth attention, and new texts in it index more slowly. A hundred strong pieces and three hundred closed ones is a site where search contains only what we are prepared to defend.

The difference is that the first version is something the search engine works out for itself and generalises to the whole section. The second is something we tell it in advance.

What discovered but not indexed actually means

This category is heavier than the previous one, though it sounds gentler. It means the crawler never came. The address is known — you submitted it in your sitemap — but no crawl was allocated to it.

We had 1331 addresses here. A day earlier there were 1135. The category was growing by roughly two hundred a day and was the largest in the report.

The distinction matters. In the first case the search engine assessed the content and declined. In the second it spent no time on the page at all — so the question is not the text, but why the site did not earn the crawl.

That is what crawl budget means. The crawler arrives with a limited allowance of time and requests. It is not obliged to cover everything you submit, and it decides for itself where to spend. On a young domain the allowance is small, and it has to be spent carefully.

Three places where the cause usually sits

When the crawler does not come, the answer is almost always in one of three places.

First, the size of the sitemap relative to the domain's authority. Two thousand addresses for a site with a hundred inbound links is a claim with nothing behind it. The search engine sees the volume, weighs the trust, and crawls a fraction.

Second, internal links. A page that no other page on the site links to barely exists for a crawler. A sitemap communicates an address; it does not communicate importance. Links communicate importance.

Third, quality signals across a whole section. If the crawler has covered a hundred pages of one section and taken none into the index, it economises on the rest as well. The section acquires a reputation, and that reputation extends to new addresses inside it.

We checked all three and found the cause in the second and the first at once.

Forty-three per cent of the sitemap went to one section

We opened the sitemap and counted what was in it. There were 2074 addresses. Of those, 900 belonged to one catalogue — a product section with three hundred and one items across three languages.

Forty-three per cent of everything we submitted to search went to a single section. And that section was barely being indexed at all.

Then we counted internal links. The section's own page linked to twenty-five of the three hundred and one items. The remaining two hundred and seventy-six were reachable only through eleven pagination pages. Those pagination pages were not in the sitemap at all.

The picture came together. We submit nine hundred addresses to search while telling the crawler, through the structure of the site, that twenty-five of them matter. It reads that literally: if the owner does not consider them worth linking to, they are probably not important.

Why this slows the whole site, not just one section

Here is the point most often missed. Crawl budget is shared across the domain. It is not divided evenly between sections and it is not reserved for what matters.

Which means a hundred service pages with prices compete for the crawler's attention with a thousand addresses that were never going to rank. And they lose, because there are simply more of the latter.

This is not a penalty. It is ordinary allocation of a limited resource. But there are exactly two levers for managing it: what you submit in the sitemap, and what you link to inside the site.

We used the first, because it works immediately, while the second requires rebuilding the section.

The mistake we made along the way

The standard advice for a large sitemap is to split it into an index: one main file pointing at several thematic ones. It makes crawl allocation easier for the search engine and per-section coverage visible to you.

We did exactly that. And broke the sitemap entirely.

The site runs on Next.js with localised routes such as /ru and /es. The moment the sitemap switches to index mode, the main file stops being served by its own handler and falls into the general route, which reads its name as a language code. The result is a 404 on the very file robots.txt points to.

We caught it before release, on a local build. Rolled it back, confirmed the sitemap returned 200 again with its previous address count. Nothing broken reached production.

The lesson is not about Next.js: any sitemap change must be verified with a request to the file itself, not by reading the code. It costs thirty seconds, and the price of the error is the loss of the sitemap entirely until somebody notices.

What we changed in the end

We removed nine hundred product pages from the sitemap and kept the section page in it. The sitemap shrank from 2074 addresses to 1174.

The pages themselves stayed exactly where they were. They open, return 200, remain reachable by people and by internal links, and take part in navigation. Only the submission to search stopped — the request that says please crawl this.

The difference between removing from a sitemap and closing to indexing is fundamental, and it is constantly confused. A noindex tag tells the search engine: do not show this page in results. Removal from the sitemap says something else: I am no longer asking you to spend a crawl on it right now.

The second is reversible in one line and leaves no trace. When the section has proper internal links, or the domain has more authority, the product pages return to the sitemap.

What to expect afterwards, and what not to

There will be no instant effect. A search engine re-reads a sitemap over one to two weeks, and the discovered category shrinks gradually rather than resetting.

Something else matters more, and it deserves saying plainly: trimming a sitemap does not create authority. It redistributes the crawler's attention without increasing it. If the domain has few inbound links, fifteen hundred addresses will also index slowly — just slowly and to some purpose rather than slowly and for nothing.

So treat this as hygiene, not strategy. It clears the noise so the rest can work honestly. Growth comes from other things: mentions on outside sites, unique text on pages, real queries that actually find you.

What we are doing next

Trimming the sitemap is the first of three steps, and the other two take longer.

The second is rebuilding internal links in that catalogue. Twenty-five links to three hundred and one items has to become a structure where any product is two or three clicks from the home page: curated groupings, cross-links between neighbouring items, pagination pages present in the sitemap. This is not about SEO text at the bottom of a page; it is about navigation people actually use.

The third is returning the product pages to the sitemap in tranches. Not all nine hundred at once, but a hundred or so at a time, starting with those that already have links pointing at them. That way each tranche's indexing is observable, and if one stops being taken, you know where to stop.

The order is deliberate. Returning addresses to the sitemap without fixing the links repeats the same story a month later.

A separate word about 404s

The report shows 673 addresses returning not found, with validation marked as failed. The number looks alarming next to 738 indexed pages.

But it is an inheritance from a platform migration. The search engine knew hundreds of old addresses for years, and after the change of engine they stopped existing.

A 404 is correct here. More than that: for pages deleted permanently with no replacement, 410 is the better answer — a direct signal of gone for good that clears such addresses from the queue faster.

What not to do: redirect old addresses en masse to the home page. Search engines recognise that as a soft error and decline to index anyway, and you lose the ability to see which old pages people still look for.

How to check the same thing on your own site

Open the page indexing report and write down every reason with its address count. Add them up and compare the total with the number of lines in your sitemap — the discrepancy alone tells you whether the search engine knows about pages you never submitted.

Find the category growing fastest. That one, rather than the largest in absolute terms, shows where the site is heading.

Count what your sitemap is made of, section by section. If one section takes more than a third and is not being indexed, you have found your bottleneck.

Look at how many internal links point into that section. A twofold gap between addresses in the sitemap and links pointing at them is already a signal.

And spot-check a dozen addresses from the sitemap for their robots meta tag. Submitting to search what you have closed to indexing is a common and expensive mistake: it burns crawl and damages trust in the sitemap as a whole.

On how often to check

One last thing, and it will save you time. Search Console data lags by two to three days.

We exported the report three days running. The second and third exports matched to the unit across all ten rows — not approximately, but literally the same figures.

Reading the report daily is pointless: you see the same snapshot and draw conclusions about a trend the data does not contain. Once a week is frequent enough to notice movement without mistaking export lag for the result of a fix.