We Tracked AI Crawlers for 30 Days: Here’s What Actually Visited Our Website

Status: Active collection — 28.2 of 30 UTC days processed. Interim evidence, not published as a completed results study.

Updated: 13 August 2026 Author: VITON13 Research Category: AI Search / Original Research Expected reading time after results: 12–16 minutes

Direct answer

VITON13 is now processing its production nginx access log for a fixed 30-day window from 15 July to 14 August 2026 UTC. At the 13 August interim snapshot, 431,701 production requests had been processed; 116,044 matched a registered crawler identity and 11,501 passed the currently supported provider-verification checks. These are interim observations, not a completed 30-day result. A user-agent string alone is never treated as proof that an operator visited the site, and the page remains noindex until the window and independent review are complete.

Live collection snapshot

The privacy-safe live aggregate contains no source IP, query value, raw user agent, request ID or individual visitor record. OpenAI and Google identities are checked against their official range files at analysis time; Bing uses forward-confirmed reverse DNS. Other identities remain unverified unless an operator-supported method is added.

Key findings available before data collection

• Search, model-training and user-triggered fetchers must be reported as different behaviors, even when they come from the same company.
• Google explicitly warns that crawler user agents can be spoofed and documents IP-range and reverse-DNS verification for Googlebot.
• OpenAI distinguishes OAI-SearchBot from GPTBot; allowing search crawling does not require opting into possible model-training crawling.
• Anthropic distinguishes Claude-SearchBot, Claude-User and ClaudeBot.
• Perplexity documents PerplexityBot separately from Perplexity-User and publishes IP ranges for both.
• A crawl request proves access, not indexing, ranking, model training, citation or a human visit.

What this report will answer

The finished study will answer six narrow questions:

1. Which declared crawler identities requested viton13.com during the study?
2. Which of those requests could be verified against operator-controlled infrastructure?
3. How many successful and unsuccessful requests did each crawler make per day?
4. Which public URL paths and content types received the most attention?
5. How often did each crawler request robots.txt, sitemaps and llms.txt?
6. Did request mix or frequency change after a documented publication or technical event?

It will not claim to answer whether a page was added to a private index, used in model training, ranked, cited, or read by a person. Those outcomes require different evidence.

Methodology

Research question and hypothesis

Question: Which verified search, AI-search, training and user-directed crawlers reached VITON13 during one uninterrupted 30-day UTC window, and what did they request?

Pre-registered hypothesis: traditional search crawlers will account for more verified requests than single-purpose AI-search crawlers. This is a hypothesis, not a result. It will be retained even if the data contradicts it.

Sample and dates

The sample will contain every HTTP request to the canonical production host during a future uninterrupted 30-day UTC window. The exact start and end dates will be written here before analysis. Preview, staging, localhost and health check hosts will be excluded.

Required fields

Crawler taxonomy

Cloudflare’s current bot reference supplies a useful cross-operator taxonomy, while each operator’s own documentation remains authoritative for its crawler’s purpose.

Identification and verification

The analysis has two layers.

Declared activity groups requests by a case-insensitive match against the complete user-agent string. This is useful for coverage but weak as identity evidence.

Verified activity requires operator-appropriate evidence. Googlebot is checked against Google’s published IP ranges or forward-confirmed reverse DNS. OpenAI and Perplexity requests are checked against their current published IP ranges. Where an operator does not offer stable ranges, a verified-bot signal from the CDN may be used and documented. Requests that cannot be verified are kept in the dataset as unverified; they are not silently discarded and are not counted as verified operator traffic.

The list of official ranges will be downloaded at analysis time and its URL, retrieval timestamp and checksum saved with the dataset. This avoids validating historical traffic against an undocumented future list.

Cleaning rules

• Convert timestamps to UTC and keep the original value in the raw dataset.
• Normalize paths only for aggregation; retain the raw path in restricted data.
• Remove query values, authentication material and personal identifiers from the publication dataset.
• Keep every status code, including blocked, redirected and rate-limited requests.
• Do not merge different user agents from one operator into an “AI bot” total without also publishing the component counts.
• Deduplicate only exact request-ID duplicates. Repeated legitimate requests are behavior, not noise.

Metrics

The report will publish request count, verified-request count, unique public paths, median and 95th-percentile interval between requests where meaningful, status-code mix, content-type mix, bytes transferred, repeat-visit rate, robots.txt/sitemap/llms.txt requests and top public paths. Percentages will always state their denominator.

What a crawler request can and cannot prove

A successful HTTP request establishes that a client received a response from the site. It does not establish that the operator indexed the URL, retained its contents, used it to train a model, showed it in search or cited it in an answer. Google’s documentation separates crawling and indexing, and OpenAI’s publisher guidance describes crawler access as a condition for discoverability rather than a placement guarantee.

This distinction is the central information gap in many crawler reports. The finished VITON13 study will keep the following outcomes separate:

Planned result tables

The final report will add, rather than pre-fill, these tables:

1. Verified crawler requests observed during the 30-day window. Operator, user agent, purpose, declared requests, verified requests, successful responses and unique paths.
2. Most-requested VITON13 paths by verified crawler. Path, page type, crawler, requests, last request and status distribution.
3. Crawler verification coverage. Requests verified, unverified and rejected by method, with the denominator shown.
4. Robots and discovery-file activity. Requests to robots.txt, sitemaps, feeds and llms.txt, without treating a missing request as non-compliance.

Planned charts

• Verified crawler requests by day. X axis: UTC date. Y axis: requests. One line per user agent; source: VITON13 Research log dataset.
• Declared versus verified requests. X axis: crawler identity. Y axis: request count; grouped bars; source: VITON13 Research log dataset.
• Response status distribution. X axis: crawler. Y axis: share of that crawler’s requests; stacked 2xx/3xx/4xx/5xx bars.
• Most-requested page types. X axis: page type. Y axis: verified requests; grouped by crawler purpose.

No analytics screenshot will be generated or reconstructed. Published charts will be rendered from the retained aggregate dataset.

Limitations

The study observes one site, one infrastructure configuration and one 30-day period. Verification support differs by operator. CDN logs may describe an edge response rather than an origin request. A crawler can change its identity, infrastructure or purpose during the window. Robots policy can change during a study, so its deployed version must be archived by date. Low request counts may make rate comparisons unstable. Most importantly, server logs cannot reveal a private index or model-training dataset.

Data required from VITON13

DATA REQUIRED

Source: production CDN and/or nginx access logs for viton13.com
Period: one uninterrupted future 30-day UTC window
Format: NDJSON, CSV, or standard combined nginx logs plus CDN verification data

Required fields:
timestamp
hostname
path
query-present flag (values redacted for publication)
user-agent
source IP or provider verified-bot ID
verification method or evidence reference
HTTP method
status
content-type
bytes sent
cache status
request ID

Also required:
robots.txt snapshots and deployment timestamps
sitemap deployment timestamps
content publication timestamps
WAF/rate-limit rule-change timestamps

The raw export may contain IP addresses and other security-sensitive fields. It must remain access-controlled. Only aggregated, privacy-reviewed data will be published.

FAQ

Is OAI-SearchBot the same as GPTBot?

No. OpenAI documents OAI-SearchBot for search discoverability and GPTBot as the user agent publishers can block when they want content excluded from potential training. They must be counted separately.

Does seeing Googlebot mean a page was indexed?

No. It means a client declaring or verified as Googlebot requested a resource. Google explicitly distinguishes crawling from indexing.

Can a crawler user agent be faked?

Yes. Any HTTP client can send a familiar user-agent string. That is why this protocol reports declared and verified activity separately.

Should private paths be added to robots.txt?

Robots.txt is not access control. RFC 9309 warns that listed paths are public and that protected resources need real authentication or another application-layer security measure.

Will allowing a search crawler guarantee citations?

No. Access can make content eligible to be read, but it does not guarantee indexing, ranking, inclusion or citation. The report will not turn access into a visibility promise.

Why keep blocked and failed requests?

They explain whether infrastructure returned 403, 429 or 5xx responses and can reveal a policy or availability problem. Deleting them would bias the study toward successful access.

When will the results be published?

After one complete 30-day window, identity verification, privacy review, calculation review and editorial fact-checking. Until then the public page is a protocol, not a result.

Sources and evidence register

Every methodological claim above is tied to an operator or standards source. The dated, annotated register is maintained separately in sources.md: OpenAI publisher and search documentation, Google crawler and verification documentation, Anthropic and Perplexity crawler references, Cloudflare’s bot reference and analytics API documentation, and IETF RFC 9309. These sources define crawler behavior and verification methods; none of them is used as evidence of VITON13 traffic.

Update plan

Review the protocol every 30 days while collection is pending. After publication, review it every 90 days and immediately after a crawler-policy, IP-range, CDN-classification or major AI-search documentation change.

Quality score before data