The AI citation dataset strategy

Why datasets, not just reports

A chart or a percentage is a conclusion someone else drew from data you can't see. A downloadable dataset is the thing itself. You can check it. You can re-analyze it. You can reuse it for a question the original report never asked.

Nearly every visibility vendor in this field publishes reports. Almost none publish datasets. That gap is deliberate on their part, since the data is their product. It's exactly the gap this site is built to fill.

There is a second, quieter reason to publish rows rather than conclusions. A summary is frozen at the moment its author wrote it. A dataset keeps answering new questions for as long as it exists, including questions that had not occurred to anyone when it was collected.

The gap this is filling

It is worth being specific about why this matters rather than asserting that open data is generally good. The gap here is structural, and it explains most of what is wrong with statistics in this field.

Nearly every widely-quoted AI-search number traces back to a visibility vendor. That is not a criticism of any individual vendor. It is a description of who has the infrastructure to measure citation at scale. Building and running a citation-collection system across seven engines is expensive, and the companies that have done it did so to sell a product.

Which produces a specific consequence. The data is the product. A vendor publishing its full corpus would be giving away the thing customers pay for. So methods stay partially described, samples stay unpublished, and nobody outside the company can check the work.

The result is a field where confident percentages circulate and essentially none of them are reproducible. Trace twelve of the most-quoted figures, as the provenance audit does, and the pattern is consistent: real sources, real numbers, no way to verify any of it independently.

An independent publication has the opposite incentive. There is no product to protect. The credibility is the product, and credibility here comes from being checkable. Publishing the raw rows costs nothing that matters and buys the one thing the vendor-reported figures cannot have.

That is the gap. Not a shortage of numbers. A shortage of numbers anyone can verify.

Datasets planned for release

What a single dataset row will actually look like

"Raw citation records" sounds abstract. Here's what one row will actually contain: the query text, the engine, a timestamp, and the run number (each query runs several times to capture variance). Also the cited URL and its domain, its position in the response, and a snippet of the surrounding context.

A researcher loading the full file could filter to every row where one domain appears. They could compute that domain's citation rate across the whole query set. They could compare rates across engines directly. That's the kind of re-analysis a summary chart alone can never support.

What each dataset will contain, field by field

"Raw citation records" describes very little on its own. Here is the intended shape of each planned dataset, so anyone deciding whether it would be useful can judge before it ships rather than after.

The Citation Index corpus. One row per citation observed. Fields: query text, query category, engine, run number, collection timestamp, cited URL, cited domain, position within the response, and a snippet of surrounding context. The run number matters more than it looks: because AI answers are non-deterministic, the same query across five runs produces the variance data that makes any single measurement interpretable.

The Standard GEO Query Set. One row per query. Fields: query text, assigned category, intent classification, and the version in which it entered the set. Publishing the query set matters as much as publishing the results. A citation rate measured against an undisclosed query set cannot be reproduced or challenged by anyone.

The AI Bot Registry, machine-readable. One row per bot. Fields: user-agent string, operator, category (training, retrieval, or agentic), documented robots.txt policy, verification hostname pattern, and the date each field was last confirmed. The last field is the one most bot lists omit, and it is what separates a maintained registry from a stale listicle.

The robots.txt census. One row per domain and bot pair. Fields: domain, category, whether a robots.txt exists at all, the rule found for that specific bot (allow, disallow, or no rule), and the collection timestamp. Structured this way, the dataset supports cross-tabulation by category and bot type, which a single aggregate blocking percentage cannot.

The Query Fan-Out Corpus. One row per inferred sub-query. Fields: parent query, inferred sub-query segment, its classification against the patent's eight documented types, and the citations attributed to that segment. This is the dataset with the largest methodological caveat attached, since sub-query boundaries are inferred from observable output rather than read from an internal log.

The citation half-life tracking set. One row per URL, engine, and collection window. Fields: URL, engine, window number, citation rate in that window, and days elapsed since first observation. Twelve months of these rows is what produces a survival curve, which is a thing no published dataset currently supports.

The RCT panel results. One row per participating page. Fields: anonymised page identifier, treatment or control assignment, before and after citation rate, collection windows, and the specific intervention applied. Page identity gets anonymised; the assignment and outcome do not.

Format and licensing

Every dataset ships as CSV and JSON, under CC BY 4.0 — free to reuse commercially or non-commercially, with attribution back to the specific version used. Consistent with the licensing already committed for the Citation Index.

Why CC BY 4.0 specifically

The licence choice is deliberate and worth explaining, because a more restrictive one would have been the easier default.

It permits commercial use. A visibility vendor could build a paid product on top of these datasets. That is allowed on purpose. A licence that blocked commercial use would exclude exactly the organisations most able to extend the work, and it would make the data less useful without making it more trustworthy.

It requires attribution, and nothing else. Attribution is what makes a citation chain traceable. If someone quotes a figure derived from this data, a reader should be able to walk back to the original rows. That is the entire mechanism this site argues the field is missing.

It does not require share-alike. A copyleft clause would force derivative datasets to carry the same licence. That sounds principled and in practice discourages exactly the reuse the data exists to enable. Someone combining this with a proprietary dataset should not have to relicense their own work.

It is a licence people already recognise. A bespoke licence, however well-intentioned, creates a legal review step for anyone in an organisation with a compliance function. CC BY 4.0 clears that step in most places without a conversation.

The trade-off accepted here is loss of control. Someone can take these datasets, build something better, and compete. That is fine. If an openly published dataset gets used to produce better research than this site produces, the goal was met.

Versioning and changelog practice

Every dataset release carries an explicit version number. Any correction or update publishes as a new version, with a changelog entry stating exactly what changed and why. We never silently edit an existing file.

This matters because a dataset that's already been cited needs a stable reference point. Silently changing a published version would break that trust. It would make any prior citation of it unreliable.

Privacy and what gets excluded

Open publication has limits, and stating them in advance is part of the commitment. A few categories of data will not appear in these releases.

Panel participant identity. The RCT and log-study panels depend on site owners volunteering their pages. Their domains get anonymised in published data. The treatment assignment, the intervention, and the outcome are all published. Which specific site produced which row is not.

Raw server logs. Where a study analyses participant server logs, the published dataset contains derived aggregates rather than the logs themselves. Raw logs contain IP addresses and request patterns that are not ours to publish, regardless of what a participant consents to.

Anything identifying an individual. None of these studies collect personal data by design. Where something could incidentally identify a person, a query containing a name for instance, it gets excluded rather than published and hoped over.

Content bodies of third-party pages. The citation corpus records URLs, domains, and short context snippets. It does not republish the full text of cited pages, which would be a copyright problem and adds nothing to the analysis.

What does not get excluded is anything inconvenient. A finding that contradicts a claim made elsewhere on this site, or a data quality problem discovered mid-collection, gets published with the rest. Privacy is a genuine constraint. It is not a category that expands to cover embarrassment.

Where they will live

Alongside each study's own page on this site, mirrored to a public code repository once the collection tooling itself is released — three surfaces for one collection effort, as described in the wider research strategy.

What someone could actually build with these

A dataset release is only as valuable as what people do with it. Here are concrete analyses these files would support that nothing currently public does.

Your own domain's citation profile. Filter the citation corpus to rows where your domain appears. Compute your citation rate across the query set, per engine, and see which query categories you win and which you never appear in. No visibility tool currently lets you check its work; this would let you compute the number yourself.

Competitive share of voice on a fixed base. Because every engine runs the same published query set, you can compute any domain's share of total citations on identical footing. Most competitive comparisons today rest on different tools measuring different query sets and calling the results comparable.

Independent verification of this site's own claims. The most important use, and the reason for publishing at all. Every figure this site reports from the Index should be recomputable from the released rows. If someone recomputes and gets a different answer, that is a finding worth publishing, and we would rather it come from an outside check than not happen.

Cross-engine concordance for any subset. The published concordance figure is one aggregate number. With the raw rows, anyone can compute concordance for their own vertical, their own query type, or any pair of engines they care about.

Research nobody has thought of yet. This is the honest and least specific reason. A dataset supports questions its collectors did not anticipate. A summary report only ever answers the question its author asked.

Tooling built on top. The robots.txt census and machine-readable bot registry are directly usable as inputs to an audit tool. Someone could ship a better crawler-access checker than this site will, using these files, and that would be a good outcome.

How to contribute

Two routes: contribute queries to the Standard GEO Query Set if you have coverage gaps you'd like included, or join the Citation Index's panel of participating sites if you'd share anonymized server logs. Both are described in the Citation Index's own page.

The quality bar a release has to clear

A file being downloadable is not the same as a dataset being usable. Several things have to be true before a release ships, and stating them in advance is how this page stays checkable later.

Every column documented. A data dictionary shipping alongside the file, defining each field, its type, its permitted values, and what a null means. A column named "position" with no definition is a column nobody can safely use.

The collection protocol stated. Exact query timing, session state, geography, retry logic. Two people running the same collection under different conditions get different results, so the conditions are part of the data.

Known quirks disclosed. Every real dataset has them. A collection window where an engine changed behaviour mid-run. A query that returns nothing on one platform. Rows affected by a bug found after collection. Hiding these makes the file look cleaner and makes any analysis built on it wrong.

A stated row count and file size. Small, but it lets someone confirm they downloaded the complete file, and decide whether they need a database or a spreadsheet.

A worked example query. One documented analysis, with the code, that reproduces a figure published on this site from the raw rows. That single example does more to make a dataset usable than pages of prose documentation.

A release missing any of these is not ready, regardless of whether the underlying collection finished. Publishing raw rows without them technically satisfies "open data" while leaving it practically unusable, which is a failure mode worth naming because it is common.

Precedent this approach is borrowed from

Open, versioned dataset releases with a stated license and a public changelog are standard practice in academic and open-science circles. They're also increasingly common in classic-SEO-adjacent open tooling.

Borrowing that discipline isn't new for research in general. It is genuinely rare in the AI-visibility vendor space specifically, where the underlying data is usually treated as the core commercial asset. That contrast is exactly the gap this strategy is built to occupy.

What publishing openly actually costs

Most arguments for open data skip the costs, which makes them less persuasive rather than more. There are real ones, and they are worth stating plainly.

Errors become permanent and public. A summary report with a mistake can be quietly corrected. A published dataset that someone has already downloaded and cited cannot. This is the strongest practical argument against open release, and the mitigation is the versioning discipline described above rather than any way of avoiding it.

Preparation is real work. Data that is merely collected is not data that is publishable. Column definitions, consistent formatting, documentation of quirks, and privacy review all take time that produces no new findings. Budgeting for it is the difference between a release plan and an intention.

Misuse is a certainty, not a risk. Someone will pull a single row out of context, or compute a ratio the sample cannot support, and publish it. Attribution requirements do not prevent this. The accepted trade is that a verifiable number occasionally misused is better than an unverifiable number universally trusted.

It forecloses a business model. The obvious commercial path in this field is to build the measurement infrastructure and sell access. Publishing the raw output closes that door deliberately. That is a real cost, and pretending otherwise would undercut the argument.

It invites scrutiny you cannot control. Anyone can check the work and publish what they find. That is the entire point, and it is still uncomfortable in a way worth acknowledging rather than performing indifference about.

Weighed against those: nobody currently has to take a vendor's word for anything they publish here. In a field where that is the norm, it is worth the costs above.

The release sequence, and why it runs this way

These datasets do not all ship together, and the ordering is not arbitrary. Each release depends on the one before it in a specific way.

The query set ships first. Before any citation data exists, the query set can be published. This is deliberate: it lets anyone challenge the sample before results are collected, rather than after. A query set published alongside favourable results invites the obvious suspicion that the queries were chosen to produce them.

The bot registry and robots.txt census ship next. Neither depends on citation collection. Both are useful immediately, and both are the kind of reference data that improves with outside corrections, so getting them out early is a net gain.

The citation corpus ships with the first full Index release. This is the largest and most consequential file. It cannot precede the collection it records.

The fan-out corpus follows the citation corpus. It depends on the same collection infrastructure and adds an inference layer on top. Shipping it second means the underlying citation data has already been checked by outside readers before a more interpretive dataset builds on it.

The half-life tracking set ships last, and slowly. A survival curve needs twelve months of repeated observation. There is no way to compress this, and any interim release would be a partial curve that invites over-reading. Interim windows get published as they complete, clearly labelled as incomplete.

The RCT panel results ship when the studies conclude. Tied to each study's own schedule, and to whether panel recruitment reaches a workable size. This is the part of the plan most exposed to slipping, because it depends on other people volunteering.

Any slip against this sequence gets posted on the relevant study page rather than quietly absorbed. A release plan that only ever reports on-time delivery is not a plan being tracked honestly.

Limitations

Stated plainly, because a release plan that describes only its strengths is a marketing page.

• None of these datasets are live today. This page is the plan, not the download page.
• A plan is not a delivery. Everything here is a commitment made before the work is done. Judge it against what actually ships, on the schedule each linked study states.
• Panel-dependent datasets carry the most risk. The RCT results and log studies need volunteers. If recruitment falls short, those releases shrink or slip, and that constraint is outside our control.
• The fan-out corpus is inference, not observation. Sub-query boundaries are reconstructed from visible output because no public interface exposes the internal list. That dataset will carry a larger error bar than the others and should be read accordingly.
• Release timing depends on each underlying study's own collection schedule, which may slip — any delay will be posted on the corresponding study page.