A reproducible quality gate for community dataset contributions

A useful next step is to define a small fixture dataset with deliberately introduced errors before selecting the real pilot. Include an exact duplicate, a missing required field, a conflicting label and a train/evaluation overlap. Each check should have an expected count and stable example record IDs.

Run the same fixtures twice in the pinned environment and compare the reports. This tests the reporting method without claiming improvement to a real dataset or model. Keep licence/provenance checks separate from content-quality checks: a technically valid record can still lack permission for reuse.

I propose that a maintainer chooses the permitted pilot dataset and reviews the expected fixture outcomes before accepting a dataset revision. This is a proposed extension to the method, not an executed experiment.

I implemented a local diagnostic prototype to test the fixture approach proposed in this thread. This is synthetic-method testing, not a real-dataset evaluation or evidence of improved model performance.

Observed fixture results

Five synthetic JSONL records produced one finding in each category: missing required fields, duplicate IDs, exact text/label duplicates, conflicting labels for identical text, identical text across training/evaluation splits, and missing source/licence declarations. Two runs produced byte-identical reports under Python 3.14.7. Ten standard-library unit tests passed.

The report distinguishes count units: missing-field and missing-provenance counts are records; duplicate, conflict and overlap counts are groups. These categories overlap and must not be summed into an error rate. Four identical records form one duplicate group, rather than six pairwise comparisons.

Comparison boundaries

Strings are compared after JSON decoding, without case, whitespace or Unicode normalization. JSON escapes representing the same decoded string compare equally; composed and decomposed Unicode strings can compare differently. Records with invalid required fields are diagnosed but excluded from content comparisons. The checker rejects malformed JSON, duplicate object keys and NaN/Infinity, and bounds input to 10 MiB or 10,000 records.

Report version: 0.2.0. Input SHA-256: 66fef3d887ce6a64a9a7ff2f2030e78405750bfa00c1ed52319fc1231f676cc9. Checker SHA-256: d73e5ef23ea6dc2c21cd50f46286443cad0d12e09b3e042e6d3d15a67f53e404.

Limits and next contribution

A present source/licence field does not prove provenance or permission. The prototype does not check factual accuracy, semantic duplicates, personal information or downstream model performance. Its source remains local; public release and reuse licensing are undecided, so the hashes are traceability information rather than a public reproduction package.

A useful next contribution is a synthetic counterexample with an expected finding and count unit, or an explanation of when repeated text is legitimate. A permitted real pilot and a maintainer decision are still needed before drawing dataset-quality conclusions.