Skip to content

Build an extract someone else can reproduce

This tutorial exports an annual TOSSD dataset to Apache Parquet format, inspects the companion JSON export manifest, and verifies dataset vintages across repeated runs.

What you'll build

An analytical Parquet data extract and its accompanying JSON export manifest.

exports/tossd_2019.parquet
exports/tossd_2019.manifest.json

What you'll learn

  • How to export a full TOSSD annual release to Parquet format.
  • How to inspect provenance metadata in the generated JSON manifest.
  • How HTTP ETag headers identify published data vintages.
  • How to verify an extract's integrity and reload it with load_export.
  • How to document and package data extracts for reproducible research handover.

What you'll need

  • Python 3.12 or newer, with tossd-reader installed.
  • About seven minutes.
  • Cached 2019 TOSSD vintage (initial calls download the published file to local storage).

Step 1: Export one year

Export the 2019 published TOSSD dataset into a dedicated export folder.

import tossd_reader as tossd

path = tossd.export("exports", years=2019)
path

UserWarning: 1 code(s) across 1 column(s) not in the packaged codelists (vintage newer than snapshot?): parent_channel_code has 45000.
PosixPath('exports/tossd_2019.parquet')

The warning means the packaged codelist snapshot hasn't caught up with one parent_channel_code value in the published 2019 data. parent_channel_name reads NA for the 116 rows carrying that code, and the rest of the export is unaffected.

Passing a directory path generates a standard filename (tossd_2019.parquet) and creates the target directory when required. The export() function preserves all published columns and original publisher units (USD thousand). Custom filtering by recipient, provider, or pillar and unit conversions take place downstream during analysis with get_tossd().

Step 2: Read the manifest

Inspect the JSON manifest file created alongside the Parquet extract to examine the recorded provenance metadata.

from pathlib import Path

print(Path("exports/tossd_2019.manifest.json").read_text())

{
  "created_at": "2026-09-02T08:24:37.680642+00:00",
  "payload_sha256": "8a6eed10875a87fcd5faedece760bc461aa5926113ba1686613728c8c27d30bf",
  "row_count": 290914,
  "schema_hash": "0a95f2c54852817a9db1a2174cffa5bd371d601e5d137a37cb27491182367df9",
  "tossd_reader_version": "0.1.0",
  "vintages": {
    "2019": {
      "etag": "\"69e6ac86-347a653\"",
      "retrieved_at": "2026-08-28T21:14:14.414671+00:00"
    }
  },
  "years": [
    2019
  ]
}

tossd_reader_version identifies the package version, while schema_hash records the SHA-256 hash of the packaged schema definition. payload_sha256 hashes the Parquet payload itself; verify_export() recomputes it on demand to confirm the file still matches this manifest. created_at records the export timestamp, and row_count records the number of exported rows. The vintages mapping records the HTTP etag and initial retrieved_at timestamp for each year included in the file.

Step 3: Pin the vintage

The International Forum on TOSSD (IFT) publishes official data at tossd.online under stable URLs, updating files in place as revisions occur. The HTTP ETag header changes whenever the publisher releases an update. The manifest records this ETag to pin the exact vintage used.

Export the 2019 dataset a second time to verify that the recorded ETag remains consistent across runs.

import json
from pathlib import Path

import tossd_reader as tossd

tossd.export("exports", years=2019)
manifest = json.loads(Path("exports/tossd_2019.manifest.json").read_text())
manifest["vintages"]["2019"]["etag"]

'"69e6ac86-347a653"'

The matching ETag confirms that both exports used the identical upstream data vintage. The library checks the upstream ETag only when a discovery sweep runs, once per process by default, or again when refresh=True forces one. Both export() calls in this tutorial run in the same process, so the second call uses the cached result of the initial discovery sweep. Passing refresh=True clears the cached discovery result and retrieves the latest published version directly from tossd.online.

Step 4: Verify and reload the extract

verify_export() recomputes the Parquet file's hash and compares it against payload_sha256 in the manifest. Run it before handing an extract to a colleague, or right after receiving one, to confirm the file wasn't corrupted or edited in transit.

tossd.verify_export("exports/tossd_2019.parquet")

A silent return means the file matches its manifest.

load_export() calls verify_export() first, then reads the Parquet file back with the schema's nullable integer dtypes intact and attaches the manifest's provenance to df.attrs.

df = tossd.load_export("exports/tossd_2019.parquet")
df.shape

(290914, 55)

df.tossd.provenance() reads that provenance back as a deep copy, without needing to reopen the manifest file.

from pprint import pprint

pprint(df.tossd.provenance())

{'created_at': '2026-09-02T08:01:05.881968+00:00',
 'package_version': '0.1.0',
 'years': {'2019': {'etag': '"69e6ac86-347a653"',
                    'retrieved_at': '2026-08-28T21:14:14.414671+00:00'}}}

The row count matches row_count in the manifest, and the etag matches the value recorded in Step 2. Provenance is accessible directly from the loaded DataFrame.

Deterministic pipelines with offline mode

An automated pipeline configured with tossd.set_offline(True) accesses only vintages already pinned in local cache, preventing unrecorded upstream updates. See Work offline.

Step 5: Hand it over

A complete reproducible research handover includes three things.

  • The Parquet data file (exports/tossd_2019.parquet).
  • The JSON export manifest (exports/tossd_2019.manifest.json).
  • Analytical documentation recording the year range, provider filtering criteria, aggregate row handling, units, and price basis.

Integrity verification on corrupted payloads

When an exported file changes in transit, verify_export() detects the modification.

ExportIntegrityError: exports/tossd_2019.parquet does not match its manifest: sha256 49d27ebdb8030316… but the manifest recorded 8a6eed10875a87fc…. The file may have been modified or corrupted since export.

Memory overhead when exporting multi-year releases

Exporting all six years without a years argument materialises 2.4 million rows in memory as an Apache Arrow table before writing to disk. Peak RAM reaches roughly 4.4 GB. The finished table itself is about 2.1 GB, but per-year tables stay alive alongside it until they are concatenated. Supply specific reporting years to years= when exporting on memory-constrained systems.

If a cached provenance file is corrupted, the manifest records null for etag and retrieved_at while emitting a warning during export. A missing provenance file records the same null values, silently.

What's next