Export

export casts and types TOSSD activity records, then writes the result to a compressed parquet file with an export manifest. Exports retain all columns (columns="all") and original units (units="usd_thousand" in USD thousands).

import tossd_reader as tossd

path = tossd.export("exports", years=2019)
print(path)

exports/tossd_2019.parquet

Write the normalised, typed get_tossd pipeline output to parquet.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	A directory (created if missing), in which case the parquet file is written as `tossd_<years-range>.parquet` inside it; or an explicit path ending in `.parquet` (its parent directories are created if missing).	required
`years`	`int \| Iterable[int] \| None`	Same semantics as `get_tossd`'s `years=`: a single year, an iterable of years, or `None` (the default) for the packaged known-years set.	`None`
`refresh`	`bool`	Same semantics as `get_tossd`'s `refresh=`.	`False`
`max_rows`	`int \| None`	Opt-in guard: once the table is built, raise `ValueError` (naming the actual row count and this limit) before writing anything, if it exceeds `max_rows`. `None` (the default) applies no limit -- the historical behaviour, unchanged.	`None`

Returns:

Type	Description
`Path`	The path the parquet file was written to. A sidecar
`Path`	`<stem>.manifest.json` is written alongside it, carrying the
`Path`	package version, a schema hash, a sha256 of the written parquet's
`Path`	own bytes (`payload_sha256`, checked by `verify_export`), the
`Path`	exported years, per-year vintage provenance (etag/retrieved_at),
`Path`	the total row count, and the export's creation timestamp.

Note

years=None (the default) materialises the full packaged known-years set as one arrow table, all in memory, before it's ever written to disk (columns="all", per this function's own contract) -- measured at roughly 4.4GB peak resident for the full set (the finished table alone is ~2.1GB; per-year tables are transiently alive alongside it). Pass an explicit years= (or max_rows= to fail fast instead) to export a smaller slice if that's a concern.

Raises:

Type	Description
`ValueError`	`years` resolves to an empty set of years; `max_rows` is given and the built table exceeds it; or `refresh=True` while offline mode is active (`config.get_offline()` is `True`).
`TossdNetworkError`	Same conditions as `get_tossd`; export applies no provider/recipient/pillar/`filters=` filters, so only the fetch/schema layer's own failure modes apply in practice.
`SchemaDriftError`	Same conditions as `get_tossd`.

Example

import tossd_reader path = tossd_reader.export( # doctest: +SKIP ... "tossd_2024.parquet", years=2024 ... ) df = tossd_reader.load_export(path) # doctest: +SKIP

Generated filenames

When path specifies a directory, export creates the directory if needed and writes tossd_<years>.parquet inside it. When path ends in .parquet, export writes directly to that file path, creating parent directories as needed.

The <years> segment reflects the resolved, sorted list of years:

Single year: tossd_2019.parquet
Contiguous range: tossd_2019-2024.parquet
Non-contiguous set: tossd_2019_2021_2024.parquet

Manifest fields

Each export writes a companion <stem>.manifest.json file beside the parquet file.

{
  "created_at": "2026-09-02T08:24:37.680642+00:00",
  "payload_sha256": "8a6eed10875a87fcd5faedece760bc461aa5926113ba1686613728c8c27d30bf",
  "row_count": 290914,
  "schema_hash": "0a95f2c54852817a9db1a2174cffa5bd371d601e5d137a37cb27491182367df9",
  "tossd_reader_version": "0.1.0",
  "vintages": {
    "2019": {
      "etag": "\"69e6ac86-347a653\"",
      "retrieved_at": "2026-08-28T21:14:14.414671+00:00"
    }
  },
  "years": [
    2019
  ]
}

Field	Type	Description
`tossd_reader_version`	string	Package version that created the export file.
`schema_hash`	string	SHA-256 hash of packaged `schema.csv`, CRLF-normalised.
`payload_sha256`	string	SHA-256 hash of the exported parquet file's own bytes. Checked by `verify_export`.
`years`	list of `int`	Sorted list of reporting years included in the export.
`row_count`	int	Total row count across all exported reporting years. Checked by `verify_export`.
`created_at`	string	Export timestamp in ISO 8601 format (UTC).
`vintages`	object	Mapping of reporting year to source metadata (`etag` and `retrieved_at`).

payload_sha256 and row_count make an export file self-checking. verify_export re-hashes the parquet payload and compares both fields against the manifest. load_export runs that check by default before reading the file back.

import tossd_reader as tossd

tossd.export("tossd_2024.parquet", years=2024)
tossd.verify_export("tossd_2024.parquet")

df = tossd.load_export("tossd_2024.parquet")
print(df.shape)
print(sorted(df.attrs["tossd_reader"]))

(474026, 55)
['created_at', 'package_version', 'years']

load_export sets df.attrs["tossd_reader"] to this same three-key shape on every call. An export is an unfiltered snapshot, so it carries no query key the way a get_tossd() or get_tossd_raw() result does. tossd.get_provenance(df) returns the payload as a deep copy, ensuring mutations to the returned dictionary do not affect df.attrs. See Verbs for get_provenance and its df.tossd.provenance() accessor equivalent.

verify_export checks only payload_sha256 and row_count. An export written by an older or newer package version can carry a different schema_hash without being corrupted, so that field sits outside the check.

Verify a parquet file previously written by export() against its manifest sidecar.

Two checks, both against <stem>.manifest.json beside path: the payload's sha256 hash (payload_sha256) and its row count (row_count). A schema_hash mismatch is deliberately not checked here -- an export written by an older or newer package version is not a corrupted file.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to a `.parquet` file previously written by `export()`.	required

Returns:

Type	Description
`None`	`None`. Only ever returns after every check passes.

Raises:

Type	Description
`ExportIntegrityError`	The manifest sidecar is missing or cannot be parsed as a JSON object; the payload's sha256 hash doesn't match the manifest's `payload_sha256`; or the payload's row count doesn't match the manifest's `row_count`.

Load a parquet file previously written by export(), with its provenance attached.

Reads the file directly (pq.read_table + to_pandas, reusing query's own types_mapper) rather than through get_tossd's pipeline: an exported parquet already carries snake_case column names and final arrow types (export() wrote it that way), so re-running _schema.apply_schema would be redundant at best and wrong if the installed package's schema has since drifted from the one that produced the file.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to a `.parquet` file previously written by `export()`.	required
`verify`	`bool`	Run `verify_export(path)` first (the default). Pass `False` to skip the payload-hash/row-count checks -- `df.attrs` provenance is still read from the manifest either way, so a missing or unparseable manifest still raises regardless of this flag.	`True`

Returns:

Type	Description
`DataFrame`	A `pandas.DataFrame` read straight from the parquet file, with
`DataFrame`	`df.attrs["tossd_reader"]` set to `{"package_version", "created_at",
`DataFrame`	"years"}`: the manifest's`tossd_reader_version`,`created_at`, and
`DataFrame`	`vintages` fields respectively.

Raises:

Type	Description
`ExportIntegrityError`	`verify=True` (the default) and the payload fails `verify_export`'s checks; or, regardless of `verify=`, the manifest sidecar is missing or cannot be parsed as a JSON object.

Example

import tossd_reader df = tossd_reader.load_export("tossd_2024.parquet") # doctest: +SKIP

Manifest validation when verify=False

verify=False skips the payload-hash and row-count checks. load_export still reads <stem>.manifest.json for df.attrs provenance, so a missing or unreadable manifest raises ExportIntegrityError either way.

A tampered or truncated file fails at the hash check. Flipping a single byte mid-file is enough.

with open("tossd_2024.parquet", "r+b") as f:
    f.seek(1_000_000)
    original_byte = f.read(1)
    f.seek(1_000_000)
    f.write(bytes([original_byte[0] ^ 0xFF]))

tossd.verify_export("tossd_2024.parquet")

ExportIntegrityError: tossd_2024.parquet does not match its manifest: sha256 86f01b27506eeaba3f4fb71fd98ffa2e910bb6fb663e8d6d88a48c219ade1c3e but the manifest recorded cbc990069898161335701299e172cd6ae39bbfef2d2543ec7044e914384b13e8. The file may have been modified or corrupted since export.

Bases: TossdReaderError

Raised when a previously exported parquet file fails verify_export's checks.

Covers a missing or unreadable manifest sidecar, a payload whose sha256 hash no longer matches the manifest's payload_sha256, and a payload whose row count no longer matches the manifest's row_count. A schema_hash difference (an export written by a different package version) is deliberately not one of these checks -- it doesn't mean the file was modified.

Performance and memory

Exporting the default full dataset (years=None, covering 2019 through 2024) materialises approximately 2.4 million rows in memory as an Apache Arrow table before writing to disk. Peak resident memory reaches roughly 4.4 GB. The finished table itself accounts for about 2.1 GB of that. Per-year tables stay alive alongside it until concatenation completes. Pass specific years to years= when working in memory-constrained environments, or pass max_rows= to enforce a maximum row threshold before writing to disk.

tossd.export("out", years=2019, max_rows=100_000)

ValueError: export() would write 290914 rows, exceeding max_rows=100000. Pass a larger max_rows=, or narrow years= to export a smaller slice.

The check runs after the table is built, so the error names the actual row count. It also runs before anything is written, so a rejected export leaves no partial file, and a pre-existing file at the target path is untouched.

Export

Generated filenames

Manifest fields

Performance and memory

Next