Skip to content

Configuration, warnings, and errors

The package exposes two independent settings. Cache location (set_cache_dir/get_cache_dir, with TOSSD_READER_CACHE_DIR as the environment override) controls where downloaded vintages live. Offline mode (set_offline/get_offline, with TOSSD_READER_OFFLINE as its own environment override) controls whether a query is allowed to touch the network at all. cache_info() and clear_cache() inspect and prune what's actually on disk.

Re-point tossd_reader's cache to path.

Parameters:

Name Type Description Default
path str | Path | None

Explicit cache-directory override, taking precedence over TOSSD_READER_CACHE_DIR. None switches to an ephemeral, session-scoped bypass instead: fetches go to a TemporaryDirectory for the life of the process (or until set_cache_dir is called again), and nothing written during the bypass persists.

 required

Resolve the effective cache directory right now.

Re-reads TOSSD_READER_CACHE_DIR on every call (unless an explicit set_cache_dir override is active), so environment changes between calls take effect without any extra reset step.

Returns:

Type Description
Path | None

The resolved, existing, writable directory, or None when
Path | None

set_cache_dir(None) put the module into ephemeral bypass mode.

Cache location and bounds

The default cache location depends on the operating system:

  • macOS: ~/Library/Caches/readerkit/v1/tossd-reader/1
  • Linux: ~/.cache/readerkit/v1/tossd-reader/1
  • Windows: %LOCALAPPDATA%\readerkit\Cache\v1\tossd-reader\1

Resolution precedence, in order of highest priority, evaluates set_cache_dir(path), the TOSSD_READER_CACHE_DIR environment variable, the BBLOCKS_CACHE_DIR environment variable, and the platform default directory.

TOSSD_READER_CACHE_DIR overrides the platform default and is re-read on every query. Changes to the environment variable take effect immediately without requiring a cache reset.

The cache retains the newest 24 artifacts or up to 4 GB of data, whichever limit is reached first. Both limits are fixed.

ETag discovery and network resolution

Discovery runs a HEAD request sweep to identify candidate ETag values for requested years. The ETag returned by the subsequent GET response is authoritative. When the GET ETag differs from the candidate, the download retries under the corrected key (up to two total attempts). If the ETag continues changing across both attempts, the fetch raises TossdNetworkError detailing the observed values.

When neither HEAD nor GET responses provide an ETag header, the downloaded vintage is stored under an unknown key with a warning. Passing refresh=True (or using readerkit.refresh_scope()) forces a fresh download.

Override tossd_reader's offline mode.

Parameters:

Name Type Description Default
flag bool | None

True forces offline mode on; False forces it off, even if TOSSD_READER_OFFLINE is set. None resets to env-var-driven resolution -- the state set_offline has never been called at all.

 required

Resolve whether tossd_reader is in offline mode right now.

Re-reads TOSSD_READER_OFFLINE on every call (unless an explicit set_offline override is active), the same precedence get_cache_dir follows for the cache directory.

Returns:

Type Description
bool

True when offline mode is active: no fetch touches the network -- every requested
bool

vintage is served from the local cache instead (a loud warning either way, the same one
bool

an unplanned network outage triggers), or TossdNetworkError is raised when nothing
bool

usable is cached. False (the default, with no override and no env var set) leaves
bool

fetch behaviour untouched. False too when the env var is set but unrecognised (neither
bool

truthy nor recognised-falsy) -- that case also warns once per process, naming the value
bool

seen, since the caller likely meant to enable offline mode and didn't.

Offline mode

TOSSD_READER_OFFLINE recognises 1, true, and yes (case-insensitive) as true, and 0, false, and no as false. Unset or empty stays false silently. Anything else (a typo like on, or a stray 2) still resolves to False, but warns once per process, since a caller who set the variable almost certainly meant to turn offline mode on.

TOSSD_READER_OFFLINE=on python analysis.py

UserWarning: TOSSD_READER_OFFLINE='on' is not a recognized value; offline mode is NOT active. Recognised truthy values are 1, true, yes (case-insensitive).

set_offline(True) or set_offline(False) overrides the environment variable outright, for the rest of the process. set_offline(None) resets to reading TOSSD_READER_OFFLINE again, the same state as never having called set_offline at all.

With offline mode active, a fetch that has a cached vintage to fall back on serves it and warns, naming the retrieval timestamp and ETag.

import tossd_reader as tossd

tossd.set_offline(True)
df = tossd.get_tossd(years=2024, columns="minimal")

UserWarning: Offline mode is active (tossd_reader.config.set_offline(False), or the TOSSD_READER_OFFLINE env var, would allow network access); serving the cached 2024 vintage retrieved 2026-09-02T12:21:52.639935+00:00 (etag "69e6ac8d-5728379").

refresh=True needs the network by definition, and offline mode exists to rule the network out, so combining the two raises ValueError naming the conflict. The same check guards get_tossd(), get_tossd_raw(), export(), and get_vintages(), with the same message shape aside from the leading function name.

tossd.set_offline(True)
tossd.get_tossd(years=2024, refresh=True)

ValueError: get_tossd(refresh=True) conflicts with offline mode (config.get_offline() is True): a forced refresh needs the network. Call tossd_reader.config.set_offline(False) first, or omit refresh=True.

The message spells out the full module path. tossd.set_offline(False) at the top level does the same thing.

List every locally cached vintage, superseded ones included -- start here to free space.

One row per get_cache().entries() entry: a year with more than one cached vintage (a republished file downloaded again under a new ETag) gets one row per vintage, not one per year. Call clear_cache() directly to drop everything but each year's newest row here.

Returns:

Type Description
DataFrame

year (parsed from the cache key; None for a foreign entry that doesn't match this
DataFrame

package's own key shape -- should not happen in practice), etag, retrieved_at (both
DataFrame

read from that vintage's own provenance sidecar, None when the sidecar is missing or
DataFrame

corrupt), downloaded_at (a tz-aware datetime, from the cache's own sidecar metadata
DataFrame

-- unlike retrieved_at, always present), size_bytes, path. Empty (with these
DataFrame

columns) when nothing is cached, including in ephemeral bypass mode
DataFrame

(set_cache_dir(None)).

Free local cache space. The bare clear_cache() call drops only superseded vintages.

Parameters:

Name Type Description Default
years int | Iterable[int] | None

Restrict to these years. None (the default) considers every cached year.

 None
before str | date | datetime | None

Restrict to vintages retrieved before this point (a date, a datetime, or an ISO 8601 string -- a bare date/date-only string means midnight UTC that day; a naive datetime/datetime string is treated as UTC). Compared against each entry's own provenance retrieved_at, falling back to the cache's downloaded_at when the sidecar is missing or corrupt (see cache_info()). None (the default) applies no date filter.

 None
keep_latest bool

True (the default) never removes the single newest entry for a year, even one that otherwise matches years=/before= -- so the bare clear_cache() call (every argument at its default) removes exactly the superseded, non-newest vintages, freeing space while a fetch for any already-downloaded year still serves instantly from cache. False removes every entry matching years=/before=, the newest included -- clear_cache(keep_latest=False) with no other arguments empties the whole cache.

 True

Returns:

Type Description
int

The number of cache entries removed.

Raises:

Type Description
OSError

A filesystem fault (e.g. a permission error) unlinking an entry's payload, sidecar, or provenance file propagates as-is, rather than being swallowed per-entry -- a silent under-deletion would be worse than an abrupt stop. The cache may be left partially cleared; the removed count from a prior, non-raising call is the only way to know how much of a partial run actually completed.

Inspecting and clearing the cache

cache_info() lists every locally cached vintage, producing one row per vintage. A year re-downloaded after a republish appears twice, once for each retrieved ETag. path is a local absolute path. The example below drops it to fit the page.

import tossd_reader as tossd

tossd.cache_info().drop(columns=["path"])

   year                etag                      retrieved_at                    downloaded_at  size_bytes
0  2019  "69e6ac86-347a653"  2026-08-28T21:14:14.414671+00:00 2026-09-02 08:05:04.508371+00:00    55027283
1  2021  "69e6ac8b-4112f49"  2026-08-28T21:14:42.382179+00:00 2026-08-28 21:14:42.356395+00:00    68235081
2  2020  "69e6ac8a-3c90446"  2026-08-28T21:14:28.358434+00:00 2026-08-28 21:14:28.331745+00:00    63505478
3  2022  "69e6ac8b-4e95ca1"  2026-08-28T21:14:57.492221+00:00 2026-08-28 21:14:57.461650+00:00    82402465
4  2024  "69e6ac8d-5728379"  2026-08-28T19:32:28.617740+00:00 2026-08-28 19:32:28.582136+00:00    91390841
5  2023  "69e6ac8c-56469db"  2026-08-28T21:15:12.643113+00:00 2026-08-28 21:15:12.608318+00:00    90466779

etag and retrieved_at come from that vintage's own provenance sidecar and read None if the sidecar is missing or corrupt. downloaded_at comes from the cache's own metadata, always present even when the sidecar is missing or corrupt.

clear_cache() with no arguments removes superseded vintages where a newer download for the same reporting year already exists. A cache holding exactly one vintage per year contains no superseded entries, so the bare call removes nothing.

tossd.clear_cache()

0

keep_latest protects each year's single newest entry even when years= or before= matches it. Pass keep_latest=False to remove that protection and empty the cache entirely when passed no other arguments.

tossd.clear_cache(keep_latest=False)

6

len(tossd.cache_info())

0

before= accepts a date, a datetime, or an ISO 8601 string. A naive value (no timezone) is treated as UTC. Every call returns the number of entries removed.

Partial cache clearance on filesystem errors

A filesystem fault removing an entry (such as a permission error) propagates as OSError instead of being skipped per-entry. The cache can be left partially cleared. The removed count from a prior, non-raising call is the only record of how much of a partial run completed.

Warnings

Source Condition Frequency
Discovery Publisher lists a reporting year outside the packaged known years. Once per newly seen year per process.
Discovery get_vintages() falls back to the local cache instead of a live sweep, because offline mode is active or the publisher host is unreachable. Emitted on every affected call.
Fetch and cache Publisher host is unreachable, offline mode is active, or a requested year is no longer published, serving a cached vintage instead. Emitted on every affected call.
Fetch and cache Neither HEAD nor GET responses returned an ETag for a year. Once per year per process.
Fetch and cache Provenance sidecar exists but contains invalid JSON, causing null vintage fields in export manifests. Emitted on every read of the corrupted file.
Config TOSSD_READER_OFFLINE is set to a value that's neither a recognised truthy nor falsy spelling. Once per process.
Query Query filters match zero records. Emitted on every empty result call.
Query Sub-pillar filter (pillars=21 or "II.A") with default years=None narrows to 2023 onward. Once per process.
Query Sub-pillar filter includes 2023, where sub-pillar reporting coverage is partial. Once per process.
Query Unmapped parent_channel_code value encountered during parent_channel_name resolution. Once per newly seen code per process.
Schema check Published dataset contains a column not defined in schema.csv (visible under columns="all"). Once per newly seen column per process.

Errors

All package exceptions inherit from TossdReaderError. Catching TossdReaderError intercepts all package-specific exceptions. Selected argument- and state-validation failures raise a plain ValueError instead, including the offline/refresh conflict described above.

Bases: Exception

Base for every error tossd_reader raises.

Bases: TossdReaderError

Raised when the publisher is unreachable and no usable cached vintage exists.

Attributes:

Name Type Description
cache_dir

The cache directory that was checked for a usable vintage, or None when the cache is in ephemeral bypass mode (set_cache_dir(None)).

__init__(message, *, cache_dir=None)

Parameters:

Name Type Description Default
message str

Human-readable description, already naming the cache directory and offline state.

 required
cache_dir Path | None

The cache directory that was checked for a usable vintage, or None in ephemeral bypass mode.

 None

Bases: TossdReaderError

Raised when a freshly downloaded vintage fails structural validation.

Never raised for an already-cached vintage; validation runs once, on a new download only.

Attributes:

Name Type Description
year

The reporting year of the rejected vintage.
url

The URL the vintage was downloaded from.

__init__(message, *, year, url)

Parameters:

Name Type Description Default
message str

Human-readable description naming the year and url.

 required
year int

The reporting year of the rejected vintage.

 required
url str

The URL the vintage was downloaded from.

 required

Bases: TossdReaderError

Raised when a published file's columns no longer match the packaged schema.

Covers three cases: two published columns whose names normalise to the same key, a schema-expected column absent from the file, and a value that cannot be cast to its schema.csv target_dtype. The message names the column, and the offending value where there is one.

Bases: TossdReaderError

Raised when a providers=/recipients= value cannot be resolved.

The message names up to five closest matches from the packaged codelist.

Bases: TossdReaderError

Raised when a sub-pillar filter is requested for a year that cannot support it.

A sub-pillar filter (pillars=21/22/"II.A"/"II.B") combined with an explicit year before 2023 raises this, naming the years that predate sub-pillar tagging.

Bases: TossdReaderError

Raised when a previously exported parquet file fails verify_export's checks.

Covers a missing or unreadable manifest sidecar, a payload whose sha256 hash no longer matches the manifest's payload_sha256, and a payload whose row count no longer matches the manifest's row_count. A schema_hash difference (an export written by a different package version) is deliberately not one of these checks -- it doesn't mean the file was modified.

Next