Query

TOSSD activity-level data, tracked by the International Forum on TOSSD (IFT), measures development finance across Pillar I (cross-border flows) and Pillar II (global public goods). The query module loads, filters, and types these records into pandas DataFrames.

Function	Output	Purpose
`get_tossd()`	`pandas.DataFrame`	Typed and filtered activity-level data. Primary entry point.
`get_available_filters()`	`dict[str, pandas.DataFrame]`	Dimension names, codes, and labels available for query filtering.
`get_codelists_version()`	`str`	Snapshot date of the packaged OECD and TOSSD codelists (ISO 8601 date).
`get_tossd_raw()`	`pandas.DataFrame`	Raw published data preserving original column names, types, and ordering.

For saving complete annual datasets to disk, see Export.

Usage

get_tossd() applies schema casting, row filtering, and derived columns (is_aggregate, unit, tossd_pillar, tossd_subpillar) across requested reporting years.

Provider codes, recipient codes, pillar categories, and reporting years each get their own DataFrame from get_available_filters().

Resolving a provider name

The providers and recipients arguments accept official numeric codes, string codes, or full names matched case-insensitively against the packaged codelist. Numeric strings evaluate as codes first and fall back to name matching. Unresolved strings raise UnknownCodeError with closest matching names.

import tossd_reader as tossd

tossd.get_tossd(years=2024, providers="Germny")

UnknownCodeError: 'Germny' did not match any providers code or name in the packaged codelist. Closest matches: Germany.

Filtering to pillars 1 and 2

pillars="standard" filters to pillars 1 and 2 together, excluding the pillar-0 placeholder rows a few years carry. Every other pillars= value already excludes pillar-0. Only the default pillars=None keeps it. See Pillars and aggregate rows for what those rows are.

import tossd_reader as tossd

len(tossd.get_tossd(years=2023, columns="minimal"))

len(tossd.get_tossd(years=2023, columns="minimal", pillars="standard"))

Filtering by dimension

filters= narrows results by any of seven dimensions with no dedicated keyword (sector, purpose, channel, modality, finance_instrument, financing_arrangement, framework_of_collaboration). Each key takes a single code, a single codelist name, or an iterable mixing both, resolved the same way providers= and recipients= resolve their own values. Filtering on more than one dimension at once narrows to rows matching every dimension given.

df = tossd.get_tossd(
    years=2024,
    columns="analysis",
    units="usd_million",
    filters={"sector": "I.2. Health"},
)
len(df)

round(df["usd_disbursement"].sum(), 1)

30794.2

Resolve a token before filtering with tossd.codes.lookup(dimension, token) to check what a code or name matches (see Codes below).

financing_arrangement and framework_of_collaboration match by token membership. Real data packs multiple codes into one row ("FA02|FA03"), so a filter for "FA02" also matches a packed row carrying it alongside another code. The other five dimensions match by exact equality.

fa = tossd.get_tossd(
    years=2024,
    columns="analysis",
    units="usd_million",
    filters={"financing_arrangement": "FA02"},
)
len(fa)

print(fa["financing_arrangement_code"].value_counts().head(2).to_string())

financing_arrangement_code
FA02         11557
FA02|FA03        4

provider, recipient, and pillar keys raise a ValueError, naming the dedicated keyword to use instead.

tossd.get_tossd(years=2024, filters={"provider": "France"})

ValueError: filters={'provider': ...} is not supported; use providers= directly.

An unrecognised dimension raises the same way, with closest matches.

tossd.get_tossd(years=2024, filters={"sectors": "x"})

ValueError: Unknown filters= dimension 'sectors'; expected one of channel, finance_instrument, financing_arrangement, framework_of_collaboration, modality, purpose, sector. Closest matches: sector.

An unresolved code or name raises UnknownCodeError with closest matching names.

tossd.get_tossd(years=2024, filters={"sector": "Helth"})

UnknownCodeError: 'Helth' did not match any sector code or name in the packaged codelist. Closest matches: I.2. Health, I.2.a. Health, general, I.2.b. Basic health, I.3. Population policies/programmes and reproductive health.

A codelist entry can resolve cleanly and still match no row. The packaged sector codelist carries sub-sector codes the published data folds into their top-level group before publishing, so a sub-sector filter resolves the code and returns nothing.

tossd.get_tossd(years=2024, filters={"sector": "I.2.b. Basic health"})

UserWarning: get_tossd's filters matched no rows; returning an empty (but correctly typed) frame. A codelist entry can sit at a finer granularity than the published data uses (sector sub-codes, for example, fold into their top-level group) -- compare against the column's own values, e.g. df['sector_code'].unique().

purpose_code carries that sub-sector-level detail instead. Filter on purpose for questions finer than the 25 top-level sector groups.

export() takes no filters=. Its contract is an unfiltered snapshot of a requested year (see Export). Build a filtered extract with get_tossd(filters=...) and your own DataFrame.to_parquet call.

Checking the forced columns

FORCED_COLUMNS names the columns present in every get_tossd result, regardless of columns=. Import it to check membership before building an explicit column list.

import tossd_reader as tossd

tossd.FORCED_COLUMNS

('year', 'tossd_pillar', 'tossd_subpillar', 'is_aggregate', 'unit')

An explicit columns= list still gets every forced column appended, so a multi-year query can group by year without naming it.

df = tossd.get_tossd(
    years=[2023, 2024],
    columns=["provider_name", "usd_disbursement"],
    units="usd_million",
)
list(df.columns)

['provider_name', 'usd_disbursement', 'year', 'tossd_pillar', 'tossd_subpillar', 'is_aggregate', 'unit']

df.groupby("year", observed=True)["usd_disbursement"].sum().round(1)

year
2023    472601.8
2024    497676.0

Converting units

units= accepts "usd_thousand" (default, as published), "usd_million", or "usd". The unit column records whichever option ran. export() takes no units= and always writes the published scale.

import tossd_reader as tossd

th = tossd.get_tossd(years=2024, columns="minimal")
us = tossd.get_tossd(years=2024, columns="minimal", units="usd")
print(round(th["usd_disbursement"].sum(), 1))

497675981.4

print(round(us["usd_disbursement"].sum(), 1))

497675981440.9

An unrecognised value raises ValueError naming the valid options.

tossd.get_tossd(years=2024, units="usd_billion")

ValueError: Unknown units 'usd_billion'; expected one of ('usd_thousand', 'usd_million', 'usd').

Excluding aggregate rows

include_aggregates=True (the default) keeps every row get_tossd() loads, including the provider_code == 0 pseudo-aggregate rows the publisher reports alongside individual providers' own records, so the output matches the published files in full. Pass include_aggregates=False to drop them before downstream operations aggregate the frame.

df = tossd.get_tossd(years=2024, columns="analysis", units="usd_million")

tossd.rank_entities(df, top=3, include_aggregates=True)[
    ["provider_code", "provider_name", "usd_disbursement", "share_pct", "rank"]
]

 provider_code   provider_name  usd_disbursement  share_pct  rank
             0       Aggregate      99379.609718  19.968737     1
           302   United States      67695.935324  13.602412     2
           918 EU Institutions      58667.476757  11.788288     3

tossd.rank_entities(df, top=3, include_aggregates=False)[
    ["provider_code", "provider_name", "usd_disbursement", "share_pct", "rank"]
]

 provider_code   provider_name  usd_disbursement  share_pct  rank
           302   United States      67695.935324  16.996373     1
           918 EU Institutions      58667.476757  14.729604     2
             4          France      25444.627005   6.388365     3

Passed to get_tossd() itself, include_aggregates=False removes the same rows one step earlier, before any grouping or ranking runs. Every tossd_reader.verbs function defaults include_aggregates=False on its own (see Verbs), so a query already excluding aggregates needs no further filtering before ranking, comparing, or totalling.

Inspecting available filters

import tossd_reader as tossd

filters = tossd.get_available_filters()
print(filters["pillar"].to_string(index=False))

code        name  tossd_only
   1    Pillar I        True
   2   Pillar II        True
  21 Pillar II.A        True
  22 Pillar II.B        True

Reading back query provenance

get_tossd() stamps df.attrs["tossd_reader"] with the call's own provenance, including the package version, a UTC timestamp, the normalised query, and each fetched year's etag, retrieval time, and source URL. get_provenance(df) (documented in Verbs) returns a deep copy of that dict.

import tossd_reader as tossd
from pprint import pprint

df = tossd.get_tossd(years=2024, columns="analysis", units="usd_million")
pprint(tossd.get_provenance(df))

{'created_at': '2026-09-02T12:23:44.126187+00:00',
 'package_version': '0.1.0',
 'query': {'columns': 'analysis',
           'filters': {},
           'include_aggregates': True,
           'pillars': None,
           'providers': None,
           'recipients': None,
           'refresh': False,
           'units': 'usd_million',
           'years': (2024,)},
 'years': {'2024': {'etag': '"69e6ac8d-5728379"',
                    'retrieved_at': '2026-09-02T12:21:52.639935+00:00',
                    'url': 'https://tossd.online/tossddata_2024.parquet'}}}

df.tossd.provenance() returns the identical dict. get_tossd_raw() sets the same key with a minimal query dictionary containing "years" and "refresh", the only two keywords it takes.

tossd_reader.verbs functions and df.tossd accessor methods copy df.attrs onto their results, so provenance survives rank_entities, explode_sdg, and the rest. Plain pandas operations like merge, concat, or some groupby calls can drop attrs. Read provenance from the query result itself, or early in the pipeline.

Passing filter arguments to `get_tossd_raw`

get_tossd_raw accepts only years and refresh. Any other keyword raises a TypeError naming it and pointing at get_tossd.

import tossd_reader as tossd

tossd.get_tossd_raw(years=2024, providers="France")

TypeError: get_tossd_raw() got unexpected keyword argument(s): providers. get_tossd_raw() only accepts years=/refresh=; for filtering, column selection, or unit conversion, use get_tossd() instead.

Return typed, filtered TOSSD activity-level data.

Parameters:

Name	Type	Description	Default
`years`	`int \| Iterable[int] \| None`	A single year, an iterable of years (a `range` included, so `range(2022, 2024)` requests 2022 and 2023 only — Python's usual exclusive-end semantics), or `None` (the default) for the packaged known-years set. A year outside that set is honoured when the publisher's discovery sweep currently lists it, otherwise raises naming the years available right now.	`None`
`providers`	`int \| str \| Iterable[int \| str] \| None`	Filter to one or more providers. An `int` (or iterable of `int`) is a provider code. A `str` is matched case-foldedly against the packaged provider codelist's `name` column; a digit-string (e.g. `"302"`) tries an exact code match first, then falls back to a name match. `None` (the default) applies no provider filter.	`None`
`recipients`	`int \| str \| Iterable[int \| str] \| None`	Same resolution rules as `providers`, against the packaged recipient codelist.	`None`
`pillars`	`int \| str \| None`	`1`/`"1"`/`"I"` (case-insensitive) for pillar 1; `2`/`"2"`/`"II"` for pillar 2 (both sub-pillars and untagged pillar-2 rows); `21`/`"21"`/`"II.A"` or `22`/`"22"`/`"II.B"` for one sub-pillar specifically; `"standard"` for pillars 1 and 2 together (still excludes pillar-`0` placeholder rows, unlike `None`). `None` (the default) applies no pillar filter — pillar-`0` placeholder rows (a 2020-2023 publisher artefact) are then included; every other `pillars=` value excludes them. A sub-pillar filter combined with an explicit `years` that includes any year before 2023 raises `InvalidPillarError` (sub-pillar tagging did not exist yet, bar a 24-row 2022 trace); with the default `years=None` it instead silently narrows to years >= 2023, with one warning. Independent of this filter, the output `tossd_subpillar` column is `NA` unless a row carries a real `"21"`/`"22"` tag -- pillar-1 rows, untagged pillar-2 rows, and pillar-0 rows all read `NA`, so `.notna()` identifies sub-pillar-tagged rows specifically, a narrower set than pillar-2 rows overall.	`None`
`filters`	`dict[str, int \| str \| Iterable[int \| str]] \| None`	A dict filtering any of seven dimensions: `sector`, `purpose`, `channel`, `modality`, `finance_instrument`, `financing_arrangement`, `framework_of_collaboration`. Each value is a single code/name or an iterable of them, resolved the same way `providers=`/`recipients=` are: an exact code match against the packaged codelist first, then a case-folded name match, else `UnknownCodeError` naming the closest matches. `provider`/`recipient`/`pillar` keys raise, naming the dedicated kwarg to use instead (`providers=`/ `recipients=`/`pillars=`) -- this dict is for the seven dimensions with no kwarg of their own. `None` (the default, or an empty dict) applies no such filter. `codes.lookup()` resolves a token through this exact path, so a lookup and a filter built from its result can never disagree.	`None`
`columns`	`Literal['all', 'minimal', 'analysis'] \| list[str]`	`"all"` (default, every packaged column), `"minimal"`, `"analysis"` (see `_schema.preset_columns`), or an explicit `list[str]` of snake_case column names. `FORCED_COLUMNS` (`year`, `tossd_pillar`, `tossd_subpillar`, `is_aggregate`, and `unit`) are always present in the result regardless of this selection.	`'all'`
`units`	`Literal['usd_thousand', 'usd_million', 'usd']`	`"usd_thousand"` (default, as published), `"usd_million"` (divides every `schema.csv` `is_usd_thousand_amount` column by 1000), or `"usd"` (multiplies the same columns by 1000, since the published scale is thousands).	`'usd_thousand'`
`include_aggregates`	`bool`	`True` (the default) keeps every row, including the `provider_code == 0` pseudo-aggregate rows the publisher includes alongside individual providers' own activity-level records -- so the default output matches the published records in full. `False` drops them (an arrow-level `provider_code != 0` filter, applied per year, before the cross-year concat). The `tossd_reader.verbs` aggregation functions default the other way (`include_aggregates=False`), since summing across both would double-count.	`True`
`refresh`	`bool`	Re-run discovery's HEAD sweep and force a readerkit conditional GET for every requested year. An enclosing `readerkit.refresh_scope()` has the same effect.	`False`

Returns:

Type	Description
`DataFrame`	A `pandas.DataFrame`, one row per activity matching every filter,
`DataFrame`	across every requested year. An empty result (after filtering)
`DataFrame`	still comes back correctly typed, with one warning. `df.attrs
`DataFrame`	["tossd_reader"]` carries this call's own provenance (the
`DataFrame`	normalised query, plus each fetched year's etag/retrieved_at/url)
`DataFrame`	-- read it back with `get_provenance(df)` rather than the raw
`DataFrame`	`attrs` key directly.

Raises:

Type	Description
`ValueError`	`units` is not `"usd_thousand"`/`"usd_million"`/`"usd"`; `columns` names an unknown column (or an unrecognised preset); `years` resolves to an empty set of years; a requested year is not currently published and nothing is cached for it; `filters` names an unrecognised dimension (or one of `provider`/`recipient`/`pillar`, naming the dedicated kwarg instead); or `refresh=True` while offline mode is active (`config.get_offline()` is `True`).
`UnknownCodeError`	A `providers`/`recipients`/`filters` token (name, code, or digit-string) does not match the packaged codelist.
`InvalidPillarError`	A sub-pillar filter is requested for an explicit year before 2023.
`TossdNetworkError`	The publisher is unreachable and nothing usable is cached for a requested year.
`SchemaDriftError`	A requested year's published file no longer matches the packaged schema.

Example

import tossd_reader df = tossd_reader.get_tossd( # doctest: +SKIP ... years=2024, filters={"modality": "B02"} ... ) sorted(df["modality_code"].unique()) # doctest: +SKIP ['B02']

Return every get_tossd filter dimension a user can browse.

Covers every packaged codelist dimension (name/code/tossd-only-flag frames for provider, recipient, pillar, financing_arrangement, framework_of_collaboration, purpose, sector, channel, modality, and finance_instrument -- sector additionally carries a source column, and every dimension but pillar gains an in_published_data column once the packaged snapshot has been annotated) plus years, the packaged known-years set.

Returns:

Type	Description
`dict[str, DataFrame]`	`{dimension: frame}`, one entry per packaged codelist dimension plus
`dict[str, DataFrame]`	a `"years"` entry (a single-column `year` frame built from
`dict[str, DataFrame]`	`_discovery.known_years()`).

Return the packaged codelist snapshot's fetch date (ISO, date only).

Returns:

Type	Description
`str`	The `_version.json` `fetched_at` stamp's date portion, e.g.
`str`	`"2026-08-28"`.

Return raw TOSSD activity-level data, exactly as published.

Publisher column names, dtypes, and column order, verbatim -- no renaming, no dtype casting, no unit conversion.

Parameters:

Name	Type	Description	Default
`years`	`int \| Iterable[int] \| None`	A single year, or an iterable of years (a `range` included). `None` (the default) fetches the packaged known-years set.	`None`
`refresh`	`bool`	Re-run discovery's HEAD sweep and force a readerkit conditional GET for every requested year. An enclosing `readerkit.refresh_scope()` has the same effect.	`False`
`**unexpected`	`object`	Not a real parameter -- captures any other keyword argument only to raise a teaching `TypeError` naming it, since `get_tossd_raw` has no filtering, column selection, or units options for a caller to reach for by mistake.	`{}`

Returns:

Type	Description
`DataFrame`	A `pandas.DataFrame`, one row per activity, across every requested
`DataFrame`	year. `df.attrs["tossd_reader"]` carries this call's own provenance
`DataFrame`	(`{"years", "refresh"}` under `"query"` -- `get_tossd_raw` has no
`DataFrame`	other kwargs to normalise -- plus each fetched year's
`DataFrame`	etag/retrieved_at/url); read it back with `get_provenance(df)`.

Raises:

Type	Description
`ValueError`	`years` resolves to an empty set of years, or `refresh=True` while offline mode is active (`config.get_offline()` is `True`).
`TypeError`	An unrecognised keyword argument was passed. Filtering (`providers=`/`recipients=`/`pillars=`), column selection (`columns=`), and unit conversion (`units=`) all live on `get_tossd()`, not here.

Codes

tossd_reader.codes browses and resolves the packaged codelists that filters=, providers=, and recipients= all draw on. browse(dimension) returns one dimension's full codelist frame. lookup(dimension, token) resolves a single code or name to the packaged code, through the same resolution path a filter uses, so a lookup and a filter built from its result always agree.

import tossd_reader as tossd

m = tossd.codes.browse("modality")
print(m.head(4).to_string(index=False))

code                                                                     name  tossd_only  in_published_data
   A                                                           Budget support        True              False
 A00                                                           Budget support        True               True
   B                       Core contributions and pooled programmes and funds        True              False
 B01 Core support to NGOs, other private bodies, PPPs and research institutes        True               True

Every dimension's frame carries code, name, and tossd_only. Every dimension but pillar also carries in_published_data, marking whether the code occurs in the published data. provider and recipient add iso3, and sector adds source.

browse() covers all 10 packaged dimensions, pillar included. lookup() covers the 9 that resolve to a flat code, every filters= dimension plus provider and recipient, and returns int for the int-coded dimensions (provider, recipient, sector, purpose, channel, finance_instrument) or str for the str-coded ones (modality, financing_arrangement, framework_of_collaboration).

tossd.codes.lookup("sector", "I.2.b. Basic health")

tossd.codes.lookup("provider", "France")

tossd.codes.lookup("modality", "B02")

'B02'

lookup() excludes pillar. Resolve a pillar token like "II.A" through get_tossd(pillars=...) instead.

tossd.codes.lookup("pillar", "II.A")

ValueError: Unknown lookup() dimension 'pillar'; expected one of provider, recipient, sector, purpose, channel, modality, finance_instrument, financing_arrangement, framework_of_collaboration. Pillar tokens ('1', 'II.A', ...) resolve via get_tossd(pillars=...), not codes.lookup().

Return one packaged codelist dimension's frame, as a fresh copy.

Parameters:

Name	Type	Description	Default
`dimension`	`str`	One of the packaged dimension names -- `provider`, `recipient`, `pillar`, `sector`, `purpose`, `channel`, `modality`, `finance_instrument`, `financing_arrangement`, `framework_of_collaboration`.	required

Returns:

Type	Description
`DataFrame`	`codelists.load_codelist(dimension)`'s frame: `code`, `name`, and
`DataFrame`	`tossd_only` columns (plus `iso3` for `provider`/`recipient`;
`DataFrame`	`source` for `sector`; `in_published_data` once the packaged
`DataFrame`	snapshot has been annotated -- pillar excepted, its rows are
`DataFrame`	structural, and the annotation scans flat data columns pillar does
`DataFrame`	not have).

Raises:

Type	Description
`ValueError`	`dimension` is not one of the packaged dimensions.

Example

import tossd_reader modality = tossd_reader.codes.browse("modality") modality.loc[modality["code"] == "B02", "name"].item() 'Core contributions to multilateral institutions'

Resolve one code/name token to its packaged code, in the type filters use.

Goes through _matching.resolve_one_code -- the same building block get_tossd's providers=/recipients=/filters= resolve each of their own tokens with, so this can never disagree with what a filter built from token would match.

Parameters:

Name	Type	Description	Default
`dimension`	`str`	One of `LOOKUP_DIMENSIONS`.	required
`token`	`int \| str`	A code, a digit-string code, or a name (case-folded for the name match) -- same resolution rules as `providers=`/ `recipients=`/`filters=`'s own values.	required

Returns:

Type	Description
`int \| str`	The resolved code: `int` for `provider`/`recipient`/`sector`/
`int \| str`	`purpose`/`channel`/`finance_instrument`; `str` (the packaged
`int \| str`	codelist's own code, e.g. `"B02"`) for `modality`/
`int \| str`	`financing_arrangement`/`framework_of_collaboration`.

Raises:

Type	Description
`ValueError`	`dimension` is not one of `LOOKUP_DIMENSIONS` (naming the valid ones; `"pillar"` points at `pillars=` instead, since a pillar token doesn't resolve to a flat code).
`UnknownCodeError`	`token` matches no code or name in `dimension`'s packaged codelist.

Example

import tossd_reader tossd_reader.codes.lookup( ... "modality", "Core contributions to multilateral institutions" ... ) 'B02' tossd_reader.codes.lookup("sector", "110") 110

Verbs. Aggregation functions (rank_entities, compare_years, and others) built on get_tossd() output.
How to look up codes and names. Browse or resolve any dimension's codes with tossd_reader.codes, then pass the result to get_tossd.
Read the published columns unchanged. Compare get_tossd_raw with typed get_tossd outputs.
Columns, presets, and units. The full column surface, including FORCED_COLUMNS.
Export. Write normalised parquet extracts with export manifests.

Query

Usage

Resolving a provider name

Filtering to pillars 1 and 2

Filtering by dimension

Checking the forced columns

Converting units

Excluding aggregate rows

Inspecting available filters

Reading back query provenance

Passing filter arguments to get_tossd_raw

Codes

Next

Passing filter arguments to `get_tossd_raw`