Skip to content

Verbs

tossd_reader.verbs aggregates a get_tossd()-shaped frame along one dimension (rank_entities, compare_years, sdg_totals, keyword_totals, and subpillar_breakdown). Each takes a DataFrame as its first argument, keyword-only arguments after that, and returns a new DataFrame, leaving the input frame untouched. Every one of the five is also a df.tossd accessor method. See The df.tossd accessor below.

The same module carries two functions that read a frame's own state. get_provenance returns its query record, and reconcile returns a one-row read-out of the checks in How to check a figure against the published total. Both take only df, no keyword arguments, and both are df.tossd methods too, spelled provenance() and reconcile().

The five aggregation verbs share three conventions.

Each drops the provider_code == 0 pseudo-aggregate rows before aggregating, with include_aggregates=False as the default. get_tossd() keeps them by default (see Excluding aggregate rows) to reproduce the published records in full, while the verb layer sums individual providers' activity.

value= names the amount column each verb sums, "usd_disbursement" on most verbs and "usd_disbursement_deflated" on compare_years.

df.attrs propagates. Every verb copies the input frame's attrs onto its result.

top= and share_pct vary by verb:

verb top= share_pct
rank_entities yes yes
sdg_totals yes yes
subpillar_breakdown no yes
compare_years no no
keyword_totals no no

Where share_pct appears, it represents the share of the included total, from 0 to 100, unrounded, computed over the full grouped result before any top= argument truncates the rows. The total reflects the active include_aggregates setting and any filters present on the frame. A call without top= returns share_pct values that sum to 100. A call with top= returns a truncated slice of that ranking, whose values sum to less than 100.

include_aggregates in practice

import tossd_reader as tossd

df = tossd.get_tossd(years=2024, columns="analysis", units="usd_million")

tossd.rank_entities(df, top=3, include_aggregates=True)[
    ["provider_code", "provider_name", "usd_disbursement", "share_pct", "rank"]
]
 provider_code   provider_name  usd_disbursement  share_pct  rank
             0       Aggregate      99379.609718  19.968737     1
           302   United States      67695.935324  13.602412     2
           918 EU Institutions      58667.476757  11.788288     3

The "Aggregate" pseudo-row outranks every real provider. The default excludes it.

tossd.rank_entities(df, top=3)[
    ["provider_code", "provider_name", "usd_disbursement", "share_pct", "rank"]
]
 provider_code   provider_name  usd_disbursement  share_pct  rank
           302   United States      67695.935324  16.996373     1
           918 EU Institutions      58667.476757  14.729604     2
             4          France      25444.627005   6.388365     3

df.tossd.rank_entities(top=3) returns the same frame. See The df.tossd accessor below.

Activity counts vs row counts

keyword_totals's n_rows counts rows whose keyword mask matches. The publisher's pre-split rows can carry different keyword tags from each other, so one activity spread across several rows counts once per matching row. rank_entities's n_activities counts distinct tossd_id values instead. The two measure different dimensions. Check the column name before comparing one verb's count against another.

get_provenance in practice

import tossd_reader as tossd
from pprint import pprint

df = tossd.get_tossd(years=2024, columns="analysis", units="usd_million")

pprint(tossd.get_provenance(df))
{'created_at': '2026-09-02T12:19:04.952853+00:00',
 'package_version': '0.1.0',
 'query': {'columns': 'analysis',
           'filters': {},
           'include_aggregates': True,
           'pillars': None,
           'providers': None,
           'recipients': None,
           'refresh': False,
           'units': 'usd_million',
           'years': (2024,)},
 'years': {'2024': {'etag': '"69e6ac8d-5728379"',
                    'retrieved_at': '2026-09-02T12:10:44.031565+00:00',
                    'url': 'https://tossd.online/tossddata_2024.parquet'}}}

get_provenance returns a deep copy of df.attrs["tossd_reader"], so mutating the returned dictionary leaves df untouched. get_tossd_raw() frames carry a minimal query dictionary (refresh and years only). A load_export() frame carries the three-key shape without a query key, reflecting an unfiltered snapshot.

loaded = tossd.load_export("exports/tossd_2019.parquet")

pprint(tossd.get_provenance(loaded))
{'created_at': '2026-09-02T12:10:39.819913+00:00',
 'package_version': '0.1.0',
 'years': {'2019': {'etag': '"69e6ac86-347a653"',
                    'retrieved_at': '2026-09-02T12:10:39.379359+00:00'}}}

The key survives every verb and accessor call above. Each one copies df.attrs onto its own result, so provenance survives rank_entities, explode_sdg, and the rest. Plain pandas operations like merge, concat, or some groupby calls can drop attrs. Calling get_provenance on a frame that does not carry the key raises ValueError.

import pandas as pd

tossd.get_provenance(pd.DataFrame({"a": [1]}))
ValueError: get_provenance() found no df.attrs['tossd_reader'] -- that
key is set by get_tossd(), get_tossd_raw(), and load_export(); a frame
built some other way (or a plain pandas operation that dropped attrs
along the way) carries none.

df.tossd.provenance() returns the identical dict and raises the same error.

reconcile in practice

import tossd_reader as tossd

df = tossd.get_tossd(years=2024, columns="analysis", units="usd_million")

df.tossd.reconcile()
unit                                 usd_million
n_aggregate_rows                            5626
aggregate_value                     99379.609718
aggregate_share_pct                    19.968737
usd_disbursement_total             497675.981441
usd_disbursement_deflated_total    497675.981441
pillars_present                           (1, 2)
year_min                                    2024
year_max                                    2024
n_years                                        1
has_provenance                              True
b02_core_contribution_value          6678.973842
b02_core_contribution_share_pct         1.342033
estimate_derived_value               2913.935663
estimate_derived_share_pct              0.585509
iso3_unmatched_value               147394.103994
iso3_unmatched_share_pct               29.616479
dtype: object

reconcile is a read-out that summarises dataset characteristics. Warnings and errors arise only when passed a frame that is not shaped by get_tossd(). Shares are calculated against df's own usd_disbursement total with aggregate rows included, which is why reconcile accepts no include_aggregates= argument. Retaining aggregate rows allows aggregate_share_pct to measure their proportion directly.

usd_disbursement_total equals usd_disbursement_deflated_total here because 2024 is the deflator base year. For multi-year frames or non-base years, these values diverge.

multi = tossd.get_tossd(years=[2023, 2024], columns="analysis")

multi.tossd.reconcile()
unit                                   usd_thousand
n_aggregate_rows                              11451
aggregate_value                    173447820.379903
aggregate_share_pct                       17.876098
usd_disbursement_total             970277829.269724
usd_disbursement_deflated_total    982043011.985994
pillars_present                           (0, 1, 2)
year_min                                       2023
year_max                                       2024
n_years                                           2
has_provenance                                 True
b02_core_contribution_value         12881067.080131
b02_core_contribution_share_pct            1.327565
estimate_derived_value               4259974.195728
estimate_derived_share_pct                 0.439047
iso3_unmatched_value               299091812.522857
iso3_unmatched_share_pct                  30.825378
dtype: object

usd_disbursement_deflated_total, b02_core_contribution_value, estimate_derived_value, and iso3_unmatched_value read pd.NA when their source column (usd_disbursement_deflated, modality_code, source_name, recipient_code respectively) is missing from df. estimate_derived_value sums usd_disbursement on rows whose source_name contains "estimate" (case-insensitive), providing a heuristic reading of the source name.

pillars_present reflects whatever pillar values are in df. The (0, 1, 2) tuple above comes from real 2023 rows. See Pillars and aggregates for pillar 0 definitions.

See How to check a figure against the published total for reading these entries against a specific published number.

Rank entities along one dimension by summed value.

Works for any dimension whose {dimension}_code/{dimension}_name columns are present in df -- "provider" (the default), "recipient", "sector", "purpose", "channel", or any other packaged dimension pair. Ranking providers across both pillars at once mixes Pillar I bilateral/multilateral outflows with Pillar II core contributions to multilateral institutions -- a double count, if the two are meant to be read as one figure.

Parameters:

Name Type Description Default
df DataFrame

A get_tossd()-shaped frame carrying {dimension}_code, {dimension}_name, and value.

required
dimension str

The dimension to group by -- any prefix with a matching {dimension}_code/{dimension}_name column pair in df.

'provider'
value str

The amount column to sum and rank by.

'usd_disbursement'
top int | None

Keep only the first top rows after ranking and sorting. None (the default) keeps every ranked row.

None
include_aggregates bool

False (the default) drops is_aggregate rows (the provider_code == 0 pseudo-aggregates) before ranking.

False

Returns:

Type Description
DataFrame

One row per distinct ({dimension}_code, {dimension}_name) pair

DataFrame

(an observed=True groupby -- an unused category never appears),

DataFrame

sorted by summed value descending: {dimension}_code,

DataFrame

{dimension}_name, value, n_activities (only when df carries

DataFrame

tossd_id -- distinct tossd_id per group, excluding the "0000"

DataFrame

placeholder used for bundled lines with no activity identifier, so

DataFrame

those lines are never counted as activities; omitted entirely,

DataFrame

rather than raising, when tossd_id isn't present), share_pct

DataFrame

(of the summed total across every included row, 0-100 scale,

DataFrame

unrounded), and rank (competition ranking -- tied entries share

DataFrame

the lower rank number -- 1-based, descending). Truncation to top

DataFrame

happens after ranking, so the numbering on the entities that made

DataFrame

the cut always matches what a caller would see without top set.

DataFrame

The "0000" placeholder also lands on a small share of real

DataFrame

providers' own rows in 2023-24 (bundled lines belonging to a real

DataFrame

provider, not just pseudo-aggregates), so n_activities can

DataFrame

slightly undercount those providers even after aggregate exclusion.

Raises:

Type Description
ValueError

df is missing {dimension}_code, {dimension}_name, or value; value is not numeric; or (when include_aggregates=False) df has no is_aggregate column.

Example

import pandas as pd from tossd_reader import rank_entities df = pd.DataFrame({ ... "provider_code": [1, 1, 2], ... "provider_name": ["Austria", "Austria", "France"], ... "usd_disbursement": [100.0, 50.0, 200.0], ... "is_aggregate": [False, False, False], ... }) result = rank_entities(df) result["provider_name"].tolist() ['France', 'Austria'] result["rank"].tolist() [1, 2]

Compare summed value year over year.

Recipients are not held constant across years -- only the provider cohort is, and only when cohort="consistent". A year-over-year rise or fall can still be a shift in which recipients each provider funded, not a change in how much any single provider gave.

Parameters:

Name Type Description Default
df DataFrame

A get_tossd()-shaped frame carrying year, provider_code, provider_name, and value.

required
value str

The amount column to sum per year.

'usd_disbursement_deflated'
cohort Literal['consistent', 'all']

"consistent" (the default) restricts every year to the (provider_code, provider_name) pairs present in every year df covers, so a year's total isn't inflated by a provider that only reported in some years. "all" disables that restriction -- every row counts, whichever years its provider happens to appear in.

'consistent'
include_aggregates bool

False (the default) drops is_aggregate rows before comparing years.

False

Returns:

Type Description
DataFrame

One row per year present in df: year, value, n_providers

DataFrame

(the cohort's size under "consistent"; the year's own distinct

DataFrame

provider-pair count under "all"), and pct_change (percent

DataFrame

change from the previous row's value, 0-100 scale, unrounded;

DataFrame

NA for the first -- or only -- year). `result.attrs

DataFrame

["structural_breaks"]carriesget_structural_breaks(years=...)`'s

DataFrame

rows intersecting the years covered, so a caller can flag a jump

DataFrame

against a known discontinuity instead of a real trend.

Raises:

Type Description
ValueError

df is missing year, provider_code, provider_name, or value; value is not numeric; cohort is not "consistent"/"all"; (when include_aggregates=False) df has no is_aggregate column; or cohort="consistent" and no provider pair is present in every year.

Example

import pandas as pd from tossd_reader import compare_years df = pd.DataFrame({ ... "year": [2019, 2020], ... "provider_code": [1, 1], ... "provider_name": ["Austria", "Austria"], ... "usd_disbursement_deflated": [100.0, 150.0], ... "is_aggregate": [False, False], ... }) result = compare_years(df) result["usd_disbursement_deflated"].tolist() [100.0, 150.0] result["pct_change"].tolist() [nan, 50.0]

Sum value by SDG tag, weighted for rows that carry more than one tag.

Internally explodes df with analysis.explode_sdg first, then sums value * sdg_weight per tag -- the weighted total this returns equals the SDG-tagged subset of df's value total, never df's grand total, since a row with no SDG tag at all contributes nothing to any group.

Parameters:

Name Type Description Default
df DataFrame

A get_tossd()-shaped frame carrying sdg_codes_raw and value.

required
value str

The amount column to sum.

'usd_disbursement'
level Literal['goal', 'code']

"goal" (the default) groups by SDG goal (sdg_goal); "code" groups by the exact token published in sdg_codes_raw (sdg_code), keeping goals and targets apart. The publisher formats some goal-level tags as a trailing-.0 token (e.g. "5.0", no real SDG target is numbered .0); at level="code" each such token gets its own row, while level="goal" folds it into its goal like any other goal-level tag.

'goal'
top int | None

Keep only the first top rows after ranking and sorting. None (the default) keeps every row.

None
include_aggregates bool

False (the default) drops is_aggregate rows before exploding.

False

Returns:

Type Description
DataFrame

One row per distinct goal or code (matching level): sdg_goal

DataFrame

or sdg_code, value (the weighted sum), share_pct (of the

DataFrame

SDG-tagged total across every included row, 0-100 scale,

DataFrame

unrounded), and rank (competition ranking, descending,

DataFrame

1-based). Truncation to top happens after ranking.

Raises:

Type Description
ValueError

value is missing or not numeric; level is not "goal"/"code"; df is missing sdg_codes_raw or already carries explode_sdg's own output columns (both raised by sdg_totals() itself, naming sdg_totals() -- not a delegated explode_sdg message); or (when include_aggregates=False) df has no is_aggregate column.

Example

import pandas as pd from tossd_reader import sdg_totals df = pd.DataFrame({ ... "sdg_codes_raw": ["1;4", "4"], ... "usd_disbursement": [100.0, 100.0], ... "is_aggregate": [False, False], ... }) result = sdg_totals(df, level="goal") result["sdg_goal"].tolist() [4, 1] result["usd_disbursement"].tolist() [150.0, 50.0]

Sum value per keyword marker, plus a union row combining them.

Marker masks are recomputed internally from keywords_raw via analysis.extract_keywords -- any kw_* columns already in df are ignored, so a caller who filtered or otherwise altered df since running extract_keywords never gets stale marker flags here.

Parameters:

Name Type Description Default
df DataFrame

A get_tossd()-shaped frame carrying keywords_raw and value.

required
markers str | Iterable[str] | None

One marker name, an iterable of them, or None (the default) for all 12 packaged markers. Each name is matched with or without its kw_ prefix ("gender" and "kw_gender" both match); an unrecognised name raises, naming the closest matches.

None
value str

The amount column to sum.

'usd_disbursement'
include_aggregates bool

False (the default) drops is_aggregate rows before summing.

False

Returns:

Type Description
DataFrame

One row per requested marker, in the order given (packaged order

DataFrame

when markers=None), plus a final "Combined" row: marker,

DataFrame

value, n_rows (a row count of the marker's mask, not a

DataFrame

distinct-tossd_id count -- the publisher's pre-split rows can

DataFrame

spread one activity across several rows, and those rows can

DataFrame

genuinely carry different keyword tags from each other, so a

DataFrame

multi-row activity counts once per matching row rather than once

DataFrame

overall, and no distinct-ID count is offered here). "Combined"'s

DataFrame

mask is the union of every requested marker's mask, so the

DataFrame

individual marker rows sum to at least "Combined"'s total, never

DataFrame

less -- a row tagged with more than one requested marker is

DataFrame

counted once per marker row but only once in "Combined", which

DataFrame

is the double count the union row exists to avoid.

Raises:

Type Description
ValueError

df is missing keywords_raw (naming any already present kw_* columns as the likely cause) or value; value is not numeric; a markers= name isn't recognised; or (when include_aggregates=False) df has no is_aggregate column.

Example

import pandas as pd from tossd_reader import keyword_totals df = pd.DataFrame({ ... "keywords_raw": ["#GENDER", "#GENDER|#MITIGATION", ""], ... "usd_disbursement": [10.0, 20.0, 5.0], ... "is_aggregate": [False, False, False], ... }) result = keyword_totals(df, markers="gender") result["marker"].tolist() ['gender', 'Combined'] result["usd_disbursement"].tolist() [30.0, 30.0]

Split Pillar II's value by sub-pillar, year over year.

Takes the tossd_pillar == 2 subset of df internally -- Pillar I rows never appear in the result. Every row's tossd_subpillar value buckets to "II.A" (a real "21" tag), "II.B" ("22"), or "Untagged" (NA, or anything else) -- an unordered category.

Parameters:

Name Type Description Default
df DataFrame

A get_tossd()-shaped frame carrying year, tossd_pillar, tossd_subpillar, and value.

required
value str

The amount column to sum.

'usd_disbursement'
include_aggregates bool

False (the default) drops is_aggregate rows before splitting.

False

Returns:

Type Description
DataFrame

One row per (year, subpillar) pair -- every year present gets

DataFrame

all three bucket rows, even a bucket with nothing in it that year

`value` reads `0` then

year, subpillar, value,

DataFrame

share_pct (that row's share of its year's Pillar II total,

DataFrame

0-100 scale, unrounded), and coverage_pct (that year's combined

DataFrame

"II.A" + "II.B" share of the same total -- the same number

DataFrame

repeated on all three of that year's rows, since it describes the

DataFrame

year, not the bucket).

DataFrame

Sub-pillar tags only exist from 2023 (a 2022 trace appearance

DataFrame

aside, per get_structural_breaks); a year with "II.A"/"II.B"

DataFrame

both reading 0 and a near-zero coverage_pct means the year

DataFrame

predates tagging, not that Pillar II went unfunded. Read "II.A"/

DataFrame

"II.B" as real amounts only where coverage_pct is non-trivial.

Raises:

Type Description
ValueError

df is missing year, tossd_pillar, tossd_subpillar, or value; value is not numeric; or (when include_aggregates=False) df has no is_aggregate column.

Example

import pandas as pd from tossd_reader import subpillar_breakdown df = pd.DataFrame({ ... "year": [2024, 2024, 2024], ... "tossd_pillar": [2, 2, 2], ... "tossd_subpillar": ["21", "22", pd.NA], ... "usd_disbursement": [100.0, 50.0, 25.0], ... "is_aggregate": [False, False, False], ... }) result = subpillar_breakdown(df) result["subpillar"].tolist() ['II.A', 'II.B', 'Untagged'] result["usd_disbursement"].tolist() [100.0, 50.0, 25.0]

Return a deep copy of df.attrs["tossd_reader"], the query's own provenance record.

get_tossd(), get_tossd_raw(), and load_export() each set this key; every verb and accessor method above copies df.attrs onto its own result (so it survives rank_entities, explode_sdg, and so on), which is what makes reading it back through this function -- a deep copy, never the live dict -- safer than reaching into df.attrs["tossd_reader"] directly, where a caller mutating the returned dict would otherwise mutate df's own attrs.

Parameters:

Name Type Description Default
df DataFrame

A frame carrying df.attrs["tossd_reader"].

required

Returns:

Type Description
dict[str, object]

A deep copy of the payload: {"package_version", "created_at", "query", "years"} for a

dict[str, object]

get_tossd()/get_tossd_raw() result ("query" holds the normalised call; "years"

dict[str, object]

maps each fetched year to {"etag", "retrieved_at", "url"}), or

dict[str, object]

{"package_version", "created_at", "years"} for a load_export() frame (no "query"

dict[str, object]

key -- an export is an unfiltered snapshot, not a query result; see that function's own

dict[str, object]

docstring for its "years" shape).

Raises:

Type Description
ValueError

df.attrs carries no "tossd_reader" key -- naming the three functions that set it.

Example

import pandas as pd from tossd_reader import get_provenance df = pd.DataFrame({"usd_disbursement": [1.0]}) df.attrs["tossd_reader"] = {"package_version": "0.1.0"} get_provenance(df) {'package_version': '0.1.0'}

Describe df against the six manual checks in "How to check a figure against the published total", plus a few figures that guide doesn't cover.

A read-out, not a validator: every entry is df's own data, described -- never a warning, never a raise triggered by what the data happens to say (only by df not being get_tossd()-shaped at all). Every share is of df's own usd_disbursement total, aggregate rows included -- reconcile takes no include_aggregates= (unlike every other verb here): dropping them first would make aggregate_share_pct describe nothing.

Parameters:

Name Type Description Default
df DataFrame

A get_tossd()-shaped frame carrying unit, is_aggregate, year, tossd_pillar, and usd_disbursement -- the one required column set, kept minimal. usd_disbursement_deflated, modality_code, source_name, and recipient_code are each optional: when one is missing, the entries that need it read pd.NA instead of raising.

required

Returns:

Type Description
Series

A pandas.Series (object-dtype, one entry per check):

Series
  • unit: df's own unit value (a tuple of every distinct value present, when there's more than one).
Series
  • n_aggregate_rows, aggregate_value, aggregate_share_pct: the is_aggregate rows' own count and usd_disbursement share of df's total.
Series
  • usd_disbursement_total, usd_disbursement_deflated_total (pd.NA if usd_disbursement_deflated isn't in df): the two price bases' own totals, to match against whichever basis an external figure used.
Series
  • pillars_present: the sorted, distinct tossd_pillar values in df.
Series
  • year_min, year_max, n_years: df's own year coverage.
Series
  • has_provenance: whether df.attrs["tossd_reader"] is set (get_provenance(df) reads it).
Series
  • b02_core_contribution_value, b02_core_contribution_share_pct (pd.NA if modality_code isn't in df): usd_disbursement on modality_code == "B02" rows (core contributions to multilateral institutions).
Series
  • estimate_derived_value, estimate_derived_share_pct (pd.NA if source_name isn't in df): usd_disbursement on rows whose source_name contains "estimate" (case-insensitive) -- a heuristic naming convention, not a packaged flag.
Series
  • iso3_unmatched_value, iso3_unmatched_share_pct (pd.NA if recipient_code isn't in df): usd_disbursement on rows whose recipient_code has no add_iso3 match (regional/multi-country codes, TOSSD-only entities).

Raises:

Type Description
ValueError

df is missing unit, is_aggregate, year, tossd_pillar, or usd_disbursement; or usd_disbursement isn't numeric.

Example

import pandas as pd from tossd_reader import reconcile df = pd.DataFrame({ ... "unit": ["usd_thousand", "usd_thousand"], ... "is_aggregate": [True, False], ... "year": [2024, 2024], ... "tossd_pillar": [1, 1], ... "usd_disbursement": [40.0, 60.0], ... }) result = reconcile(df) float(result["usd_disbursement_total"]) 100.0 float(result["aggregate_share_pct"]) 40.0

The df.tossd accessor

Every tossd_reader query or helper import registers df.tossd as a pandas.DataFrame accessor, a pd.api.extensions.register_dataframe_accessor call that runs once as a side effect of importing tossd_reader.query or tossd_reader.analysis. A frame built without importing either module in a fresh session (such as an unpickled frame) requires import tossd_reader.analysis to register the accessor.

df.tossd carries the five verbs above, reconcile(), plus add_iso3, add_recipient_group, add_instrument_group, extract_keywords, explode_sdg, and filter_provider_costs from Helpers, each taking the same keyword arguments as its module function. The corresponding accessor method for get_provenance is provenance(). Three methods exist only on the accessor: summary() (a one-row printable summary of the frame), exclude_aggregates() (drops is_aggregate rows outside any verb), and groupby_entity(dimension=) (a raw GroupBy over {dimension}_code/{dimension}_name for custom aggregations).

Every method above returns a DataFrame carrying its own .tossd, so calls chain, with four exceptions. provenance() returns a dict. reconcile() and summary() each return a pandas.Series. groupby_entity() returns a pandas.api.typing.DataFrameGroupBy. A chain stops at any of the four.

df.tossd.exclude_aggregates().tossd.rank_entities(top=5)
 provider_code                provider_name  usd_disbursement  n_activities  share_pct  rank
           302                United States      67695.935324         61832  16.996373     1
           918              EU Institutions      58667.476757         85406  14.729604     2
             4                       France      25444.627005         14066   6.388365     3
           915 Asian Development Bank Group      18558.332668          4671   4.659428     4
           701                        Japan      17339.414452         17981   4.353395     5

How-to pages and the tutorial use this accessor spelling throughout. These module functions are the canonical documentation. The accessor methods delegate to them and raise the same errors on the same input.

Next