Verbs
tossd_reader.verbs aggregates a get_tossd()-shaped frame along one dimension (rank_entities, compare_years, sdg_totals, keyword_totals, and subpillar_breakdown). Each takes a DataFrame as its first argument, keyword-only arguments after that, and returns a new DataFrame, leaving the input frame untouched. Every one of the five is also a df.tossd accessor method. See The df.tossd accessor below.
The same module carries two functions that read a frame's own state. get_provenance returns its query record, and reconcile returns a one-row read-out of the checks in How to check a figure against the published total. Both take only df, no keyword arguments, and both are df.tossd methods too, spelled provenance() and reconcile().
The five aggregation verbs share three conventions.
Each drops the provider_code == 0 pseudo-aggregate rows before aggregating, with include_aggregates=False as the default. get_tossd() keeps them by default (see Excluding aggregate rows) to reproduce the published records in full, while the verb layer sums individual providers' activity.
value= names the amount column each verb sums, "usd_disbursement" on most verbs and "usd_disbursement_deflated" on compare_years.
df.attrs propagates. Every verb copies the input frame's attrs onto its result.
top= and share_pct vary by verb:
| verb | top= |
share_pct |
|---|---|---|
rank_entities |
yes | yes |
sdg_totals |
yes | yes |
subpillar_breakdown |
no | yes |
compare_years |
no | no |
keyword_totals |
no | no |
Where share_pct appears, it represents the share of the included total, from 0 to 100, unrounded, computed over the full grouped result before any top= argument truncates the rows. The total reflects the active include_aggregates setting and any filters present on the frame. A call without top= returns share_pct values that sum to 100. A call with top= returns a truncated slice of that ranking, whose values sum to less than 100.
include_aggregates in practice
import tossd_reader as tossd
df = tossd.get_tossd(years=2024, columns="analysis", units="usd_million")
tossd.rank_entities(df, top=3, include_aggregates=True)[
["provider_code", "provider_name", "usd_disbursement", "share_pct", "rank"]
]
provider_code provider_name usd_disbursement share_pct rank
0 Aggregate 99379.609718 19.968737 1
302 United States 67695.935324 13.602412 2
918 EU Institutions 58667.476757 11.788288 3
The "Aggregate" pseudo-row outranks every real provider. The default excludes it.
tossd.rank_entities(df, top=3)[
["provider_code", "provider_name", "usd_disbursement", "share_pct", "rank"]
]
provider_code provider_name usd_disbursement share_pct rank
302 United States 67695.935324 16.996373 1
918 EU Institutions 58667.476757 14.729604 2
4 France 25444.627005 6.388365 3
df.tossd.rank_entities(top=3) returns the same frame. See The df.tossd accessor below.
Activity counts vs row counts
keyword_totals's n_rows counts rows whose keyword mask matches. The publisher's pre-split rows can carry different keyword tags from each other, so one activity spread across several rows counts once per matching row. rank_entities's n_activities counts distinct tossd_id values instead. The two measure different dimensions. Check the column name before comparing one verb's count against another.
get_provenance in practice
import tossd_reader as tossd
from pprint import pprint
df = tossd.get_tossd(years=2024, columns="analysis", units="usd_million")
pprint(tossd.get_provenance(df))
{'created_at': '2026-09-02T12:19:04.952853+00:00',
'package_version': '0.1.0',
'query': {'columns': 'analysis',
'filters': {},
'include_aggregates': True,
'pillars': None,
'providers': None,
'recipients': None,
'refresh': False,
'units': 'usd_million',
'years': (2024,)},
'years': {'2024': {'etag': '"69e6ac8d-5728379"',
'retrieved_at': '2026-09-02T12:10:44.031565+00:00',
'url': 'https://tossd.online/tossddata_2024.parquet'}}}
get_provenance returns a deep copy of df.attrs["tossd_reader"], so mutating the returned dictionary leaves df untouched. get_tossd_raw() frames carry a minimal query dictionary (refresh and years only). A load_export() frame carries the three-key shape without a query key, reflecting an unfiltered snapshot.
{'created_at': '2026-09-02T12:10:39.819913+00:00',
'package_version': '0.1.0',
'years': {'2019': {'etag': '"69e6ac86-347a653"',
'retrieved_at': '2026-09-02T12:10:39.379359+00:00'}}}
The key survives every verb and accessor call above. Each one copies df.attrs onto its own result, so provenance survives rank_entities, explode_sdg, and the rest. Plain pandas operations like merge, concat, or some groupby calls can drop attrs. Calling get_provenance on a frame that does not carry the key raises ValueError.
ValueError: get_provenance() found no df.attrs['tossd_reader'] -- that
key is set by get_tossd(), get_tossd_raw(), and load_export(); a frame
built some other way (or a plain pandas operation that dropped attrs
along the way) carries none.
df.tossd.provenance() returns the identical dict and raises the same error.
reconcile in practice
import tossd_reader as tossd
df = tossd.get_tossd(years=2024, columns="analysis", units="usd_million")
df.tossd.reconcile()
unit usd_million
n_aggregate_rows 5626
aggregate_value 99379.609718
aggregate_share_pct 19.968737
usd_disbursement_total 497675.981441
usd_disbursement_deflated_total 497675.981441
pillars_present (1, 2)
year_min 2024
year_max 2024
n_years 1
has_provenance True
b02_core_contribution_value 6678.973842
b02_core_contribution_share_pct 1.342033
estimate_derived_value 2913.935663
estimate_derived_share_pct 0.585509
iso3_unmatched_value 147394.103994
iso3_unmatched_share_pct 29.616479
dtype: object
reconcile is a read-out that summarises dataset characteristics. Warnings and errors arise only when passed a frame that is not shaped by get_tossd(). Shares are calculated against df's own usd_disbursement total with aggregate rows included, which is why reconcile accepts no include_aggregates= argument. Retaining aggregate rows allows aggregate_share_pct to measure their proportion directly.
usd_disbursement_total equals usd_disbursement_deflated_total here because 2024 is the deflator base year. For multi-year frames or non-base years, these values diverge.
unit usd_thousand
n_aggregate_rows 11451
aggregate_value 173447820.379903
aggregate_share_pct 17.876098
usd_disbursement_total 970277829.269724
usd_disbursement_deflated_total 982043011.985994
pillars_present (0, 1, 2)
year_min 2023
year_max 2024
n_years 2
has_provenance True
b02_core_contribution_value 12881067.080131
b02_core_contribution_share_pct 1.327565
estimate_derived_value 4259974.195728
estimate_derived_share_pct 0.439047
iso3_unmatched_value 299091812.522857
iso3_unmatched_share_pct 30.825378
dtype: object
usd_disbursement_deflated_total, b02_core_contribution_value, estimate_derived_value, and iso3_unmatched_value read pd.NA when their source column (usd_disbursement_deflated, modality_code, source_name, recipient_code respectively) is missing from df. estimate_derived_value sums usd_disbursement on rows whose source_name contains "estimate" (case-insensitive), providing a heuristic reading of the source name.
pillars_present reflects whatever pillar values are in df. The (0, 1, 2) tuple above comes from real 2023 rows. See Pillars and aggregates for pillar 0 definitions.
See How to check a figure against the published total for reading these entries against a specific published number.
Rank entities along one dimension by summed value.
Works for any dimension whose {dimension}_code/{dimension}_name
columns are present in df -- "provider" (the default),
"recipient", "sector", "purpose", "channel", or any other
packaged dimension pair. Ranking providers across both pillars at once
mixes Pillar I bilateral/multilateral outflows with Pillar II core
contributions to multilateral institutions -- a double count, if the
two are meant to be read as one figure.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
A |
required |
dimension
|
str
|
The dimension to group by -- any prefix with a matching
|
'provider'
|
value
|
str
|
The amount column to sum and rank by. |
'usd_disbursement'
|
top
|
int | None
|
Keep only the first |
None
|
include_aggregates
|
bool
|
|
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
One row per distinct |
DataFrame
|
(an |
DataFrame
|
sorted by summed |
DataFrame
|
|
DataFrame
|
|
DataFrame
|
placeholder used for bundled lines with no activity identifier, so |
DataFrame
|
those lines are never counted as activities; omitted entirely, |
DataFrame
|
rather than raising, when |
DataFrame
|
(of the summed total across every included row, 0-100 scale, |
DataFrame
|
unrounded), and |
DataFrame
|
the lower rank number -- 1-based, descending). Truncation to |
DataFrame
|
happens after ranking, so the numbering on the entities that made |
DataFrame
|
the cut always matches what a caller would see without |
DataFrame
|
The |
DataFrame
|
providers' own rows in 2023-24 (bundled lines belonging to a real |
DataFrame
|
provider, not just pseudo-aggregates), so |
DataFrame
|
slightly undercount those providers even after aggregate exclusion. |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Example
import pandas as pd from tossd_reader import rank_entities df = pd.DataFrame({ ... "provider_code": [1, 1, 2], ... "provider_name": ["Austria", "Austria", "France"], ... "usd_disbursement": [100.0, 50.0, 200.0], ... "is_aggregate": [False, False, False], ... }) result = rank_entities(df) result["provider_name"].tolist() ['France', 'Austria'] result["rank"].tolist() [1, 2]
Compare summed value year over year.
Recipients are not held constant across years -- only the provider
cohort is, and only when cohort="consistent". A year-over-year rise
or fall can still be a shift in which recipients each provider funded,
not a change in how much any single provider gave.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
A |
required |
value
|
str
|
The amount column to sum per year. |
'usd_disbursement_deflated'
|
cohort
|
Literal['consistent', 'all']
|
|
'consistent'
|
include_aggregates
|
bool
|
|
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
One row per year present in |
DataFrame
|
(the cohort's size under |
DataFrame
|
provider-pair count under |
DataFrame
|
change from the previous row's |
DataFrame
|
|
DataFrame
|
["structural_breaks"] |
DataFrame
|
rows intersecting the years covered, so a caller can flag a jump |
DataFrame
|
against a known discontinuity instead of a real trend. |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Example
import pandas as pd from tossd_reader import compare_years df = pd.DataFrame({ ... "year": [2019, 2020], ... "provider_code": [1, 1], ... "provider_name": ["Austria", "Austria"], ... "usd_disbursement_deflated": [100.0, 150.0], ... "is_aggregate": [False, False], ... }) result = compare_years(df) result["usd_disbursement_deflated"].tolist() [100.0, 150.0] result["pct_change"].tolist() [nan, 50.0]
Sum value by SDG tag, weighted for rows that carry more than one tag.
Internally explodes df with analysis.explode_sdg first, then sums
value * sdg_weight per tag -- the weighted total this returns equals
the SDG-tagged subset of df's value total, never df's grand
total, since a row with no SDG tag at all contributes nothing to any
group.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
A |
required |
value
|
str
|
The amount column to sum. |
'usd_disbursement'
|
level
|
Literal['goal', 'code']
|
|
'goal'
|
top
|
int | None
|
Keep only the first |
None
|
include_aggregates
|
bool
|
|
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
One row per distinct goal or code (matching |
DataFrame
|
or |
DataFrame
|
SDG-tagged total across every included row, 0-100 scale, |
DataFrame
|
unrounded), and |
DataFrame
|
1-based). Truncation to |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Example
import pandas as pd from tossd_reader import sdg_totals df = pd.DataFrame({ ... "sdg_codes_raw": ["1;4", "4"], ... "usd_disbursement": [100.0, 100.0], ... "is_aggregate": [False, False], ... }) result = sdg_totals(df, level="goal") result["sdg_goal"].tolist() [4, 1] result["usd_disbursement"].tolist() [150.0, 50.0]
Sum value per keyword marker, plus a union row combining them.
Marker masks are recomputed internally from keywords_raw via
analysis.extract_keywords -- any kw_* columns already in df are
ignored, so a caller who filtered or otherwise altered df since
running extract_keywords never gets stale marker flags here.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
A |
required |
markers
|
str | Iterable[str] | None
|
One marker name, an iterable of them, or |
None
|
value
|
str
|
The amount column to sum. |
'usd_disbursement'
|
include_aggregates
|
bool
|
|
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
One row per requested marker, in the order given (packaged order |
DataFrame
|
when |
DataFrame
|
|
DataFrame
|
distinct- |
DataFrame
|
spread one activity across several rows, and those rows can |
DataFrame
|
genuinely carry different keyword tags from each other, so a |
DataFrame
|
multi-row activity counts once per matching row rather than once |
DataFrame
|
overall, and no distinct-ID count is offered here). |
DataFrame
|
mask is the union of every requested marker's mask, so the |
DataFrame
|
individual marker rows sum to at least |
DataFrame
|
less -- a row tagged with more than one requested marker is |
DataFrame
|
counted once per marker row but only once in |
DataFrame
|
is the double count the union row exists to avoid. |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Example
import pandas as pd from tossd_reader import keyword_totals df = pd.DataFrame({ ... "keywords_raw": ["#GENDER", "#GENDER|#MITIGATION", ""], ... "usd_disbursement": [10.0, 20.0, 5.0], ... "is_aggregate": [False, False, False], ... }) result = keyword_totals(df, markers="gender") result["marker"].tolist() ['gender', 'Combined'] result["usd_disbursement"].tolist() [30.0, 30.0]
Split Pillar II's value by sub-pillar, year over year.
Takes the tossd_pillar == 2 subset of df internally -- Pillar I
rows never appear in the result. Every row's tossd_subpillar value
buckets to "II.A" (a real "21" tag), "II.B" ("22"), or
"Untagged" (NA, or anything else) -- an unordered category.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
A |
required |
value
|
str
|
The amount column to sum. |
'usd_disbursement'
|
include_aggregates
|
bool
|
|
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
One row per |
DataFrame
|
all three bucket rows, even a bucket with nothing in it that year |
`value` reads `0` then
|
|
DataFrame
|
|
DataFrame
|
0-100 scale, unrounded), and |
DataFrame
|
|
DataFrame
|
repeated on all three of that year's rows, since it describes the |
DataFrame
|
year, not the bucket). |
DataFrame
|
Sub-pillar tags only exist from 2023 (a 2022 trace appearance |
DataFrame
|
aside, per |
DataFrame
|
both reading |
DataFrame
|
predates tagging, not that Pillar II went unfunded. Read |
DataFrame
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Example
import pandas as pd from tossd_reader import subpillar_breakdown df = pd.DataFrame({ ... "year": [2024, 2024, 2024], ... "tossd_pillar": [2, 2, 2], ... "tossd_subpillar": ["21", "22", pd.NA], ... "usd_disbursement": [100.0, 50.0, 25.0], ... "is_aggregate": [False, False, False], ... }) result = subpillar_breakdown(df) result["subpillar"].tolist() ['II.A', 'II.B', 'Untagged'] result["usd_disbursement"].tolist() [100.0, 50.0, 25.0]
Return a deep copy of df.attrs["tossd_reader"], the query's own provenance record.
get_tossd(), get_tossd_raw(), and load_export() each set this key; every verb and
accessor method above copies df.attrs onto its own result (so it survives rank_entities,
explode_sdg, and so on), which is what makes reading it back through this function -- a
deep copy, never the live dict -- safer than reaching into df.attrs["tossd_reader"]
directly, where a caller mutating the returned dict would otherwise mutate df's own attrs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
A frame carrying |
required |
Returns:
| Type | Description |
|---|---|
dict[str, object]
|
A deep copy of the payload: |
dict[str, object]
|
|
dict[str, object]
|
maps each fetched year to |
dict[str, object]
|
|
dict[str, object]
|
key -- an export is an unfiltered snapshot, not a query result; see that function's own |
dict[str, object]
|
docstring for its |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Example
import pandas as pd from tossd_reader import get_provenance df = pd.DataFrame({"usd_disbursement": [1.0]}) df.attrs["tossd_reader"] = {"package_version": "0.1.0"} get_provenance(df) {'package_version': '0.1.0'}
Describe df against the six manual checks in "How to check a figure against the
published total", plus a few figures that guide doesn't cover.
A read-out, not a validator: every entry is df's own data, described -- never a warning,
never a raise triggered by what the data happens to say (only by df not being
get_tossd()-shaped at all). Every share is of df's own usd_disbursement total,
aggregate rows included -- reconcile takes no include_aggregates= (unlike every other
verb here): dropping them first would make aggregate_share_pct describe nothing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
A |
required |
Returns:
| Type | Description |
|---|---|
Series
|
A |
Series
|
|
Series
|
|
Series
|
|
Series
|
|
Series
|
|
Series
|
|
Series
|
|
Series
|
|
Series
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Example
import pandas as pd from tossd_reader import reconcile df = pd.DataFrame({ ... "unit": ["usd_thousand", "usd_thousand"], ... "is_aggregate": [True, False], ... "year": [2024, 2024], ... "tossd_pillar": [1, 1], ... "usd_disbursement": [40.0, 60.0], ... }) result = reconcile(df) float(result["usd_disbursement_total"]) 100.0 float(result["aggregate_share_pct"]) 40.0
The df.tossd accessor
Every tossd_reader query or helper import registers df.tossd as a pandas.DataFrame accessor, a pd.api.extensions.register_dataframe_accessor call that runs once as a side effect of importing tossd_reader.query or tossd_reader.analysis. A frame built without importing either module in a fresh session (such as an unpickled frame) requires import tossd_reader.analysis to register the accessor.
df.tossd carries the five verbs above, reconcile(), plus add_iso3, add_recipient_group, add_instrument_group, extract_keywords, explode_sdg, and filter_provider_costs from Helpers, each taking the same keyword arguments as its module function. The corresponding accessor method for get_provenance is provenance(). Three methods exist only on the accessor: summary() (a one-row printable summary of the frame), exclude_aggregates() (drops is_aggregate rows outside any verb), and groupby_entity(dimension=) (a raw GroupBy over {dimension}_code/{dimension}_name for custom aggregations).
Every method above returns a DataFrame carrying its own .tossd, so calls chain, with four exceptions. provenance() returns a dict. reconcile() and summary() each return a pandas.Series. groupby_entity() returns a pandas.api.typing.DataFrameGroupBy. A chain stops at any of the four.
provider_code provider_name usd_disbursement n_activities share_pct rank
302 United States 67695.935324 61832 16.996373 1
918 EU Institutions 58667.476757 85406 14.729604 2
4 France 25444.627005 14066 6.388365 3
915 Asian Development Bank Group 18558.332668 4671 4.659428 4
701 Japan 17339.414452 17981 4.353395 5
How-to pages and the tutorial use this accessor spelling throughout. These module functions are the canonical documentation. The accessor methods delegate to them and raise the same errors on the same input.
Next
- Rank providers by disbursement. Applies
rank_entitiesend to end, including thetop=/share_pctinterplay. - Split disbursements across SDG goals. Works through
sdg_totals's weighting against a real SDG-tagged query. - Check a figure against the published total. Runs
reconcile()against a real published number, then digs into individual checks.