Helpers

The tossd_reader.analysis module provides analytical helper functions for working with get_tossd() outputs. Six functions accept a pandas.DataFrame and return a new DataFrame, leaving the input data unchanged. Three take no DataFrame: get_structural_breaks (call it with an optional years= to narrow the packaged structural-break reference table to the years a query touches, or with no arguments for all five rows), get_recipient_groups_version, and get_instrument_groups_version (each returns the version stamp of a packaged classification table). All nine functions operate entirely offline without network access.

Function names use one verb per operation consistently across the package (get_ fetches, add_ annotates rows, explode_ expands rows, extract_ derives flag columns, and filter_ subsets rows).

Each DataFrame helper validates required columns before processing and raises a ValueError naming any missing column. When a missing column ships in the "analysis" preset, the message also names the fix.

tossd.explode_sdg(tossd.get_tossd(years=2024, columns="minimal"))

ValueError: explode_sdg() needs column(s) sdg_codes_raw, not present in df. Re-query with columns='analysis', or add sdg_codes_raw to your columns= list.

Helper	Input Requirements	Minimum Preset	Output
`explode_sdg`	`sdg_codes_raw` (plus `value` when `value=` is given)	`"analysis"`	Expanded DataFrame with `sdg_code`, `sdg_goal`, `sdg_is_target`, `sdg_weight`, and (with `value=`) `{value}_weighted`
`extract_keywords`	`keywords_raw`	`"analysis"`	DataFrame with 12 `kw_<marker>` boolean columns
`add_iso3`	`provider_code` or `recipient_code`	`"minimal"`	DataFrame with `provider_iso3` or `recipient_iso3` categoricals
`add_recipient_group`	`recipient_code`	`"minimal"`	DataFrame with a `recipient_group` categorical (`scheme="ldc"`/`"income"`/`"region"`)
`get_recipient_groups_version`	None	n/a	Version stamp (`str`) of the packaged recipient-groups table
`add_instrument_group`	`finance_instrument_code`, `concessionality_flag`	`"analysis"`	DataFrame with an `instrument_group` categorical
`get_instrument_groups_version`	None	n/a	Version stamp (`str`) of the packaged instrument-groups table
`filter_provider_costs`	`tossd_pillar`, `sector_code`	`"analysis"`	DataFrame filtered to Pillar II domestic expenditures (sectors 910 and 930)
`get_structural_breaks`	None; optional `years=` narrows the rows	n/a	Reference DataFrame of dataset discontinuities (5 rows, fewer with `years=`)

Explode sdg_codes_raw into one row per SDG code token.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	A `get_tossd()`-shaped frame carrying `sdg_codes_raw`.	required
`value`	`str \| None`	An amount column name. When given, the result gains a sibling `{value}_weighted` column (`df[value] * sdg_weight`) alongside the untouched original `value` column -- never overwriting it, since the same column name would mean a different thing depending on what last touched the frame. `None` (the default): today's behaviour, byte-identical output.	`None`

Returns:

Type	Description
`DataFrame`	A new frame: every input column, plus one row per `;`-delimited SDG
`DataFrame`	token found in `sdg_codes_raw` -- `sdg_code` (the token exactly as
`DataFrame`	published), `sdg_goal` (`Int8`, the token's integer goal part),
`DataFrame`	`sdg_is_target` (`bool`, whether the token names a specific target
`DataFrame`	rather than the goal as a whole), and `sdg_weight` (`float64`,
`DataFrame`	`1 / n` for the `n` tokens that source row carried, so a grouped sum
`DataFrame`	of `amount * sdg_weight` renormalises to that row's original
`DataFrame`	amount). When `value` is given, also `{value}_weighted`
`DataFrame`	(`df[value] * sdg_weight`, `float64`) -- a grouped sum of it equals
`DataFrame`	the SDG-tagged subset's original `value` total, the same
`DataFrame`	renormalisation identity `sdg_weight` already gives any ad hoc
`DataFrame`	`amount * sdg_weight` multiplication, just precomputed and named.
`DataFrame`	Rows with no SDG tag at all (an empty or null `sdg_codes_raw`)
`DataFrame`	contribute nothing to the result: the exploded frame's weighted
`DataFrame`	total equals only the SDG-tagged subset of `df`'s totals, never
`DataFrame`	`df`'s grand total. Row order is otherwise preserved.

Raises:

Type	Description
`ValueError`	`df` has no `sdg_codes_raw` column (or, when `value` is given, no `value` column); `value` is given but `df[value]` isn't numeric-dtyped; or `df` already carries one of this function's own output columns (`sdg_code`, `sdg_goal`, `sdg_is_target`, `sdg_weight`, and `{value}_weighted` when `value` is given) -- re-running `explode_sdg` on its own output would otherwise silently duplicate them.

Example

import pandas as pd from tossd_reader import explode_sdg df = pd.DataFrame({"sdg_codes_raw": ["1;4", ""]}) result = explode_sdg(df) result["sdg_code"].tolist() ['1', '4'] result["sdg_weight"].tolist() [0.5, 0.5]

Add one boolean kw_<marker> column per packaged keyword marker.

Matching casefolds each keywords_raw token and strips a leading #, so "COVID-19" and "#COVID-19" both count as the covid_19 marker. Only the fixed 12-marker vocabulary shipped as _data/keyword_markers.csv is recognised; every other token in keywords_raw (315 distinct tokens across the full vocabulary) is ignored.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	A `get_tossd()`-shaped frame carrying `keywords_raw`.	required

Returns:

Type	Description
`DataFrame`	`df` plus one `kw_<marker>` boolean column per packaged marker
`DataFrame`	(`kw_gender`, `kw_adaptation`, `kw_mitigation`, `kw_biodiversity`,
`DataFrame`	`kw_ppr_preparedness`, `kw_ppr_response`, `kw_covid_19`,
`DataFrame`	`kw_refugees_hostcommunities`, `kw_idps_hostcommunities`,
`DataFrame`	`kw_voluntaryrefugeereturn_reintegration`,
`DataFrame`	`kw_transnational_benefits_global`, `kw_non_17_3_1`).
`DataFrame`	`keywords_raw` itself is left untouched.

Raises:

Type	Description
`ValueError`	`df` has no `keywords_raw` column.

Example

import pandas as pd from tossd_reader import extract_keywords df = pd.DataFrame({"keywords_raw": ["#GENDER", "#MITIGATION"]}) extract_keywords(df)["kw_gender"].tolist() [True, False]

import tossd_reader as tossd

sen_a = tossd.get_tossd(years=2024, recipients="Senegal", columns="analysis")
kw = tossd.extract_keywords(sen_a)
kw_cols = [c for c in kw.columns if c.startswith("kw_")]
print(kw[kw_cols].sum().sort_values(ascending=False).head(6))

kw_gender              1308
kw_adaptation           599
kw_biodiversity         398
kw_mitigation           378
kw_ppr_preparedness     119
kw_covid_19              83
dtype: int64

Add provider_iso3/recipient_iso3 via resolvekit's OECD numeric-code link.

ISO3 is looked up by code. provider_code 913 and 914 both carry provider_name "African Development Bank Group" in the published files, as do 909 and 1019 for "Inter-American Development Bank Group", so a name-keyed join would collapse distinct providers into one row. The packaged codelist names them apart, so codes carry no such collision.

Uses resolvekit's bundled geo.countries module, which runs entirely offline, with no download step. That module carries the OECD DAC provider/recipient numeric codelists as the oecd:provider and oecd:recipient code systems, linked to iso3, and its link was checked against all 159 packaged provider codes and all 177 recipient codes with no mismatch against the packaged codelist's own iso3 column.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	A `get_tossd()`-shaped frame carrying `provider_code` and/or `recipient_code`.	required

Returns:

Type	Description
`DataFrame`	`df` plus `provider_iso3` and/or `recipient_iso3` (`category`
`DataFrame`	dtype), for whichever of `provider_code`/`recipient_code` is
`DataFrame`	present. Aggregates (code `0`), multilaterals, and TOSSD-only
`DataFrame`	entities all map to `NA`.

Raises:

Type	Description
`ValueError`	Neither `provider_code` nor `recipient_code` is present in `df`.

Example

import pandas as pd from tossd_reader import add_iso3 df = pd.DataFrame({"recipient_code": [55]}) add_iso3(df)["recipient_iso3"].tolist() ['TUR']

Add recipient_group, joined from the packaged recipient-groups table.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	A `get_tossd()`-shaped frame carrying `recipient_code`.	required
`scheme`	`Literal['ldc', 'income', 'region']`	Which grouping to apply -- `"ldc"` (Least Developed Countries / Other Developing Countries), `"income"` (World Bank income group), or `"region"` (the UN region TOSSD itself publishes per recipient, the same value `get_tossd()`'s own `region_name` column carries).	`'ldc'`

Returns:

Type	Description
`DataFrame`	`df` plus `recipient_group` (`category`), one value per
`DataFrame`	`recipient_code` read from the table `get_recipient_groups_version()`
`DataFrame`	names.
`DataFrame`	Regional/multi-country recipient codes (e.g. "Europe, regional",
`DataFrame`	"Global" -- codes with no `iso3` in the packaged recipient codelist)
`DataFrame`	carry an explicit `"Regional / Multi-country Unallocated"` value under
`DataFrame`	`"ldc"`/`"income"`; `"region"` never produces it, since TOSSD
`DataFrame`	publishes a real region for these codes too. Six non-self-governing
`DataFrame`	territories the World Bank does not publish independent GNI data for
`DataFrame`	(Saint Helena, Montserrat, Cook Islands, Niue, Tokelau, Wallis and
`DataFrame`	Futuna) carry `"Unclassified"` under `"income"` -- distinct from
`DataFrame`	`"Regional / Multi-country Unallocated"`, since these are real,
`DataFrame`	single-territory codes, not aggregates.
`DataFrame`	A `recipient_code` absent from the packaged table (a TOSSD vintage
`DataFrame`	newer than the snapshot) resolves to `NA` and triggers a
`DataFrame`	once-per-code warning rather than an error, so a full-frame call
`DataFrame`	stays usable.
`DataFrame`	São Tomé and Príncipe (`recipient_code` 268) graduated from LDC
`DataFrame`	status on 2024-12-06. The packaged table reflects the current LDC
`DataFrame`	list, so it classifies STP as `"Other Developing Countries"` even
`DataFrame`	for 2024 rows reported before that date -- an analyst reconciling
`DataFrame`	against a 2024-vintage LDC classification should treat STP as LDC
`DataFrame`	for that year instead of trusting this column.

Raises:

Type	Description
`ValueError`	`df` has no `recipient_code` column, `scheme` is not one of `"ldc"`/`"income"`/`"region"`, or `df` already carries a `recipient_group` column.

Example

import pandas as pd from tossd_reader import add_recipient_group df = pd.DataFrame({"recipient_code": [269]}) # Senegal add_recipient_group(df)["recipient_group"].tolist() ['Least Developed Countries']

Return the packaged recipient-groups table's version stamp.

Returns:

Type	Description
`str`	The composite stamp recorded in `_data/recipient_groups_version.json`
`str`	(its `"version"` field), e.g. `"ldc-2024review/wb-fy27"` --
`str`	independently naming the UN LDC-list vintage (`ldc`/`region` schemes
`str`	don't move on their own cadence) and the World Bank income
`str`	classification's fiscal year (`income` scheme) the packaged table
`str`	was built from.

import tossd_reader as tossd

df = tossd.get_tossd(years=2024, columns="analysis", units="usd_million")
rg = tossd.add_recipient_group(df, scheme="ldc")
rg.groupby("recipient_group", observed=True)["usd_disbursement"].sum().round(1)

Least Developed Countries                88056.9
Other Developing Countries              262225.0
Regional / Multi-country Unallocated    147394.1

tossd.get_recipient_groups_version()

'ldc-2024review/wb-fy27'

Add instrument_group, joined from the packaged instrument-groups table.

finance_instrument_code alone decides every group except the debt-instrument family (420 "DEBT INSTRUMENTS" and its children 421 "Standard loan" through 425 "Other debt securities", plus 4221 "Loan-type reimbursable grant" and 4222 "Reflow-based reimbursable grant" -- both 422 "Reimbursable grant" sub-variants by name and real-data concessionality profile), which additionally needs concessionality_flag: flag 1 -> "Concessional Loans", flag 0 -> "Non-concessional Loans", a blank flag -> NA (46 rows in the 2024 vintage, spread across the debt codes, 17 of them on code 421, the highest-volume debt instrument -- there is no separate concessional/non-concessional code, only the one flag column).

Groups: "Grants" (100/110), "Non-concessional Loans" / "Concessional Loans" (the debt family, see above), "Hybrid/Mezzanine" (430-434, the mezzanine-finance family), "Equity" (500/510/520), "Guarantees" (1000/1100, plus 1101 "Individual loan guarantee" and 1102 "Loan portfolio guarantee" by name and real-data concessionality profile -- overwhelmingly blank, matching 1100, not the debt family's mixed split), "Direct Provider Spending" (2000/2100, its own group -- not a transfer to a recipient at all), "Subsidies" (3000/3100), and "Other Instruments" (310 "Capital subscription on deposit basis", orphaned in the codelist's own header hierarchy -- no 300 parent row exists unlike every other family; and 0 "NON FLOW ITEMS", 3 real rows across 2019-2024, one provider (Korea, provider_code 742), each carrying a real usd_disbursement -- OECD's own List 15 flags it CRS-only, and its name argues for excluding it from a financing-flow breakdown entirely, but it is a real, non-blank code on real rows with real dollar amounts, not a structurally-blank pseudo-aggregate, so it gets an honest bucket here rather than NA).

100, 110, 310, 420-425, 430-434, 500, 510, 520, 1000, 1100, 2000, 2100, 3000, 3100 come from the packaged finance_instrument codelist; 0, 1101, 1102, 4221, 4222 do not -- OECD's own live List 15 still flags all five tossd="0" (CRS-only) as of get_instrument_groups_version()'s fetch date, even though real 2023-2024 TOSSD submissions carry them, so a codelist refresh alone can never add them (see scripts/refresh_codelists.py's own --check, which reports zero drift against the live source). They are mapped directly here instead, from each code's official List-15 name and its real-data profile, distinguished from the codelist-sourced rows by the packaged table's own source column ("codelist"/"observed") for anyone auditing where a mapping came from.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	A `get_tossd()`-shaped frame carrying `finance_instrument_code` and `concessionality_flag`.	required

Returns:

Type	Description
`DataFrame`	`df` plus `instrument_group` (`category`). A blank/`NA`
`DataFrame`	`finance_instrument_code` (every pseudo-aggregate row --
`DataFrame`	`provider_code == 0` -- carries one) resolves to `NA`, not an error,
`DataFrame`	so a full-frame call including aggregates stays usable. `NA` rows
`DataFrame`	carry material dollar volume of their own (pseudo-aggregates plus
`DataFrame`	blank-concessionality debt rows, roughly 20% of 2024 disbursements
`DataFrame`	combined) and pandas' own `groupby` drops `NaN` keys by default, so
`DataFrame`	totalling by `instrument_group` needs `dropna=False` or excluding
`DataFrame`	aggregates first to avoid silently losing that share.

Raises:

Type	Description
`ValueError`	`df` is missing `finance_instrument_code` or `concessionality_flag`, or already carries an `instrument_group` column.
`UnknownCodeError`	A non-null `finance_instrument_code` value is absent from the packaged table -- never silently grouped into `"Other Instruments"`. Reserved for a genuinely new or malformed code (a typo, or a future vintage's drift): every code observed in the six cached 2019-2024 TOSSD vintages is mapped, whether or not OECD's own codelist covers it yet.

Example

import pandas as pd from tossd_reader import add_instrument_group df = pd.DataFrame({ ... "finance_instrument_code": [110], ... "concessionality_flag": [pd.NA], ... }) add_instrument_group(df)["instrument_group"].tolist() ['Grants']

Return the packaged instrument-groups table's version stamp.

Returns:

Type	Description
`str`	The composite stamp recorded in `_data/instrument_groups_version.json`
`str`	(its `"version"` field), e.g.
`str`	`"oecd-dac-cl15-2026-09-01/instrument-groups-methodology-v2"` --
`str`	independently naming the OECD DAC CRS++ "List 15: Type of finance"
`str`	fetch date the packaged `finance_instrument` codelist (and this
`str`	table) were checked against, and this project's own group-assignment
`str`	methodology revision, which moves independently of the OECD list
`str`	(v2 additionally maps the codes real submissions carry that List 15
`str`	itself doesn't yet flag `tossd`-applicable -- see
`str`	`add_instrument_group`).

ig = tossd.add_instrument_group(df)
ig.groupby("instrument_group", dropna=False, observed=True)[
    "usd_disbursement"
].sum().round(1)

Concessional Loans           47194.6
Direct Provider Spending     49378.8
Equity                        2796.1
Grants                      200931.0
Guarantees                     882.3
Hybrid/Mezzanine               467.2
Non-concessional Loans       95923.0
Other Instruments                8.7
NaN                         100094.2

tossd.get_instrument_groups_version()

'oecd-dac-cl15-2026-09-01/instrument-groups-methodology-v2'

The NaN row (pseudo-aggregates plus blank-concessionality loans) carries roughly 20% of 2024 dollar volume. dropna=False keeps it visible. pandas' default groupby drops NaN keys, which silently loses that share.

Filter pillar-2 rows to the provider-costs carve-out.

Sector family 930 ("Domestic expenditures for refugees/asylum seekers") records spending inside the provider's own territory by definition. Sector family 910 ("Administrative Costs of Donors") is a proxy for donor administrative overhead, which predominantly stays in the provider country, though some administrative costs are incurred at the recipient end. Together the two are the share that the AidWatch, Oxfam, and ActionAid critique of TOSSD Pillar II identifies as the domestic-spending share of Pillar II.

On the 2024 vintage the two families cover 27,275 of 155,908 pillar-2 rows (17.5%), 35.6% of pillar-2 gross disbursements, and 31.0% of pillar-2 commitments, consistent with the roughly 30% share that critique attributes to these costs. Sector family 720 ("Humanitarian Assistance") sits outside the carve-out. Those rows are in-country humanitarian aid delivered by agencies such as UNHCR and UNICEF.

TOSSD's Reporting Instructions describe this category as "expenditures in the provider country". Analysts commonly call this spending "in-donor" costs. TOSSD does not publish a ready-made carve-out matching that label, so this is a heuristic built from sector families 910 and 930 that approximate it.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	A `get_tossd()`-shaped frame carrying `tossd_pillar` and `sector_code` (`sector_code` is present under `columns="analysis"`/`"all"`, or any explicit `columns=` list naming it; `tossd_pillar` is always present in `get_tossd()` output).	required

Returns:

Type	Description
`DataFrame`	`df` filtered to `tossd_pillar == 2` rows whose `sector_code` is
`DataFrame`	`910` ("Administrative Costs of Donors") or `930` ("Domestic
`DataFrame`	expenditures for refugees/asylum seekers").

Raises:

Type	Description
`ValueError`	`df` is missing `tossd_pillar` or `sector_code`.

Example

import pandas as pd from tossd_reader import filter_provider_costs df = pd.DataFrame({ ... "tossd_pillar": [2, 2, 1], ... "sector_code": [910, 220, 910], ... }) filter_provider_costs(df)["sector_code"].tolist() [910]

Return the packaged structural-breaks reference table.

Documents five verified TOSSD-vintage discontinuities relevant to cross-year analysis: sub-pillar tagging's 2022 trace appearance and 2023 partial rollout, R&D modality (K02)'s 2021 introduction, reporter base growth spanning 2019-2024, and the 2026 RDRM methodology change. This is reference data for a caller to consult -- it does not validate or warn against any particular query.

Parameters:

Name	Type	Description	Default
`years`	`int \| Iterable[int] \| None`	A single year, an iterable of years, or `None` (the default) for every row. When given, keeps only the rows whose `[break_year, end_year]` interval intersects at least one requested year -- so `get_structural_breaks(years=query_years)` names only the breaks relevant to a `get_tossd(years=query_years)` call.	`None`

Returns:

Type	Description
`DataFrame`	A new `pandas.DataFrame` (a copy of the cached table, so editing it
`DataFrame`	leaves later calls unaffected) with columns `dimension`,
`DataFrame`	`break_year`, `end_year`, `description`, `source` -- all 5 rows with
`DataFrame`	`years=None`, fewer (or none) once `years=` narrows them. For the
`DataFrame`	four discrete breaks `end_year` equals `break_year`; the
`DataFrame`	`reporters` row's `end_year` (2024) marks the end of that row's
`DataFrame`	continuous 2019-2024 drift, so every year from `break_year` through
`DataFrame`	`end_year` counts as affected.

Example

from tossd_reader import get_structural_breaks breaks = get_structural_breaks(years=2026) breaks["dimension"].tolist() ['methodology']

import tossd_reader as tossd

breaks = tossd.get_structural_breaks()
print(
    breaks[["dimension", "break_year", "end_year", "description"]].to_string(
        index=False
    )
)

  dimension  break_year  end_year                                                                                                                                                         description
 sub_pillar        2022      2022                                                                                Sub-pillar tagging (Tossdpillar2 21/22) first appears as trace data: 24 rows in 2022
 sub_pillar        2023      2023                                           Sub-pillar coverage ~51% of pillar-2 rows in 2023; ~99% in 2024 -- cross-year sub-pillar analysis is only clean from 2024
   modality        2021      2021                                                                                                                             Modality code K02 first appears in 2021
  reporters        2019      2024 Reporter base grows from 97 (2019) to 130 (2024) distinct provider codes, counting provider_code != 0; apparent growth in totals partly reflects reporting coverage
methodology        2026      2026                                              RDRM (revised debt-relief reporting methodology) takes effect May 2026 -- applies to vintages published from that date

The table's fifth column, source, names the verification reference for each discontinuity.

Pass years= to narrow the table to breaks that intersect a query's own years, so get_structural_breaks(years=query_years) names only what's relevant to a matching get_tossd(years=query_years) call.

tossd.get_structural_breaks(years=2021)

dimension  break_year  end_year                                                                                                                                                         description                                        source
 modality        2021      2021                                                                                                                             Modality code K02 first appears in 2021                      audit of published files
reporters        2019      2024 Reporter base grows from 97 (2019) to 130 (2024) distinct provider codes, counting provider_code != 0; apparent growth in totals partly reflects reporting coverage distinct provider_code in the published files

Split disbursements across SDG goals. Practical workflow for explode_sdg with weighted sums.
Measure climate and gender finance. Working with the 12 policy marker columns.
Measure Pillar II expenditures in the provider country. In-donor cost methodology and sector breakdown.
Join TOSSD to other country datasets. Joining country data on provider_iso3 and recipient_iso3.

Helpers

Next