How factors are cleaned

Numbers checked against the live data on

This page shows queries.

Every factor is cleaned the same way whatever its library: a unit is scaled only by a fixed ratio, a label is mapped to a closed vocabulary only when the mapping is certain, and each change is recorded on the row it touched. Nothing is guessed. When a conversion would need information the publisher did not give, the row keeps its own basis or is not imported, and you can see which. Each library's own steps are described in Libraries.

Units: one canonical basis per factor#

A value is stored as kg CO2e per one canonical unit (kg, kwh, l, km, t.km, p.km, m2, unit, eur, usd and a few more). Only definitional conversions are applied, and the value keeps its multiplier:

  • stored value = published value × unit_conversion_factor;
  • published value = value_co2e_native / unit_conversion_factor, in the publisher's unit_raw.

ADEME's 2014 edition of butane combustion, which covers accounting years up to 2018, is published as 63.8 kg CO2e per GJ; it is stored as 63.8 × 0.0036 = 0.22968 per kWh (the relational model runs that trace). Its 2019 edition publishes the factor per kWh already, so the current row's multiplier is 1.0 and its value is 0.23. EXIOBASE publishes per million euros:

One EXIOBASE spend factor as stored (per euro) beside the unit the publisher printed (per million euro) and the conversion factor between them, which recovers the published figure.

Show the SQL
SELECT f.unit_code, v.unit_raw,
  round(v.unit_conversion_factor::numeric, 6) AS unit_conversion_factor,
  round(f.value_co2e_native::numeric, 8) AS value_co2e_native,
  round((f.value_co2e_native / v.unit_conversion_factor)::numeric, 0) AS published_value
FROM open_ef.factors_flat f
JOIN open_ef.emission_factor_value v ON v.id = f.value_id
WHERE f.library = 'exiobase'
  AND f.slug = 'ef-exiobase-animal_products_nec-fr-eur-cradle_to_gate-e2b36c2f'
  AND f.is_default_indicator
  AND f.applies_from_year = f.currency_year
unit_code unit_raw unit_conversion_factor value_co2e_native published_value
eur kg CO2 eq / Meuro 0.000001 3.76017805 3760178

Watch out. Spend factors are served once per accounting year, re-priced by price_index_ratio. The division gives back the published number only on the row where applies_from_year equals currency_year (ratio 1), as above. On the underlying emission_factor_value it holds on every row. See Match a spend factor to your spend year.

Every conversion that carries a published label, on default top-level factors of the public libraries, counting a publisher's own factor as top-level where the only thing above it is a total open-climate.ai estimated (electricity, fuel):

library unit_raw (as published) unit_code unit_conversion_factor
exiobase kg CO2 eq / Meuro eur 0.000001
ademe kgCO2e/keuro (2023) HT, kgCO2e/keuro eur 0.001
ademe kgCO2e/ton, kgCO2e/ton of waste, kgCO2e/ton of clinker, kgCO2e/tonne, kgCO2e/t kg 0.001
desnz tonnes kg 0.001
okobaudat kg kg 0.001
epa kg / short ton material, kg / short ton kg 0.001102311
ademe kgCO2e/ton of N kg_n 0.001
ademe kgCO2e/ton of P2O5 kg_p2o5 0.001
ademe kgCO2e/ton of K2O kg_k2o 0.001
ademe kgCO2e/100 A4 sheets count_item 0.01
ademe kgCO2e/GJ ICV kwh 0.0036
ademe kgCO2e/MJ ICV, kg CO2 eq/MJ kwh 3.6
aib gCO2/kWh kwh 0.001
epa kg / MWh kwh 0.001
epa kg / mmBtu kwh_hhv 0.003412142
ademe kgCO2e/m3, kgCO2e/m3 (n), kgCO2e/m³ l 0.001
desnz cubic metres l 0.001
okobaudat m3 l 0.001
epa kg / gallon l 0.264172052
epa kg / scf l 0.035314667
okobaudat m m 0.001
ademe kgCO2e/ha m2 0.0001
epa kg / passenger-mile p.km 0.621371192
epa kg / vehicle-mile km 0.621371192
epa kg / short ton-mile t.km 0.684944493

An ÖKOBAUDAT declaration reports results for a declared amount, often 1,000 kg, so its multiplier absorbs that amount as well as the unit.

EPA's vehicle-mile row is a published label, not a different counted quantity: DESNZ and ADEME file the same kind of car under plain km, so the mile folds onto that same code (the five transport bases).

A conversion is missing from that table only where the value carries no label of its own, and at this scope every one of those sits on a total open_ef summed from children: the 13 DESNZ well-to-use rows per kWh, for example, are stored through the same ×0.0036 as the GJ rows they sum. No estimated leg is left without a total above it.

Heating value and context-dependent bases#

Two rules keep a lookalike unit from matching the wrong query:

  • Gross heating value is its own unit. kWh PCS, Gross CV, SCV and mmBtu all land on kwh_hhv, never on kwh. A plain energy label is on the net (lower) heating value.
  • A unit that needs your context keeps a qualified code. A factor per kg of nitrogen is kg_n, not kg: it applies to the nitrogen in a fertiliser, not to its mass. Such a factor has basis_kind = 'context_dependent' and a reference_basis saying what you must supply (kg_n, kg_p2o5, kg_k2o, kg_active_substance, ha_yr, m2_nfa, m2_floor, m2_wall, m2_roof, kwh_hhv, and kg_reference_mass for AGRIBALYSE's live animals, per kg of live weight). Units with no safe bridge at all are not imported: toe and peq.km resolve to nothing, and so does any label no rule covers, such as kgCO2e/plant or kgCO2e/m of road. A refused row is archived in source_row with its reason, skipped_unit: non-canonicalizable basis 'kgCO2e/toe ICV' on ADEME's tonne-of-oil-equivalent rows. teu.km was in that group until it became a canonical code in its own right.

How many top-level factors per library carry a unit that needs your context, such as per kg of nitrogen rather than per kg.

Show the SQL
SELECT f.library, count(*) AS context_dependent_factors
FROM open_ef.factors_flat f
JOIN open_ef.emission_factor ef ON ef.id = f.factor_id
WHERE f.library IN ('ademe','agribalyse','exiobase','desnz','miterd','aib','epa','useeio','openceda','okobaudat')
  AND ef.basis_kind = 'context_dependent'
  AND f.is_default_indicator AND f.is_latest_activity_year AND f.is_top_parent
GROUP BY f.library
ORDER BY f.library
library context_dependent_factors
ademe 247
agribalyse 79
desnz 31
epa 63
miterd 1

Units gives the rules behind the refusals.

Regions: one code space#

Every factor has a region_code from one registry: ISO country codes (FR, ES, GB), subdivisions (FR-34), macro regions (EUROPE, GLOBAL) and publisher complements such as EXIOBASE_ROW_EUROPE. The publisher's label survives in region_raw, and geography_resolution records why each label became its code: France continentale became FR_MAINLAND by rule, WA became EXIOBASE_ROW_ASIA_PACIFIC (the full table).

Labels are matched after Unicode normalization, case folding and whitespace collapsing. A label with no known mapping is never guessed.

Where a publisher states no region, the library's documented region is assigned and region_origin says assigned: AGRIBALYSE models French consumption, so its food factors are FR; its farm-gate agriculture factors state their own region and keep source. The view's region_is_publisher_default flags the same thing.

System boundary: one vocabulary#

The publisher's description of what a number covers is mapped to a closed list of system_boundary codes: fuel burning only is combustion, extraction and delivery before use is upstream_fuel, raw materials to factory gate is cradle_to_gate, a transport stage inside a product total is the contribution transport. System boundaries lists them.

system_boundary_origin records how the code was reached: source (the record states it), assigned (the publisher states a default for the whole library, or open-climate.ai set it on a leg or total it estimated), inferred, or unknown. unknown means the publisher states no boundary, and the code is then unknown too: an explicit gap, never a stand-in.

What a boundary actually includes#

The boundary code names the slice; the scope says what the slice contains. 24 boundary_scope_term questions (raw_materials, packaging, capital_goods, use_phase, end_of_life and the rest) are answered per boundary in system_boundary_scope with in, out or varies, where varies means the boundary alone does not decide. Two more layers answer on top of it. A library's method document can settle a term for the whole library, for one of its categories or for one sub-category (publisher_boundary_scope, which requires the document's link), and the source can prove more for one factor, for example a capital-goods component published under a product total, which that factor carries in its own boundary_profile.

boundary_scope on the view is the merged answer: the most specific layer wins, and boundary_scope_source names the layer each term came from (factor, library_sub_category, library_category, library or boundary), or factor_deduction where no layer states the term and open-climate.ai read the mark off the factor's own value, its activity name or the EN 15804 modules it sums, under a published rule. A varies that no layer settles is served as not_stated, so varies never reaches one factor. A term that cannot apply to what the factor measures, such as end of life on a kilowatt-hour of grid electricity or infrastructure on a finished product, is left out: each term lists the subject kinds it fits, and each category has one kind.

Per library, the top-level factors that carry at least one answer of their own, at least one answer from the library's method document, and at least one term nobody states.

Show the SQL
SELECT f.library,
       count(*) AS factors,
       count(*) FILTER (WHERE jsonb_path_exists(f.boundary_scope_source, '$.* ? (@ == "factor")')) AS own_answer,
       count(*) FILTER (WHERE jsonb_path_exists(f.boundary_scope_source, '$.* ? (@ starts with "library")')) AS library_answer,
       count(*) FILTER (WHERE jsonb_path_exists(f.boundary_scope, '$.* ? (@ == "not_stated")')) AS not_stated
FROM open_ef.factors_flat f
WHERE f.library IN ('ademe','agribalyse','exiobase','desnz','miterd','aib','epa','useeio','openceda','okobaudat')
  AND f.is_default_indicator AND f.is_latest_activity_year AND f.is_top_parent
GROUP BY f.library
ORDER BY f.library
library factors own_answer library_answer not_stated
ademe 3148 2047 1345 1002
agribalyse 2879 2451 0 0
aib 34 34 0 0
desnz 1111 529 180 292
epa 360 92 184 64
exiobase 14805 0 14805 0
miterd 730 72 0 72
okobaudat 4130 3907 3859 4084
openceda 119076 0 119076 4146
useeio 392 255 392 392

The three input-output libraries (EXIOBASE, openCEDA, USEEIO) answer capital goods, business travel, commuting and overheads once for the whole library, from their method papers, and USEEIO's 255 totals with margins take retail from their margins leg. ÖKOBAUDAT's totals take each declared EN 15804 module from the module rows under them, and library rows say that module D sits beside the total and that a partial EPD total leaves out the modules it does not declare. ADEME's and DESNZ's energy and vehicle factors say per factor whether a grid or a fuel is in them, ADEME's freight postes say which leg of the journey they price, and library rows answer ADEME's heat networks and electricity legs and DESNZ's and EPA's waste routes. Every AGRIBALYSE food product takes use_phase from its consumption stage. The AIB, EPA and MITECO factors with answers of their own are open-climate.ai's estimates: the totals it built over a publisher's generation or combustion factor, whose profile says which legs they hold (upstream_fuel: in, and td_losses: in for electricity) (electricity, fuel). What stays not_stated is mostly what the method documents leave open: capital goods, travel and overheads on ÖKOBAUDAT and ADEME products (EN 15804 lets a declaration exclude them, so the standard alone does not settle them), infrastructure on openCEDA, and the fuel chain and purchased electricity on every USEEIO total, which its store gate leaves open and its method rows do not answer.

Watch out. A term missing from boundary_scope means it does not apply to what the factor measures, or that nothing establishes it; not_stated names a gap the boundary leaves open and no source closes. Never read either as out: that understates a factor exactly where it is least documented.

GWP characterization#

Every CO2e number was characterized under some GWP set. The value records it in gwp_method_native and gwp_horizon_years, the named impact method in characterization_model where there is one, and where the information came from in gwp_origin.

library gwp_method_native characterization_model gwp_origin
ademe ar6 none assigned (from ADEME's methodology), source where the record names the report
agribalyse ar6 ef_3.1 assigned
aib ar5 on the 34 estimated totals, none (pure CO2) on the generation values under them none assigned
desnz ar5, mixed on the 47 well-to-use totals that sum an ar4 child with an ar5 one none source
epa ar6, ar4, mixed on 82 of the 92 estimated totals none source, assigned on the estimated totals
exiobase ar5 on the default row ipcc_ar5_direct source
miterd ar6, none on CO2-only rows, mixed on the 72 estimated fuel totals none source, assigned on the estimated totals
okobaudat ar6 under ef_3.1, ar5 under ef_3.0, ar4 under cml_2001_baseline ef_3.1, ef_3.0, cml_2001_baseline assigned
openceda ar6 none inferred
useeio ar6 none source

An estimated total adds legs open-climate.ai computed to the publisher's own factor, and it reads mixed where the two were characterized under different reports; a pure-CO2 value is the same under every report, so a total over one takes its legs' report, which is why AIB's totals read ar5 (estimated fuel legs, estimated electricity legs).

Beside the native value, value_co2e_normalized carries the same measure on AR6 GWP100, and gwp_normalization_basis says how: native, invariant, recharacterized, or unconverted when no exact conversion exists for the value or for another part of its family (why) and the column repeats the native number. Only rows whose basis is not unconverted are comparable across libraries. GWP has the per-library counts and the refusal rules.

Where a source ships several characterizations of one measurement, each is a sibling value, and is_default_indicator marks the canonical one (for EXIOBASE, ipcc_ar5_direct). Siblings are alternatives: pick one, never add them (GWP siblings).

Grid electricity mixes#

One country's grid is published under several accounting mixes. They are sibling values of one factor, tagged by grid_mix (production, consumption, marginal, residual, total_supplier_mix, supplier_specific, green), with calculation_approach naming the Scope 2 method each serves. A grid factor is stored as published, generation only: transmission losses and the fuels' upstream are separate factors. Where a publisher prints generation only, open-climate.ai estimates those legs, and where it can a total above them, for the libraries listed here; each is marked is_estimated. Where a publisher prints a residual mix and no green one, open-climate.ai also adds a green value of 0 on the residual's factor, marked grid_mix_origin assigned (why). The green values of the legs and totals open-climate.ai estimates above such a factor are assigned too. Market-based vs location-based electricity covers which library publishes which mix.

Gases, biogenic carbon and land use#

value_co2e_native is the headline. The other value_*_native columns are components in kg CO2e, in two families: per gas (value_co2_native, value_ch4_native, value_n2o_native, value_hfcs_native, value_pfcs_native, value_sf6_native, value_nf3_native, value_other_native, value_ch4_biogenic_native) and by carbon origin (value_co2_fossil_native, value_co2_biogenic_native, value_co2_luluc_native, value_removals_native).

NULL means the publisher did not report the component. 0.0 means it reported zero. ADEME's butane combustion row shows both:

ADEME's butane combustion row: a total, a component reported as zero, and a component the publisher did not report at all, which stays NULL.

Show the SQL
SELECT slug, value_co2e_native, value_other_native, value_hfcs_native
FROM open_ef.factors_flat
WHERE library = 'ademe'
  AND slug = 'ef-ademe-butane_maritime_included-europe-kwh-combustion-896083c5'
  AND is_default_indicator
  AND is_latest_activity_year
slug value_co2e_native value_other_native value_hfcs_native
ef-ademe-butane_maritime_included-europe-kwh-combustion-896083c5 0.23 0.0 NULL

Where a source ships the carbon-origin split, the parts add up to the headline within the publisher's rounding:

An AGRIBALYSE row with its fossil, biogenic and land-use CO2 parts, and their sum beside the published total.

Show the SQL
SELECT slug, value_co2e_native,
  round(value_co2_fossil_native::numeric, 9) AS value_co2_fossil_native,
  round(value_co2_biogenic_native::numeric, 9) AS value_co2_biogenic_native,
  round(value_co2_luluc_native::numeric, 9) AS value_co2_luluc_native,
  round((value_co2_fossil_native + value_co2_biogenic_native + value_co2_luluc_native)::numeric, 7) AS sum_of_parts
FROM open_ef.factors_flat
WHERE library = 'agribalyse'
  AND slug = 'ef-agribalyse-liqueur-fr-kg-distribution-2c9c00ac'
  AND is_default_indicator
  AND is_latest_activity_year
slug value_co2e_native value_co2_fossil_native value_co2_biogenic_native value_co2_luluc_native sum_of_parts
ef-agribalyse-liqueur-fr-kg-distribution-2c9c00ac 0.0153 0.015223822 0.000039996 0.000008169 0.0152720

Three more rules:

  • value_co2_luluc_native is signed as published: negative means net uptake.
  • value_removals_native is filled only when the source reports removals separately. It is never derived from a negative land-use number, which would count the removal twice.
  • co2e_origin says where the headline came from: source (the publisher printed it), computed_from_gases (the publisher printed a per-gas vector and it was summed under the stated GWP set), summed_from_children (nobody printed a total, and open_ef summed the factor's parts), estimated_from_reference (the publisher never printed this part, and open-climate.ai estimated it from published figures), or differenced_from_parent (the publisher printed the total and every other part, and open-climate.ai subtracted: ICM Australia's input-output supplement is its hybrid total minus its process total), or apportioned_from_parent (the publisher printed the total and the share one part takes of it, and open-climate.ai multiplied: Boavizta's use and embodied parts of a device's lifecycle total). An estimated part carries is_estimated, and so does any total that includes one; see estimated electricity legs and estimated fuel legs.

Data quality: one comparable tier#

Each value keeps the publisher's own quality signal (data_quality_rating_native with its data_quality_scheme_native and data_quality_detail_native, or uncertainty_pct) and one cross-library tier, data_quality_normalized: high, medium, low or very_low. data_quality_reason_normalized explains a tier set deliberately rather than read from a native signal. A value marked problematic is kept in the tables but not served in factors_flat. Data quality gives the rules and the per-library tiers.

Time: reference year and applicable years#

reference_year is the year the publisher attaches to the value. applies_from_year and applies_to_year turn successive editions into ranges of accounting years, so each year has exactly one value, and is_latest_activity_year flags the newest. A publisher's own validity dates are kept as publisher_valid_from and publisher_valid_to on the value and never shorten the applicable years. Reference year has the worked example.

Activity classification#

category and sub_category are the branch and leaf labels of open_ef_category, with category_code and sub_category_code beside them. activity_classification keeps the publisher's whole tree with its native codes, plus any standard scheme the source carries (the schemes per library); the publisher's own top and leaf labels are also kept flat on emission_factor as publisher_sector and publisher_sub_sector.

Every published factor carries open_ef_category, the one scheme authored by open_ef: 13 top-level branches and 80 leaves in use across the public libraries, fewer than the leaves the code list defines (categories.xlsx is the same tree as a spreadsheet). It says what a factor measures, never its life-cycle stage (that is the boundary) and never the buyer's sector. A source value that maps to no node stops the load rather than landing on a guess.

The supplier registry#

Where a publisher names the company a factor describes, the name is resolved to a supplier row and served as supplier_code and supplier_name. supplier_resolution keeps every raw spelling ever seen, so a respelled name lands on the same supplier. The fold normalizes accents, case, punctuation and whitespace but keeps legal-form suffixes: a missed merge can be fixed later, a wrong merge cannot be seen.

Among the public libraries, ÖKOBAUDAT, MITECO and ADEME fill it. On default top-level factors, ÖKOBAUDAT names 396 suppliers on 3,580 factors (the declaration owner), MITECO 531 on 531 factors (the Spanish retailer behind a supplier-specific mix) and ADEME 1,033 on 1,041 factors (the operator of a French heat or cold network). The supplier is an annotation: it records who the record names, not corporate ownership, and it is never part of a factor's identity.

Provenance markers#

Every field open_ef could have filled on the publisher's behalf says who filled it. Filter on source when you need publisher-stated provenance; assigned and inferred are documented decisions, never hidden ones.

column values what it qualifies
emission_factor.region_origin source, assigned, inferred The region
emission_factor.system_boundary_origin source, assigned, inferred, unknown The system boundary
emission_factor_value.gwp_origin source, assigned, inferred The GWP set
emission_factor_value.co2e_origin source, computed_from_gases, summed_from_children, estimated_from_reference, differenced_from_parent, apportioned_from_parent The headline total
emission_factor_value.grid_mix_origin source, assigned The grid mix
emission_factor_text.origin source, translated, generated A label in one language
activity_classification.origin source, mapped A classification code

For three libraries, how many top-level factors had their region and their boundary assigned by open_ef rather than read from the publisher, or left as unknown.

Show the SQL
SELECT f.library, ef.region_origin, ef.system_boundary_origin, count(*) AS factors
FROM open_ef.factors_flat f
JOIN open_ef.emission_factor ef ON ef.id = f.factor_id
WHERE f.library IN ('ademe', 'agribalyse', 'okobaudat')
  AND f.is_default_indicator AND f.is_latest_activity_year AND f.is_top_parent
GROUP BY 1, 2, 3
ORDER BY 1, 2, 3
library region_origin system_boundary_origin factors
ademe assigned assigned 32
ademe source assigned 2896
ademe source inferred 26
ademe source unknown 194
agribalyse assigned assigned 2451
agribalyse source source 428
okobaudat source assigned 3856
okobaudat source source 274

Under every marker sits the evidence: activity_raw, region_raw and unit_raw on the value, the source_record, and the archived line in source_row.

Next#