How factors are cleaned
Numbers checked against the live data on
Every factor is cleaned the same way whatever its library: a unit is scaled only by a fixed ratio, a label is mapped to a closed vocabulary only when the mapping is certain, and each change is recorded on the row it touched. Nothing is guessed. When a conversion would need information the publisher did not give, the row keeps its own basis or is not imported, and you can see which. Each library's own steps are described in Libraries.
Units: one canonical basis per factor#
A value is stored as kg CO2e per one canonical unit (kg, kwh, l, km, t.km, p.km, m2, unit, eur, usd and a few more). Only definitional conversions are applied, and the value keeps its multiplier:
- stored value = published value ×
unit_conversion_factor; - published value =
value_co2e_native / unit_conversion_factor, in the publisher'sunit_raw.
ADEME's 2014 edition of butane combustion, which covers accounting years up to 2018, is published as 63.8 kg CO2e per GJ; it is stored as 63.8 × 0.0036 = 0.22968 per kWh (the relational model runs that trace). Its 2019 edition publishes the factor per kWh already, so the current row's multiplier is 1.0 and its value is 0.23. EXIOBASE publishes per million euros:
One EXIOBASE spend factor as stored (per euro) beside the unit the publisher printed (per million euro) and the conversion factor between them, which recovers the published figure.
Show the SQL
SELECT f.unit_code, v.unit_raw,
round(v.unit_conversion_factor::numeric, 6) AS unit_conversion_factor,
round(f.value_co2e_native::numeric, 8) AS value_co2e_native,
round((f.value_co2e_native / v.unit_conversion_factor)::numeric, 0) AS published_value
FROM open_ef.factors_flat f
JOIN open_ef.emission_factor_value v ON v.id = f.value_id
WHERE f.library = 'exiobase'
AND f.slug = 'ef-exiobase-animal_products_nec-fr-eur-cradle_to_gate-e2b36c2f'
AND f.is_default_indicator
AND f.applies_from_year = f.currency_year
| unit_code | unit_raw | unit_conversion_factor | value_co2e_native | published_value |
|---|---|---|---|---|
| eur | kg CO2 eq / Meuro | 0.000001 | 3.76017805 | 3760178 |
Watch out. Spend factors are served once per accounting year, re-priced by
price_index_ratio. The division gives back the published number only on the row whereapplies_from_yearequalscurrency_year(ratio 1), as above. On the underlyingemission_factor_valueit holds on every row. See Match a spend factor to your spend year.
Every conversion that carries a published label, on default top-level factors of the public libraries, counting a publisher's own factor as top-level where the only thing above it is a total open-climate.ai estimated (electricity, fuel):
| library | unit_raw (as published) |
unit_code |
unit_conversion_factor |
|---|---|---|---|
| exiobase | kg CO2 eq / Meuro | eur | 0.000001 |
| ademe | kgCO2e/keuro (2023) HT, kgCO2e/keuro | eur | 0.001 |
| ademe | kgCO2e/ton, kgCO2e/ton of waste, kgCO2e/ton of clinker, kgCO2e/tonne, kgCO2e/t | kg | 0.001 |
| desnz | tonnes | kg | 0.001 |
| okobaudat | kg | kg | 0.001 |
| epa | kg / short ton material, kg / short ton | kg | 0.001102311 |
| ademe | kgCO2e/ton of N | kg_n | 0.001 |
| ademe | kgCO2e/ton of P2O5 | kg_p2o5 | 0.001 |
| ademe | kgCO2e/ton of K2O | kg_k2o | 0.001 |
| ademe | kgCO2e/100 A4 sheets | count_item | 0.01 |
| ademe | kgCO2e/GJ ICV | kwh | 0.0036 |
| ademe | kgCO2e/MJ ICV, kg CO2 eq/MJ | kwh | 3.6 |
| aib | gCO2/kWh | kwh | 0.001 |
| epa | kg / MWh | kwh | 0.001 |
| epa | kg / mmBtu | kwh_hhv | 0.003412142 |
| ademe | kgCO2e/m3, kgCO2e/m3 (n), kgCO2e/m³ | l | 0.001 |
| desnz | cubic metres | l | 0.001 |
| okobaudat | m3 | l | 0.001 |
| epa | kg / gallon | l | 0.264172052 |
| epa | kg / scf | l | 0.035314667 |
| okobaudat | m | m | 0.001 |
| ademe | kgCO2e/ha | m2 | 0.0001 |
| epa | kg / passenger-mile | p.km | 0.621371192 |
| epa | kg / vehicle-mile | km | 0.621371192 |
| epa | kg / short ton-mile | t.km | 0.684944493 |
An ÖKOBAUDAT declaration reports results for a declared amount, often 1,000 kg, so its multiplier absorbs that amount as well as the unit.
EPA's vehicle-mile row is a published label, not a different counted quantity: DESNZ and ADEME file the same kind of car under plain km, so the mile folds onto that same code (the five transport bases).
A conversion is missing from that table only where the value carries no label of its own, and at this scope every one of those sits on a total open_ef summed from children: the 13 DESNZ well-to-use rows per kWh, for example, are stored through the same ×0.0036 as the GJ rows they sum. No estimated leg is left without a total above it.
Heating value and context-dependent bases#
Two rules keep a lookalike unit from matching the wrong query:
- Gross heating value is its own unit.
kWh PCS,Gross CV,SCVandmmBtuall land onkwh_hhv, never onkwh. A plain energy label is on the net (lower) heating value. - A unit that needs your context keeps a qualified code. A factor per kg of nitrogen is
kg_n, notkg: it applies to the nitrogen in a fertiliser, not to its mass. Such a factor hasbasis_kind = 'context_dependent'and areference_basissaying what you must supply (kg_n,kg_p2o5,kg_k2o,kg_active_substance,ha_yr,m2_nfa,m2_floor,m2_wall,m2_roof,kwh_hhv, andkg_reference_massfor AGRIBALYSE's live animals, per kg of live weight). Units with no safe bridge at all are not imported:toeandpeq.kmresolve to nothing, and so does any label no rule covers, such askgCO2e/plantorkgCO2e/m of road. A refused row is archived insource_rowwith its reason,skipped_unit: non-canonicalizable basis 'kgCO2e/toe ICV'on ADEME's tonne-of-oil-equivalent rows.teu.kmwas in that group until it became a canonical code in its own right.
How many top-level factors per library carry a unit that needs your context, such as per kg of nitrogen rather than per kg.
Show the SQL
SELECT f.library, count(*) AS context_dependent_factors
FROM open_ef.factors_flat f
JOIN open_ef.emission_factor ef ON ef.id = f.factor_id
WHERE f.library IN ('ademe','agribalyse','exiobase','desnz','miterd','aib','epa','useeio','openceda','okobaudat')
AND ef.basis_kind = 'context_dependent'
AND f.is_default_indicator AND f.is_latest_activity_year AND f.is_top_parent
GROUP BY f.library
ORDER BY f.library
| library | context_dependent_factors |
|---|---|
| ademe | 247 |
| agribalyse | 79 |
| desnz | 31 |
| epa | 63 |
| miterd | 1 |
Units gives the rules behind the refusals.
Regions: one code space#
Every factor has a region_code from one registry: ISO country codes (FR, ES, GB), subdivisions (FR-34), macro regions (EUROPE, GLOBAL) and publisher complements such as EXIOBASE_ROW_EUROPE. The publisher's label survives in region_raw, and geography_resolution records why each label became its code: France continentale became FR_MAINLAND by rule, WA became EXIOBASE_ROW_ASIA_PACIFIC (the full table).
Labels are matched after Unicode normalization, case folding and whitespace collapsing. A label with no known mapping is never guessed.
Where a publisher states no region, the library's documented region is assigned and region_origin says assigned: AGRIBALYSE models French consumption, so its food factors are FR; its farm-gate agriculture factors state their own region and keep source. The view's region_is_publisher_default flags the same thing.
System boundary: one vocabulary#
The publisher's description of what a number covers is mapped to a closed list of system_boundary codes: fuel burning only is combustion, extraction and delivery before use is upstream_fuel, raw materials to factory gate is cradle_to_gate, a transport stage inside a product total is the contribution transport. System boundaries lists them.
system_boundary_origin records how the code was reached: source (the record states it), assigned (the publisher states a default for the whole library, or open-climate.ai set it on a leg or total it estimated), inferred, or unknown. unknown means the publisher states no boundary, and the code is then unknown too: an explicit gap, never a stand-in.
What a boundary actually includes#
The boundary code names the slice; the scope says what the slice contains. 24 boundary_scope_term questions (raw_materials, packaging, capital_goods, use_phase, end_of_life and the rest) are answered per boundary in system_boundary_scope with in, out or varies, where varies means the boundary alone does not decide. Two more layers answer on top of it. A library's method document can settle a term for the whole library, for one of its categories or for one sub-category (publisher_boundary_scope, which requires the document's link), and the source can prove more for one factor, for example a capital-goods component published under a product total, which that factor carries in its own boundary_profile.
boundary_scope on the view is the merged answer: the most specific layer wins, and boundary_scope_source names the layer each term came from (factor, library_sub_category, library_category, library or boundary), or factor_deduction where no layer states the term and open-climate.ai read the mark off the factor's own value, its activity name or the EN 15804 modules it sums, under a published rule. A varies that no layer settles is served as not_stated, so varies never reaches one factor. A term that cannot apply to what the factor measures, such as end of life on a kilowatt-hour of grid electricity or infrastructure on a finished product, is left out: each term lists the subject kinds it fits, and each category has one kind.
Per library, the top-level factors that carry at least one answer of their own, at least one answer from the library's method document, and at least one term nobody states.
Show the SQL
SELECT f.library,
count(*) AS factors,
count(*) FILTER (WHERE jsonb_path_exists(f.boundary_scope_source, '$.* ? (@ == "factor")')) AS own_answer,
count(*) FILTER (WHERE jsonb_path_exists(f.boundary_scope_source, '$.* ? (@ starts with "library")')) AS library_answer,
count(*) FILTER (WHERE jsonb_path_exists(f.boundary_scope, '$.* ? (@ == "not_stated")')) AS not_stated
FROM open_ef.factors_flat f
WHERE f.library IN ('ademe','agribalyse','exiobase','desnz','miterd','aib','epa','useeio','openceda','okobaudat')
AND f.is_default_indicator AND f.is_latest_activity_year AND f.is_top_parent
GROUP BY f.library
ORDER BY f.library
| library | factors | own_answer | library_answer | not_stated |
|---|---|---|---|---|
| ademe | 3148 | 2047 | 1345 | 1002 |
| agribalyse | 2879 | 2451 | 0 | 0 |
| aib | 34 | 34 | 0 | 0 |
| desnz | 1111 | 529 | 180 | 292 |
| epa | 360 | 92 | 184 | 64 |
| exiobase | 14805 | 0 | 14805 | 0 |
| miterd | 730 | 72 | 0 | 72 |
| okobaudat | 4130 | 3907 | 3859 | 4084 |
| openceda | 119076 | 0 | 119076 | 4146 |
| useeio | 392 | 255 | 392 | 392 |
The three input-output libraries (EXIOBASE, openCEDA, USEEIO) answer capital goods, business travel, commuting and overheads once for the whole library, from their method papers, and USEEIO's 255 totals with margins take retail from their margins leg. ÖKOBAUDAT's totals take each declared EN 15804 module from the module rows under them, and library rows say that module D sits beside the total and that a partial EPD total leaves out the modules it does not declare. ADEME's and DESNZ's energy and vehicle factors say per factor whether a grid or a fuel is in them, ADEME's freight postes say which leg of the journey they price, and library rows answer ADEME's heat networks and electricity legs and DESNZ's and EPA's waste routes. Every AGRIBALYSE food product takes use_phase from its consumption stage. The AIB, EPA and MITECO factors with answers of their own are open-climate.ai's estimates: the totals it built over a publisher's generation or combustion factor, whose profile says which legs they hold (upstream_fuel: in, and td_losses: in for electricity) (electricity, fuel). What stays not_stated is mostly what the method documents leave open: capital goods, travel and overheads on ÖKOBAUDAT and ADEME products (EN 15804 lets a declaration exclude them, so the standard alone does not settle them), infrastructure on openCEDA, and the fuel chain and purchased electricity on every USEEIO total, which its store gate leaves open and its method rows do not answer.
Watch out. A term missing from
boundary_scopemeans it does not apply to what the factor measures, or that nothing establishes it;not_statednames a gap the boundary leaves open and no source closes. Never read either asout: that understates a factor exactly where it is least documented.
GWP characterization#
Every CO2e number was characterized under some GWP set. The value records it in gwp_method_native and gwp_horizon_years, the named impact method in characterization_model where there is one, and where the information came from in gwp_origin.
| library | gwp_method_native |
characterization_model |
gwp_origin |
|---|---|---|---|
| ademe | ar6 | none | assigned (from ADEME's methodology), source where the record names the report |
| agribalyse | ar6 | ef_3.1 | assigned |
| aib | ar5 on the 34 estimated totals, none (pure CO2) on the generation values under them | none | assigned |
| desnz | ar5, mixed on the 47 well-to-use totals that sum an ar4 child with an ar5 one | none | source |
| epa | ar6, ar4, mixed on 82 of the 92 estimated totals | none | source, assigned on the estimated totals |
| exiobase | ar5 on the default row | ipcc_ar5_direct | source |
| miterd | ar6, none on CO2-only rows, mixed on the 72 estimated fuel totals | none | source, assigned on the estimated totals |
| okobaudat | ar6 under ef_3.1, ar5 under ef_3.0, ar4 under cml_2001_baseline | ef_3.1, ef_3.0, cml_2001_baseline | assigned |
| openceda | ar6 | none | inferred |
| useeio | ar6 | none | source |
An estimated total adds legs open-climate.ai computed to the publisher's own factor, and it reads mixed where the two were characterized under different reports; a pure-CO2 value is the same under every report, so a total over one takes its legs' report, which is why AIB's totals read ar5 (estimated fuel legs, estimated electricity legs).
Beside the native value, value_co2e_normalized carries the same measure on AR6 GWP100, and gwp_normalization_basis says how: native, invariant, recharacterized, or unconverted when no exact conversion exists for the value or for another part of its family (why) and the column repeats the native number. Only rows whose basis is not unconverted are comparable across libraries. GWP has the per-library counts and the refusal rules.
Where a source ships several characterizations of one measurement, each is a sibling value, and is_default_indicator marks the canonical one (for EXIOBASE, ipcc_ar5_direct). Siblings are alternatives: pick one, never add them (GWP siblings).
Grid electricity mixes#
One country's grid is published under several accounting mixes. They are sibling values of one factor, tagged by grid_mix (production, consumption, marginal, residual, total_supplier_mix, supplier_specific, green), with calculation_approach naming the Scope 2 method each serves. A grid factor is stored as published, generation only: transmission losses and the fuels' upstream are separate factors. Where a publisher prints generation only, open-climate.ai estimates those legs, and where it can a total above them, for the libraries listed here; each is marked is_estimated. Where a publisher prints a residual mix and no green one, open-climate.ai also adds a green value of 0 on the residual's factor, marked grid_mix_origin assigned (why). The green values of the legs and totals open-climate.ai estimates above such a factor are assigned too. Market-based vs location-based electricity covers which library publishes which mix.
Gases, biogenic carbon and land use#
value_co2e_native is the headline. The other value_*_native columns are components in kg CO2e, in two families: per gas (value_co2_native, value_ch4_native, value_n2o_native, value_hfcs_native, value_pfcs_native, value_sf6_native, value_nf3_native, value_other_native, value_ch4_biogenic_native) and by carbon origin (value_co2_fossil_native, value_co2_biogenic_native, value_co2_luluc_native, value_removals_native).
NULL means the publisher did not report the component. 0.0 means it reported zero. ADEME's butane combustion row shows both:
ADEME's butane combustion row: a total, a component reported as zero, and a component the publisher did not report at all, which stays NULL.
Show the SQL
SELECT slug, value_co2e_native, value_other_native, value_hfcs_native
FROM open_ef.factors_flat
WHERE library = 'ademe'
AND slug = 'ef-ademe-butane_maritime_included-europe-kwh-combustion-896083c5'
AND is_default_indicator
AND is_latest_activity_year
| slug | value_co2e_native | value_other_native | value_hfcs_native |
|---|---|---|---|
| ef-ademe-butane_maritime_included-europe-kwh-combustion-896083c5 | 0.23 | 0.0 | NULL |
Where a source ships the carbon-origin split, the parts add up to the headline within the publisher's rounding:
An AGRIBALYSE row with its fossil, biogenic and land-use CO2 parts, and their sum beside the published total.
Show the SQL
SELECT slug, value_co2e_native,
round(value_co2_fossil_native::numeric, 9) AS value_co2_fossil_native,
round(value_co2_biogenic_native::numeric, 9) AS value_co2_biogenic_native,
round(value_co2_luluc_native::numeric, 9) AS value_co2_luluc_native,
round((value_co2_fossil_native + value_co2_biogenic_native + value_co2_luluc_native)::numeric, 7) AS sum_of_parts
FROM open_ef.factors_flat
WHERE library = 'agribalyse'
AND slug = 'ef-agribalyse-liqueur-fr-kg-distribution-2c9c00ac'
AND is_default_indicator
AND is_latest_activity_year
| slug | value_co2e_native | value_co2_fossil_native | value_co2_biogenic_native | value_co2_luluc_native | sum_of_parts |
|---|---|---|---|---|---|
| ef-agribalyse-liqueur-fr-kg-distribution-2c9c00ac | 0.0153 | 0.015223822 | 0.000039996 | 0.000008169 | 0.0152720 |
Three more rules:
value_co2_luluc_nativeis signed as published: negative means net uptake.value_removals_nativeis filled only when the source reports removals separately. It is never derived from a negative land-use number, which would count the removal twice.co2e_originsays where the headline came from:source(the publisher printed it),computed_from_gases(the publisher printed a per-gas vector and it was summed under the stated GWP set),summed_from_children(nobody printed a total, and open_ef summed the factor's parts),estimated_from_reference(the publisher never printed this part, and open-climate.ai estimated it from published figures), ordifferenced_from_parent(the publisher printed the total and every other part, and open-climate.ai subtracted: ICM Australia's input-output supplement is its hybrid total minus its process total), orapportioned_from_parent(the publisher printed the total and the share one part takes of it, and open-climate.ai multiplied: Boavizta's use and embodied parts of a device's lifecycle total). An estimated part carriesis_estimated, and so does any total that includes one; see estimated electricity legs and estimated fuel legs.
Data quality: one comparable tier#
Each value keeps the publisher's own quality signal (data_quality_rating_native with its data_quality_scheme_native and data_quality_detail_native, or uncertainty_pct) and one cross-library tier, data_quality_normalized: high, medium, low or very_low. data_quality_reason_normalized explains a tier set deliberately rather than read from a native signal. A value marked problematic is kept in the tables but not served in factors_flat. Data quality gives the rules and the per-library tiers.
Time: reference year and applicable years#
reference_year is the year the publisher attaches to the value. applies_from_year and applies_to_year turn successive editions into ranges of accounting years, so each year has exactly one value, and is_latest_activity_year flags the newest. A publisher's own validity dates are kept as publisher_valid_from and publisher_valid_to on the value and never shorten the applicable years. Reference year has the worked example.
Activity classification#
category and sub_category are the branch and leaf labels of open_ef_category, with category_code and sub_category_code beside them. activity_classification keeps the publisher's whole tree with its native codes, plus any standard scheme the source carries (the schemes per library); the publisher's own top and leaf labels are also kept flat on emission_factor as publisher_sector and publisher_sub_sector.
Every published factor carries open_ef_category, the one scheme authored by open_ef: 13 top-level branches and 80 leaves in use across the public libraries, fewer than the leaves the code list defines (categories.xlsx is the same tree as a spreadsheet). It says what a factor measures, never its life-cycle stage (that is the boundary) and never the buyer's sector. A source value that maps to no node stops the load rather than landing on a guess.
The supplier registry#
Where a publisher names the company a factor describes, the name is resolved to a supplier row and served as supplier_code and supplier_name. supplier_resolution keeps every raw spelling ever seen, so a respelled name lands on the same supplier. The fold normalizes accents, case, punctuation and whitespace but keeps legal-form suffixes: a missed merge can be fixed later, a wrong merge cannot be seen.
Among the public libraries, ÖKOBAUDAT, MITECO and ADEME fill it. On default top-level factors, ÖKOBAUDAT names 396 suppliers on 3,580 factors (the declaration owner), MITECO 531 on 531 factors (the Spanish retailer behind a supplier-specific mix) and ADEME 1,033 on 1,041 factors (the operator of a French heat or cold network). The supplier is an annotation: it records who the record names, not corporate ownership, and it is never part of a factor's identity.
Provenance markers#
Every field open_ef could have filled on the publisher's behalf says who filled it. Filter on source when you need publisher-stated provenance; assigned and inferred are documented decisions, never hidden ones.
| column | values | what it qualifies |
|---|---|---|
emission_factor.region_origin |
source, assigned, inferred | The region |
emission_factor.system_boundary_origin |
source, assigned, inferred, unknown | The system boundary |
emission_factor_value.gwp_origin |
source, assigned, inferred | The GWP set |
emission_factor_value.co2e_origin |
source, computed_from_gases, summed_from_children, estimated_from_reference, differenced_from_parent, apportioned_from_parent | The headline total |
emission_factor_value.grid_mix_origin |
source, assigned | The grid mix |
emission_factor_text.origin |
source, translated, generated | A label in one language |
activity_classification.origin |
source, mapped | A classification code |
For three libraries, how many top-level factors had their region and their boundary assigned by open_ef rather than read from the publisher, or left as unknown.
Show the SQL
SELECT f.library, ef.region_origin, ef.system_boundary_origin, count(*) AS factors
FROM open_ef.factors_flat f
JOIN open_ef.emission_factor ef ON ef.id = f.factor_id
WHERE f.library IN ('ademe', 'agribalyse', 'okobaudat')
AND f.is_default_indicator AND f.is_latest_activity_year AND f.is_top_parent
GROUP BY 1, 2, 3
ORDER BY 1, 2, 3
| library | region_origin | system_boundary_origin | factors |
|---|---|---|---|
| ademe | assigned | assigned | 32 |
| ademe | source | assigned | 2896 |
| ademe | source | inferred | 26 |
| ademe | source | unknown | 194 |
| agribalyse | assigned | assigned | 2451 |
| agribalyse | source | source | 428 |
| okobaudat | source | assigned | 3856 |
| okobaudat | source | source | 274 |
Under every marker sits the evidence: activity_raw, region_raw and unit_raw on the value, the source_record, and the archived line in source_row.