How factors are cleaned

Every factor is cleaned the same way whatever its library: a unit is scaled only by a fixed ratio, a label is mapped to a closed vocabulary only when the mapping is certain, and each change is recorded on the row it touched. Nothing is guessed. When a conversion would need information the publisher did not give, the row keeps its own basis or is not imported, and you can see which. Each library's own steps are described in Libraries.

Units: one canonical basis per factor#

A value is stored as kg CO2e per one canonical unit (kg, kwh, l, km, t.km, p.km, m2, unit, eur, usd, jpy and a few more). Only definitional conversions are applied, and the value keeps its multiplier:

• stored value = published value × unit_conversion_factor;
• published value = value_co2e_native / unit_conversion_factor, in the publisher's unit_raw.

ADEME's 2014 edition of butane combustion, which covers accounting years up to 2018, is published as 63.8 kg CO2e per GJ; it is stored as 63.8 × 0.0036 = 0.22968 per kWh (the relational model runs that trace). Its 2019 edition publishes the factor per kWh already, so the current row's multiplier is 1.0 and its value is 0.23. EXIOBASE publishes per million euros:

SELECT f.unit_code, v.unit_raw,
  round(v.unit_conversion_factor::numeric, 6) AS unit_conversion_factor,
  round(f.value_co2e_native::numeric, 8) AS value_co2e_native,
  round((f.value_co2e_native / v.unit_conversion_factor)::numeric, 0) AS published_value
FROM open_ef.factors_flat f
JOIN open_ef.emission_factor_value v ON v.id = f.value_id
WHERE f.library = 'exiobase'
  AND f.slug = 'ef-exiobase-animal_products_nec-fr-eur-cradle_to_gate-e2b36c2f'
  AND f.is_default_indicator
  AND f.applies_from_year = f.currency_year

Watch out. Spend factors are served once per accounting year, re-priced by price_index_ratio. The division gives back the published number only on the row where applies_from_year equals currency_year (ratio 1), as above. On the underlying emission_factor_value it holds on every row. See Match a spend factor to your spend year.

Every conversion that carries a published label, on default top-level factors of the public libraries:

An ÖKOBAUDAT declaration reports results for a declared amount, often 1,000 kg, so its multiplier absorbs that amount as well as the unit.

EPA's vehicle-mile row is a published label, not a different counted quantity: DESNZ and ADEME file the same kind of car under plain km, and nothing converts between the two (the five transport bases).

A conversion is missing from that table only where the value carries no label of its own, and at this scope every one of those sits on a total open_ef summed from children: the 13 DESNZ well-to-use rows per kWh, for example, are stored through the same ×0.0036 as the GJ rows they sum.

Heating value and context-dependent bases#

Two rules keep a lookalike unit from matching the wrong query:

• Gross heating value is its own unit. kWh PCS, Gross CV, SCV and mmBtu all land on kwh_hhv, never on kwh. A plain energy label is on the net (lower) heating value.
• A unit that needs your context keeps a qualified code. A factor per kg of nitrogen is kg_n, not kg: it applies to the nitrogen in a fertiliser, not to its mass. Such a factor has basis_kind = 'context_dependent' and a reference_basis saying what you must supply (kg_n, kg_p2o5, kg_k2o, kg_active_substance, ha_yr, m2_nfa, m2_floor, m2_wall, m2_roof, kwh_hhv). Units with no safe bridge at all are not imported: toe, kg live weight and peq.km resolve to nothing, and so does any label no rule covers, such as kgCO2e/m3.km or kgCO2e/meal. A refused row is archived in source_row with its reason, skipped_unit: non-canonicalizable basis 'kgCO2e/toe ICV' on ADEME's tonne-of-oil-equivalent rows. teu.km was in that group until it became a canonical code in its own right.

SELECT f.library, count(*) AS context_dependent_factors
FROM open_ef.factors_flat f
JOIN open_ef.emission_factor ef ON ef.id = f.factor_id
WHERE f.library IN ('ademe','agribalyse','exiobase','desnz','miterd','aib','epa','useeio','openceda','okobaudat')
  AND ef.basis_kind = 'context_dependent'
  AND f.is_default_indicator AND f.is_latest_activity_year AND f.is_top_parent
GROUP BY f.library
ORDER BY f.library

Units gives the rules behind the refusals.

Regions: one code space#

Every factor has a region_code from one registry: ISO country codes (FR, ES, GB), subdivisions (FR-34), macro regions (EUROPE, GLOBAL) and publisher complements such as EXIOBASE_ROW_EUROPE. The publisher's label survives in region_raw, and geography_resolution records why each label became its code: France continentale became FR by rule, WA became EXIOBASE_ROW_ASIA_PACIFIC (the full table).

Labels are matched after Unicode normalization, case folding and whitespace collapsing. A label with no known mapping is never guessed.

Where a publisher states no region, the library's documented region is assigned and region_origin says assigned: AGRIBALYSE models French consumption, so its factors are FR. The view's region_is_publisher_default flags the same thing.

System boundary: one vocabulary#

The publisher's description of what a number covers is mapped to a closed list of system_boundary codes: fuel burning only is combustion, extraction and delivery before use is upstream_fuel, raw materials to factory gate is cradle_to_gate, a transport stage inside a product total is the contribution transport. System boundaries lists them.

system_boundary_origin records how the code was reached: source (the record states it), assigned (the publisher states a default for the whole library), inferred, or unknown. unknown means the publisher states no boundary, and the code is then unknown too: an explicit gap, never a stand-in.

What a boundary actually includes#

The boundary code names the slice; the scope says what the slice contains. 21 boundary_scope_term questions (raw_materials, packaging, capital_goods, use_phase, end_of_life and the rest) are answered per boundary in system_boundary_scope with in, out or varies. Where the source proves more for one factor, for example a capital-goods component published under a product total, that factor carries its own boundary_profile. boundary_scope on the view is the merged answer.

SELECT f.library, count(*) AS factors_with_own_profile
FROM open_ef.factors_flat f
WHERE f.library IN ('ademe','agribalyse','exiobase','desnz','miterd','aib','epa','useeio','openceda','okobaudat')
  AND f.boundary_profile IS NOT NULL
  AND f.is_default_indicator AND f.is_latest_activity_year AND f.is_top_parent
GROUP BY f.library
ORDER BY f.library

The other public libraries read their boundary's defaults and nothing more.

Watch out. A term missing from boundary_scope means not established, never excluded. Reading an absent term as out understates a factor exactly where it is least documented.

GWP characterization#

Every CO2e number was characterized under some GWP set. The value records it in gwp_method_native and gwp_horizon_years, the named impact method in characterization_model where there is one, and where the information came from in gwp_origin.

Beside the native value, value_co2e_normalized carries the same measure on AR6 GWP100, and gwp_normalization_basis says how: native, invariant, recharacterized, or unconverted when no exact conversion exists and the column repeats the native number. Only rows whose basis is not unconverted are comparable across libraries. GWP has the per-library counts and the refusal rules.

Where a source ships several characterizations of one measurement, each is a sibling value, and is_default_indicator marks the canonical one (for EXIOBASE, ipcc_ar5_direct). Siblings are alternatives: pick one, never add them (GWP siblings).

Grid electricity mixes#

One country's grid is published under several accounting mixes. They are sibling values of one factor, tagged by grid_mix (production, consumption, marginal, residual, supplier_specific, green), with calculation_approach naming the Scope 2 method each serves. A grid factor is stored as published, generation only: transmission losses and the fuels' upstream are separate factors. Market-based vs location-based electricity covers which library publishes which mix.

Gases, biogenic carbon and land use#

value_co2e_native is the headline. The other value_*_native columns are components in kg CO2e, in two families: per gas (value_co2_native, value_ch4_native, value_n2o_native, value_hfcs_native, value_pfcs_native, value_sf6_native, value_nf3_native, value_other_native, value_ch4_biogenic_native) and by carbon origin (value_co2_fossil_native, value_co2_biogenic_native, value_co2_luluc_native, value_removals_native).

NULL means the publisher did not report the component. 0.0 means it reported zero. ADEME's butane combustion row shows both:

SELECT slug, value_co2e_native, value_other_native, value_hfcs_native
FROM open_ef.factors_flat
WHERE library = 'ademe'
  AND slug = 'ef-ademe-butane_maritime_included-europe-kwh-combustion-896083c5'
  AND is_default_indicator
  AND is_latest_activity_year

Where a source ships the carbon-origin split, the parts add up to the headline within the publisher's rounding:

SELECT slug, value_co2e_native,
  round(value_co2_fossil_native::numeric, 9) AS value_co2_fossil_native,
  round(value_co2_biogenic_native::numeric, 9) AS value_co2_biogenic_native,
  round(value_co2_luluc_native::numeric, 9) AS value_co2_luluc_native,
  round((value_co2_fossil_native + value_co2_biogenic_native + value_co2_luluc_native)::numeric, 7) AS sum_of_parts
FROM open_ef.factors_flat
WHERE library = 'agribalyse'
  AND slug = 'ef-agribalyse-liqueur-fr-kg-distribution-2c9c00ac'
  AND is_default_indicator
  AND is_latest_activity_year

Three more rules:

• value_co2_luluc_native is signed as published: negative means net uptake.
• value_removals_native is filled only when the source reports removals separately. It is never derived from a negative land-use number, which would count the removal twice.
• co2e_origin says where the headline came from: source (the publisher printed it), computed_from_gases (the publisher printed a per-gas vector and it was summed under the stated GWP set), or summed_from_children (nobody printed a total, and open_ef summed the factor's parts).

Data quality: one comparable tier#

Each value keeps the publisher's own quality signal (data_quality_rating_native with its data_quality_scheme_native and data_quality_detail_native, or uncertainty_pct) and one cross-library tier, data_quality_normalized: high, medium, low or very_low. data_quality_reason_normalized explains a tier set deliberately rather than read from a native signal. A value marked problematic is kept in the tables but not served in factors_flat. Data quality gives the rules and the per-library tiers.

Time: reference year and applicable years#

reference_year is the year the publisher attaches to the value. applies_from_year and applies_to_year turn successive editions into ranges of accounting years, so each year has exactly one value, and is_latest_activity_year flags the newest. A publisher's own validity dates are kept as publisher_valid_from and publisher_valid_to on the value and never shorten the applicable years. Reference year has the worked example.

Activity classification#

category and sub_category give a top label and a leaf. activity_classification keeps the publisher's whole tree with its native codes, plus any standard scheme the source carries (the schemes per library).

Every published factor also carries open_ef_category, the one scheme authored by open_ef: 13 top-level branches and 80 leaves in use across the public libraries, fewer than the leaves the code list defines (categories.xlsx is the same tree as a spreadsheet). It says what a factor measures, never its life-cycle stage (that is the boundary) and never the buyer's sector. A source value that maps to no node stops the load rather than landing on a guess.

The supplier registry#

Where a publisher names the company a factor describes, the name is resolved to a supplier row and served as supplier_code and supplier_name. supplier_resolution keeps every raw spelling ever seen, so a respelled name lands on the same supplier. The fold normalizes accents, case, punctuation and whitespace but keeps legal-form suffixes: a missed merge can be fixed later, a wrong merge cannot be seen.

Among the public libraries, ÖKOBAUDAT and MITECO fill it. On default top-level factors, ÖKOBAUDAT names 405 suppliers on 3,636 factors (the declaration owner) and MITECO 531 on 531 factors (the Spanish retailer behind a supplier-specific mix). The supplier is an annotation: it records who the record names, not corporate ownership, and it is never part of a factor's identity.

Provenance markers#

Every field open_ef could have filled on the publisher's behalf says who filled it. Filter on source when you need publisher-stated provenance; assigned and inferred are documented decisions, never hidden ones.

SELECT f.library, ef.region_origin, ef.system_boundary_origin, count(*) AS factors
FROM open_ef.factors_flat f
JOIN open_ef.emission_factor ef ON ef.id = f.factor_id
WHERE f.library IN ('ademe', 'agribalyse', 'okobaudat')
  AND f.is_default_indicator AND f.is_latest_activity_year AND f.is_top_parent
GROUP BY 1, 2, 3
ORDER BY 1, 2, 3

Under every marker sits the evidence: activity_raw, region_raw and unit_raw on the value, the source_record, and the archived line in source_row.

Next#

• Guarantees you can rely on
• Parents, children, double counting
• Trace a served value back to its source line