fuel-price/docs/superpowers/specs/2026-04-03-fuel-api-ingestion-design.md

Fuel API Ingestion & Historic Storage Design

Date: 2026-04-03 Scope: UK Fuel Finder API integration, database schema for station metadata and historic price storage.

---

Context

The app polls the UK gov.uk Fuel Finder API to collect petrol station prices across the UK (~14,500 stations). Prices are used by the scoring engine to produce fill-up recommendations for users. Historic data is retained indefinitely — a hot table covers the last year for scoring queries, an archive table holds everything older for graphs and comparisons.

---

API

Base URL: https://www.fuel-finder.service.gov.uk/api/v1

Authentication

OAuth 2.0 via JSON POST (not form-encoded).

• Get token: POST /oauth/generate_access_token {"client_id": "...", "client_secret": "..."}
• Refresh token: POST /oauth/regenerate_access_token same payload
• Response includes access_token (Bearer), expires_in: 3600, refresh_token
• Cache token at key fuel_finder_access_token with TTL = expires_in - 60 (3540s)
• On cache miss: fetch new token, store, return

Endpoints

Method	Path	Description
GET	/pfs?batch-number={n}	Station metadata, 500 per batch
GET	/pfs/fuel-prices?batch-number={n}	All station prices, 500 per batch
GET	/pfs/fuel-prices	Incremental prices (recently changed only)

• node_id is the station identifier — consistent across both endpoints (verified against live API)
• Both endpoints return a flat JSON array (no pagination wrapper)
• Total stations: ~14,500 across ~30 batches
• Fuel types in production: E10, E5, B7_STANDARD, B7_PREMIUM, HVO, B10

Polling strategy

• Every 15 minutes: call /pfs/fuel-prices (no batch-number) — returns only recently changed prices
• Once daily (3am): full refresh — iterate all batches of both /pfs and /pfs/fuel-prices to catch any drift

---

Database Schema

stations

One row per petrol filling station. Upserted on full daily refresh and when an incremental poll encounters a new node_id.

node_id                        VARCHAR(64)   PRIMARY KEY
trading_name                   VARCHAR(128)
brand_name                     VARCHAR(64)   NULLABLE
is_same_trading_and_brand      TINYINT(1)
is_supermarket                 TINYINT(1)    DEFAULT 0     — set by StationTaggingService
is_motorway_service_station    TINYINT(1)    DEFAULT 0
is_supermarket_service_station TINYINT(1)    DEFAULT 0
temporary_closure              TINYINT(1)    DEFAULT 0
permanent_closure              TINYINT(1)    DEFAULT 0
permanent_closure_date         DATE          NULLABLE
public_phone_number            VARCHAR(20)   NULLABLE
address_line_1                 VARCHAR(255)  NULLABLE
address_line_2                 VARCHAR(255)  NULLABLE
city                           VARCHAR(100)  NULLABLE
county                         VARCHAR(100)  NULLABLE
country                        VARCHAR(64)   NULLABLE
postcode                       VARCHAR(10)
lat                            DECIMAL(10,7)
lng                            DECIMAL(10,7)
amenities                      JSON          NULLABLE
opening_times                  JSON          NULLABLE
fuel_types                     JSON          NULLABLE      — array of supported fuel type strings
last_seen_at                   DATETIME

station_prices_current

One row per (station_id, fuel_type). Upserted on every price change. Used by scoring engine for current-price lookups — never needs to touch the history table.

station_id          VARCHAR(64)      FK → stations.node_id
fuel_type           ENUM('e10','e5','b7_standard','b7_premium','b10','hvo')
price_pence         SMALLINT UNSIGNED   — price × 100 (e.g. 15990 = 159.90p, never float)
price_effective_at  DATETIME            — price_change_effective_timestamp from API
price_reported_at   DATETIME            — price_last_updated from API
recorded_at         DATETIME            — when this row was last upserted

PRIMARY KEY (station_id, fuel_type)

station_prices

Append-only price history. One row per price change per station+fuel. Partitioned monthly on price_effective_at. Covers the last 12 months (hot table).

id                  BIGINT UNSIGNED  AUTO_INCREMENT
station_id          VARCHAR(64)      FK → stations.node_id
fuel_type           ENUM('e10','e5','b7_standard','b7_premium','b10','hvo')
price_pence         SMALLINT UNSIGNED
price_effective_at  DATETIME
price_reported_at   DATETIME
recorded_at         DATETIME

PRIMARY KEY (id, price_effective_at)
INDEX (station_id, fuel_type, price_effective_at)
INDEX (price_effective_at)
PARTITION BY RANGE (YEAR(price_effective_at) * 100 + MONTH(price_effective_at))

Deduplication: only insert a new row if price_pence differs from the most recent stored value for that (station_id, fuel_type). This prevents duplicates on full refreshes when prices haven't changed.

station_prices_archive

Identical schema to station_prices but no partitioning. Rows older than 12 months are moved here by a monthly scheduled command. Used only for trend graphs and historical comparisons — never queried by the scoring engine.

(same columns as station_prices — no partition)
INDEX (station_id, fuel_type, price_effective_at)
INDEX (price_effective_at)

---

Relationships

stations.node_id  ←──  station_prices_current.station_id
                  ←──  station_prices.station_id
                  ←──  station_prices_archive.station_id

---

Service responsibilities

FuelPriceService

1. Fetch/cache OAuth token
2. Incremental poll every 15 min: GET /pfs/fuel-prices, upsert station_prices_current, insert into station_prices where price changed
3. Full refresh daily: iterate all batches of /pfs (upsert stations) and /pfs/fuel-prices (same price logic)
4. Call StationTaggingService to set is_supermarket and normalise brand_name
5. Dispatch PricesUpdatedEvent after each poll

StationTaggingService

• Matches trading_name against known supermarket brands (case-insensitive)
• Sets is_supermarket = 1 and normalises brand_name

Scheduled archive command

• Runs monthly
• Moves rows from station_prices where price_effective_at < NOW() - 12 months into station_prices_archive
• Drops the corresponding old partition from station_prices

---

Open questions / adjustable later

• Exact partition pre-creation strategy (how many months ahead to create partitions)
• Whether station_prices_archive needs its own partitioning if it grows very large
• Additional fuel types if the API introduces new ones (extend ENUM in migration)