Add fuel API ingestion and historic storage design spec
Includes verified API authentication flow, correct base URL, all DB table schemas for stations, current prices, history, and archive. Fuel types corrected to match live API (B7_STANDARD, B7_PREMIUM). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Ovidiu U
432
.claude/rules/api-data.md
Normal file
432
.claude/rules/api-data.md
Normal file
@@ -0,0 +1,432 @@
|
||||
# External API & Data Sources
|
||||
|
||||
## UK Fuel Finder API (gov.uk) — PRIMARY SOURCE
|
||||
|
||||
- Base URL: `https://www.fuel-finder.service.gov.uk/api/v1/`
|
||||
- Returns: all UK station prices + station metadata (~14,500 stations)
|
||||
- Update frequency: stations report within 30 minutes of price change
|
||||
- Our polling interval: every 15 minutes via scheduler (incremental), full refresh once daily
|
||||
|
||||
### Authentication
|
||||
|
||||
OAuth 2.0 using JSON body POST (not form-encoded). Credentials in `.env` as `FUEL_FINDER_CLIENT_ID` / `FUEL_FINDER_CLIENT_SECRET`.
|
||||
|
||||
**Get token:**
|
||||
```
|
||||
POST /api/v1/oauth/generate_access_token
|
||||
Content-Type: application/json
|
||||
{"client_id": "...", "client_secret": "..."}
|
||||
|
||||
Response: {"access_token": "eyJ...", "token_type": "Bearer", "expires_in": 3600, "refresh_token": "eyJ..."}
|
||||
```
|
||||
|
||||
**Refresh token:**
|
||||
```
|
||||
POST /api/v1/oauth/regenerate_access_token
|
||||
Content-Type: application/json
|
||||
{"client_id": "...", "client_secret": "..."}
|
||||
```
|
||||
|
||||
**Token caching strategy (FuelPriceService):**
|
||||
- Cache the `access_token` with TTL = `expires_in` - 60 seconds (3540s)
|
||||
- Cache key: `fuel_finder_access_token`
|
||||
- On cache miss: call `generate_access_token`, store result, return token
|
||||
- Use the `refresh_token` to regenerate before expiry if needed
|
||||
- Include token in every API request: `Authorization: Bearer {token}`
|
||||
|
||||
Fuel Finder REST API
|
||||
The Fuel Finder API is a REST API that gives a simple, consistent way to request, create and update data. REST stands for Representational State Transfer which is an architectural software style in which standard HTTP request methods are used to retrieve and modify representations of data. This is identical to the process of retrieving a web page or submitting a web form.
|
||||
|
||||
Representational State Transfer (REST) web services
|
||||
In a RESTful API, each data resource has a unique URL and is manipulated using standard HTTP verbs such as:
|
||||
|
||||
GET to request a resource
|
||||
POST to create a resource (not used for read-only endpoints)
|
||||
PUT to change a resource (not used for read-only endpoints)
|
||||
DELETE to remove a resource (not used for read-only endpoints)
|
||||
Example: request a price resource
|
||||
GET: https://api.fuelfinder.service.gov.uk/v1/prices/GB-12345 HTTP/1.1
|
||||
The request uses GET and does not include a request body.
|
||||
|
||||
In a RESTful API, a resource is modified by POSTing a revised resource representation, in this case JSON, to the same resource URL:
|
||||
|
||||
POST: https://api.fuelfinder.service.gov.uk/v1/<endpoint>
|
||||
Content-Type: text/json
|
||||
{
|
||||
"CustomerName": "Joe Bloggs",
|
||||
"Address": "",
|
||||
"etc": etc
|
||||
}
|
||||
REST builds on the features of HTTP. Because each resource has a globally unique URL and can be fetched with GET, REST APIs can benefit from existing network components such as caches and proxies.
|
||||
|
||||
The JSON data format
|
||||
Responses use JSON (JavaScript Object Notation). JSON is a compact, widely used format for storing and exchanging data. Most programming languages support JSON, which makes it well suited to HTTP-based API services.
|
||||
|
||||
#### Endpoints
|
||||
- Endpoints
|
||||
- Method Endpoint
|
||||
- GET Fetch all PFS fuel prices
|
||||
- GET Fetch incremental PFS fuel prices
|
||||
- GET Fetch PFS information
|
||||
- GET Fetch incremental PFS information
|
||||
|
||||
|
||||
```
|
||||
https://www.fuel-finder.service.gov.uk/api/v1/pfs/fuel-prices?batch-number
|
||||
[
|
||||
{
|
||||
"node_id": "0028acef5f3afc41c7e7d56fb285a940dfb64d6fea01cb4accd79c148321112d",
|
||||
"public_phone_number": null,
|
||||
"trading_name": "Alex Fuel Station",
|
||||
"fuel_prices": [
|
||||
{
|
||||
"fuel_type": "E5",
|
||||
"price": 159.9,
|
||||
"price_last_updated": "2026-02-17T16:03:04.938Z",
|
||||
"price_change_effective_timestamp": "2026-02-17T16:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"fuel_type": "E10",
|
||||
"price": 132.9,
|
||||
"price_last_updated": "2026-02-17T16:03:04.938Z",
|
||||
"price_change_effective_timestamp": "2026-02-17T16:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"fuel_type": "B7_STANDARD",
|
||||
"price": 141.9,
|
||||
"price_last_updated": "2026-02-17T16:03:04.938Z",
|
||||
"price_change_effective_timestamp": "2026-02-17T16:00:00.000Z"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"node_id": "01da92125c3751767044d06b202f45da5933f0e16e256fa3e98a16af8386308d",
|
||||
"public_phone_number": "",
|
||||
"trading_name": "Star Garage",
|
||||
"fuel_prices": [
|
||||
{
|
||||
"fuel_type": "E5",
|
||||
"price": 159.9,
|
||||
"price_last_updated": "2026-02-17T16:03:04.938Z",
|
||||
"price_change_effective_timestamp": "2026-02-17T16:00:00.000Z"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"node_id": "020592cd81196efdb61ab2135f837ddf3d2bee4e64346810270f0b088b4c09d8",
|
||||
"public_phone_number": null,
|
||||
"trading_name": "Blue Hills Fuel Station",
|
||||
"fuel_prices": [
|
||||
{
|
||||
"fuel_type": "E5",
|
||||
"price": 159.9,
|
||||
"price_last_updated": "2026-02-17T16:03:04.938Z",
|
||||
"price_change_effective_timestamp": "2026-02-17T16:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"fuel_type": "B7_STANDARD",
|
||||
"price": 141.9,
|
||||
"price_last_updated": "2026-02-17T16:03:04.938Z",
|
||||
"price_change_effective_timestamp": "2026-02-17T16:00:00.000Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
```
|
||||
https://www.fuel-finder.service.gov.uk/api/v1/pfs?batch-number=1
|
||||
[
|
||||
{
|
||||
"node_id": "9b275ab576eeba3c6677984be15ee22a74e54fdfe8e5ea700e84a03178dc4ac1",
|
||||
"public_phone_number": null,
|
||||
"trading_name": "TEST",
|
||||
"is_same_trading_and_brand_name": true,
|
||||
"brand_name": "TEST",
|
||||
"temporary_closure": false,
|
||||
"permanent_closure": false,
|
||||
"permanent_closure_date": null,
|
||||
"is_motorway_service_station": false,
|
||||
"is_supermarket_service_station": false,
|
||||
"location": {
|
||||
"address_line_1": "HALL & WOODHOUSE, TAPLOW BOATYARD, MILL LANE, TAPLOW, MAIDENHEAD, SL6 0AA",
|
||||
"address_line_2": null,
|
||||
"city": "MAIDENHEAD",
|
||||
"country": "England",
|
||||
"county": null,
|
||||
"postcode": "SL6 0AA",
|
||||
"latitude": 51.5268585,
|
||||
"longitude": -0.700361
|
||||
},
|
||||
"amenities": [
|
||||
"water_filling"
|
||||
],
|
||||
"opening_times": {
|
||||
"usual_days": {
|
||||
"monday": {
|
||||
"open": "00:00:00",
|
||||
"close": "00:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"tuesday": {
|
||||
"open": "00:00:00",
|
||||
"close": "00:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"wednesday": {
|
||||
"open": "00:00:00",
|
||||
"close": "00:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"thursday": {
|
||||
"open": "00:00:00",
|
||||
"close": "00:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"friday": {
|
||||
"open": "00:00:00",
|
||||
"close": "00:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"saturday": {
|
||||
"open": "00:00:00",
|
||||
"close": "00:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"sunday": {
|
||||
"open": "00:00:00",
|
||||
"close": "23:59:00",
|
||||
"is_24_hours": true
|
||||
}
|
||||
},
|
||||
"bank_holiday": {
|
||||
"type": "bank holiday",
|
||||
"open_time": "00:00:00",
|
||||
"close_time": "00:00:00",
|
||||
"is_24_hours": false
|
||||
}
|
||||
},
|
||||
"fuel_types": [
|
||||
"E10",
|
||||
"E5",
|
||||
"HVO",
|
||||
"B10"
|
||||
]
|
||||
},
|
||||
{
|
||||
"node_id": "4fd9a4c6b48358b9b5c95989fba100fdcbb87c9e909ed4ce1ad96f64ffb8b56a",
|
||||
"public_phone_number": "+44 7723608248",
|
||||
"trading_name": "TEST FORECOURT 1",
|
||||
"is_same_trading_and_brand_name": true,
|
||||
"brand_name": "TEXACO ONE",
|
||||
"temporary_closure": false,
|
||||
"permanent_closure": null,
|
||||
"permanent_closure_date": null,
|
||||
"is_motorway_service_station": false,
|
||||
"is_supermarket_service_station": false,
|
||||
"location": {
|
||||
"address_line_1": "NEWPORT",
|
||||
"address_line_2": "",
|
||||
"city": "BROUGH",
|
||||
"country": "ENGLAND",
|
||||
"county": "EAST YORKSHIRE",
|
||||
"postcode": "HU15 2RD",
|
||||
"latitude": 51.258503,
|
||||
"longitude": -3.417567
|
||||
},
|
||||
"amenities": [
|
||||
"adblue_packaged",
|
||||
"adblue_pumps",
|
||||
"car_wash",
|
||||
"customer_toilets"
|
||||
],
|
||||
"opening_times": {
|
||||
"usual_days": {
|
||||
"monday": {
|
||||
"open": "06:00:01",
|
||||
"close": "23:00:01",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"tuesday": {
|
||||
"open": "06:00:01",
|
||||
"close": "23:00:01",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"wednesday": {
|
||||
"open": "06:00:01",
|
||||
"close": "23:00:01",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"thursday": {
|
||||
"open": "06:00:01",
|
||||
"close": "23:00:01",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"friday": {
|
||||
"open": "06:00:01",
|
||||
"close": "23:00:01",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"saturday": {
|
||||
"open": "06:00:01",
|
||||
"close": "23:00:01",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"sunday": {
|
||||
"open": "06:00:01",
|
||||
"close": "23:00:01",
|
||||
"is_24_hours": false
|
||||
}
|
||||
},
|
||||
"bank_holiday": {
|
||||
"type": "standard",
|
||||
"open_time": "06:00:01",
|
||||
"close_time": "23:00:01",
|
||||
"is_24_hours": false
|
||||
}
|
||||
},
|
||||
"fuel_types": [
|
||||
"B10"
|
||||
]
|
||||
},
|
||||
{
|
||||
"node_id": "91bdda1c07fa05110a31639cc66932f9ed8bd388d4f6be542a423365bcfd53e1",
|
||||
"public_phone_number": "+442071930000",
|
||||
"trading_name": "SUPERFUEL LOUGHBOROUGH 12",
|
||||
"is_same_trading_and_brand_name": true,
|
||||
"brand_name": "SUPERFUEL STATION 4",
|
||||
"temporary_closure": false,
|
||||
"permanent_closure": null,
|
||||
"permanent_closure_date": null,
|
||||
"is_motorway_service_station": false,
|
||||
"is_supermarket_service_station": false,
|
||||
"location": {
|
||||
"address_line_1": "14 LONDON ROAD",
|
||||
"address_line_2": "FUELVILLE",
|
||||
"city": "LOUGHBOROUGH",
|
||||
"country": "ENGLAND",
|
||||
"county": "LEICESTERSHIRE",
|
||||
"postcode": "LE11 9AA",
|
||||
"latitude": 50.503343,
|
||||
"longitude": -2.12444
|
||||
},
|
||||
"amenities": [
|
||||
"adblue_packaged",
|
||||
"adblue_pumps",
|
||||
"car_wash",
|
||||
"customer_toilets",
|
||||
"water_filling"
|
||||
],
|
||||
"opening_times": {
|
||||
"usual_days": {
|
||||
"monday": {
|
||||
"open": "06:00:00",
|
||||
"close": "22:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"tuesday": {
|
||||
"open": "06:00:00",
|
||||
"close": "22:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"wednesday": {
|
||||
"open": "06:00:00",
|
||||
"close": "22:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"thursday": {
|
||||
"open": "06:00:00",
|
||||
"close": "22:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"friday": {
|
||||
"open": "06:00:00",
|
||||
"close": "22:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"saturday": {
|
||||
"open": "06:00:00",
|
||||
"close": "22:00:00",
|
||||
"is_24_hours": false
|
||||
},
|
||||
"sunday": {
|
||||
"open": "06:00:00",
|
||||
"close": "22:00:00",
|
||||
"is_24_hours": false
|
||||
}
|
||||
},
|
||||
"bank_holiday": {
|
||||
"type": "standard",
|
||||
"open_time": "08:00:00",
|
||||
"close_time": "20:00:00",
|
||||
"is_24_hours": false
|
||||
}
|
||||
},
|
||||
"fuel_types": [
|
||||
"E5",
|
||||
"HVO",
|
||||
"B10",
|
||||
"B7_PREMIUM",
|
||||
"B7_STANDARD"
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### FuelPriceService responsibilities
|
||||
1. Fetch OAuth token (cache it)
|
||||
2. GET all station prices
|
||||
3. Upsert `stations` table with metadata
|
||||
4. Insert new rows into `station_prices` only when price has changed for that station+fuel combo
|
||||
5. Call StationTaggingService to set `is_supermarket` and `brand`
|
||||
6. Dispatch `PricesUpdatedEvent` for downstream processing
|
||||
|
||||
### Deduplication
|
||||
Only insert a new `station_prices` row if price differs from the most recent stored price
|
||||
for that `(station_id, fuel_type)` combination. Avoids row explosion on unchanged prices.
|
||||
|
||||
### Credentials in .env
|
||||
```
|
||||
FUEL_FINDER_CLIENT_ID=
|
||||
FUEL_FINDER_CLIENT_SECRET=
|
||||
FUEL_FINDER_BASE_URL=https://api.fuel-finder.service.gov.uk
|
||||
```
|
||||
|
||||
## Postcodes.io — postcode → lat/lng
|
||||
|
||||
- URL: `https://api.postcodes.io/postcodes/{postcode}`
|
||||
- Free, no API key required
|
||||
- Called once on user registration / when postcode changes
|
||||
- Store resolved `lat` + `lng` on `users` table
|
||||
- Cache postcode lookups for 30 days (postcodes rarely change coordinates)
|
||||
|
||||
## FRED API (St. Louis Fed) — Brent crude direction
|
||||
|
||||
- Series: `DCOILBRENTEU` (daily Brent spot price)
|
||||
- URL: `https://api.stlouisfed.org/fred/series/observations?series_id=DCOILBRENTEU&api_key={key}&sort_order=desc&limit=10&file_type=json`
|
||||
- Free API key required — stored as `FRED_API_KEY` in .env
|
||||
- Fetched once daily via scheduler at 7am
|
||||
- Stored in `brent_prices` table: `(date DATE, price_usd DECIMAL(8,2))`
|
||||
- Only the 5-day trend direction is used by the scoring engine
|
||||
|
||||
## OneSignal — push notifications
|
||||
|
||||
- REST API: `https://oapi.onesignal.com/notifications`
|
||||
- App ID + REST API key stored in .env as `ONESIGNAL_APP_ID`, `ONESIGNAL_API_KEY`
|
||||
- Target by `player_id` (stored in `users.push_token`)
|
||||
- No official Laravel package needed — use Laravel HTTP client (`Http::post(...)`)
|
||||
- Free plan: 10,000 subscribers — sufficient for v1
|
||||
|
||||
## Vonage — WhatsApp + SMS
|
||||
|
||||
- Package: `vonage/client-core` via Composer
|
||||
- Credentials: `VONAGE_KEY`, `VONAGE_SECRET`, `VONAGE_WHATSAPP_FROM` in .env
|
||||
- WhatsApp: Messages API, utility template category (pre-approved)
|
||||
- SMS: SMS API, alphanumeric sender ID "FuelAlert"
|
||||
- All Vonage calls go through NotificationDispatchService — never call Vonage directly from components
|
||||
|
||||
## HTTP client
|
||||
|
||||
Use Laravel's built-in `Http` facade for all external API calls.
|
||||
Always set a timeout: `Http::timeout(10)->get(...)`.
|
||||
Wrap in try/catch — log failures, never let a failed API call crash the scheduler.
|
||||
@@ -26,7 +26,7 @@ Tier is read via `$user->subscribed('basic')`, `->subscribed('plus')`, `->subscr
|
||||
```
|
||||
id BIGINT UNSIGNED AUTO_INCREMENT
|
||||
station_id VARCHAR(64) — Fuel Finder API station identifier
|
||||
fuel_type ENUM('e10','e5','diesel','super_diesel','b10','hvo')
|
||||
fuel_type ENUM('e10','e5','b7_standard','b7_premium','b10','hvo')
|
||||
price_pence SMALLINT UNSIGNED — e.g. 14523 = 145.23p (store as integer × 100)
|
||||
is_supermarket TINYINT(1) DEFAULT 0
|
||||
brand VARCHAR(64) NULLABLE
|
||||
@@ -1,33 +0,0 @@
|
||||
# Local overrides (gitignored — personal only)
|
||||
|
||||
## Local environment
|
||||
|
||||
- PhpStorm with SFTP sync to Proxmox LXC (CT 150, 192.168.1.150)
|
||||
- MySQL on IONOS VPS: 192.168.x.x (update with actual)
|
||||
- Local dev URL: http://fuel-alert.test (Valet) or http://localhost:8000
|
||||
|
||||
## Test credentials (local only)
|
||||
|
||||
- Stripe test publishable key: pk_test_...
|
||||
- Fuel Finder sandbox credentials (if available): see 1Password
|
||||
- Vonage test account: see 1Password
|
||||
- OneSignal test app: see 1Password
|
||||
|
||||
## Deployment
|
||||
|
||||
- Production: IONOS VPS behind Traefik (same setup as uovidiu.com portfolio)
|
||||
- Deploy: git push → SSH → composer install --no-dev → php artisan migrate --force → php artisan queue:restart
|
||||
- Redis: Docker container on IONOS VPS
|
||||
|
||||
## Useful local commands
|
||||
|
||||
```bash
|
||||
# Manually trigger fuel price poll
|
||||
php artisan app:poll-fuel-prices
|
||||
|
||||
# Run scoring for a specific user
|
||||
php artisan app:score-user {user_id}
|
||||
|
||||
# Clear scored results cache
|
||||
php artisan cache:forget scoring_results
|
||||
```
|
||||
68
api-data.md
68
api-data.md
@@ -1,68 +0,0 @@
|
||||
# External API & Data Sources
|
||||
|
||||
## UK Fuel Finder API (gov.uk) — PRIMARY SOURCE
|
||||
|
||||
- Base URL: `https://api.fuel-finder.service.gov.uk/`
|
||||
- Auth: OAuth 2.0 client credentials (client_id + client_secret → Bearer token)
|
||||
- Token stored in cache with TTL matching expiry minus 60 seconds
|
||||
- Returns: all UK station prices + station metadata
|
||||
- Update frequency: stations report within 30 minutes of price change
|
||||
- Our polling interval: every 15 minutes via scheduler
|
||||
|
||||
### FuelPriceService responsibilities
|
||||
1. Fetch OAuth token (cache it)
|
||||
2. GET all station prices
|
||||
3. Upsert `stations` table with metadata
|
||||
4. Insert new rows into `station_prices` only when price has changed for that station+fuel combo
|
||||
5. Call StationTaggingService to set `is_supermarket` and `brand`
|
||||
6. Dispatch `PricesUpdatedEvent` for downstream processing
|
||||
|
||||
### Deduplication
|
||||
Only insert a new `station_prices` row if price differs from the most recent stored price
|
||||
for that `(station_id, fuel_type)` combination. Avoids row explosion on unchanged prices.
|
||||
|
||||
### Credentials in .env
|
||||
```
|
||||
FUEL_FINDER_CLIENT_ID=
|
||||
FUEL_FINDER_CLIENT_SECRET=
|
||||
FUEL_FINDER_BASE_URL=https://api.fuel-finder.service.gov.uk
|
||||
```
|
||||
|
||||
## Postcodes.io — postcode → lat/lng
|
||||
|
||||
- URL: `https://api.postcodes.io/postcodes/{postcode}`
|
||||
- Free, no API key required
|
||||
- Called once on user registration / when postcode changes
|
||||
- Store resolved `lat` + `lng` on `users` table
|
||||
- Cache postcode lookups for 30 days (postcodes rarely change coordinates)
|
||||
|
||||
## FRED API (St. Louis Fed) — Brent crude direction
|
||||
|
||||
- Series: `DCOILBRENTEU` (daily Brent spot price)
|
||||
- URL: `https://api.stlouisfed.org/fred/series/observations?series_id=DCOILBRENTEU&api_key={key}&sort_order=desc&limit=10&file_type=json`
|
||||
- Free API key required — stored as `FRED_API_KEY` in .env
|
||||
- Fetched once daily via scheduler at 7am
|
||||
- Stored in `brent_prices` table: `(date DATE, price_usd DECIMAL(8,2))`
|
||||
- Only the 5-day trend direction is used by the scoring engine
|
||||
|
||||
## OneSignal — push notifications
|
||||
|
||||
- REST API: `https://oapi.onesignal.com/notifications`
|
||||
- App ID + REST API key stored in .env as `ONESIGNAL_APP_ID`, `ONESIGNAL_API_KEY`
|
||||
- Target by `player_id` (stored in `users.push_token`)
|
||||
- No official Laravel package needed — use Laravel HTTP client (`Http::post(...)`)
|
||||
- Free plan: 10,000 subscribers — sufficient for v1
|
||||
|
||||
## Vonage — WhatsApp + SMS
|
||||
|
||||
- Package: `vonage/client-core` via Composer
|
||||
- Credentials: `VONAGE_KEY`, `VONAGE_SECRET`, `VONAGE_WHATSAPP_FROM` in .env
|
||||
- WhatsApp: Messages API, utility template category (pre-approved)
|
||||
- SMS: SMS API, alphanumeric sender ID "FuelAlert"
|
||||
- All Vonage calls go through NotificationDispatchService — never call Vonage directly from components
|
||||
|
||||
## HTTP client
|
||||
|
||||
Use Laravel's built-in `Http` facade for all external API calls.
|
||||
Always set a timeout: `Http::timeout(10)->get(...)`.
|
||||
Wrap in try/catch — log failures, never let a failed API call crash the scheduler.
|
||||
162
docs/superpowers/specs/2026-04-03-fuel-api-ingestion-design.md
Normal file
162
docs/superpowers/specs/2026-04-03-fuel-api-ingestion-design.md
Normal file
@@ -0,0 +1,162 @@
|
||||
# Fuel API Ingestion & Historic Storage Design
|
||||
|
||||
**Date:** 2026-04-03
|
||||
**Scope:** UK Fuel Finder API integration, database schema for station metadata and historic price storage.
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
The app polls the UK gov.uk Fuel Finder API to collect petrol station prices across the UK (~14,500 stations). Prices are used by the scoring engine to produce fill-up recommendations for users. Historic data is retained indefinitely — a hot table covers the last year for scoring queries, an archive table holds everything older for graphs and comparisons.
|
||||
|
||||
---
|
||||
|
||||
## API
|
||||
|
||||
**Base URL:** `https://www.fuel-finder.service.gov.uk/api/v1`
|
||||
|
||||
### Authentication
|
||||
|
||||
OAuth 2.0 via JSON POST (not form-encoded).
|
||||
|
||||
- **Get token:** `POST /oauth/generate_access_token` `{"client_id": "...", "client_secret": "..."}`
|
||||
- **Refresh token:** `POST /oauth/regenerate_access_token` same payload
|
||||
- Response includes `access_token` (Bearer), `expires_in: 3600`, `refresh_token`
|
||||
- Cache token at key `fuel_finder_access_token` with TTL = `expires_in - 60` (3540s)
|
||||
- On cache miss: fetch new token, store, return
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/pfs?batch-number={n}` | Station metadata, 500 per batch |
|
||||
| GET | `/pfs/fuel-prices?batch-number={n}` | All station prices, 500 per batch |
|
||||
| GET | `/pfs/fuel-prices` | Incremental prices (recently changed only) |
|
||||
|
||||
- `node_id` is the station identifier — consistent across both endpoints (verified against live API)
|
||||
- Both endpoints return a flat JSON array (no pagination wrapper)
|
||||
- Total stations: ~14,500 across ~30 batches
|
||||
- Fuel types in production: `E10`, `E5`, `B7_STANDARD`, `B7_PREMIUM`, `HVO`, `B10`
|
||||
|
||||
### Polling strategy
|
||||
|
||||
- **Every 15 minutes:** call `/pfs/fuel-prices` (no batch-number) — returns only recently changed prices
|
||||
- **Once daily (3am):** full refresh — iterate all batches of both `/pfs` and `/pfs/fuel-prices` to catch any drift
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
### `stations`
|
||||
|
||||
One row per petrol filling station. Upserted on full daily refresh and when an incremental poll encounters a new `node_id`.
|
||||
|
||||
```
|
||||
node_id VARCHAR(64) PRIMARY KEY
|
||||
trading_name VARCHAR(128)
|
||||
brand_name VARCHAR(64) NULLABLE
|
||||
is_same_trading_and_brand TINYINT(1)
|
||||
is_supermarket TINYINT(1) DEFAULT 0 — set by StationTaggingService
|
||||
is_motorway_service_station TINYINT(1) DEFAULT 0
|
||||
is_supermarket_service_station TINYINT(1) DEFAULT 0
|
||||
temporary_closure TINYINT(1) DEFAULT 0
|
||||
permanent_closure TINYINT(1) DEFAULT 0
|
||||
permanent_closure_date DATE NULLABLE
|
||||
public_phone_number VARCHAR(20) NULLABLE
|
||||
address_line_1 VARCHAR(255) NULLABLE
|
||||
address_line_2 VARCHAR(255) NULLABLE
|
||||
city VARCHAR(100) NULLABLE
|
||||
county VARCHAR(100) NULLABLE
|
||||
country VARCHAR(64) NULLABLE
|
||||
postcode VARCHAR(10)
|
||||
lat DECIMAL(10,7)
|
||||
lng DECIMAL(10,7)
|
||||
amenities JSON NULLABLE
|
||||
opening_times JSON NULLABLE
|
||||
fuel_types JSON NULLABLE — array of supported fuel type strings
|
||||
last_seen_at DATETIME
|
||||
```
|
||||
|
||||
### `station_prices_current`
|
||||
|
||||
One row per `(station_id, fuel_type)`. Upserted on every price change. Used by scoring engine for current-price lookups — never needs to touch the history table.
|
||||
|
||||
```
|
||||
station_id VARCHAR(64) FK → stations.node_id
|
||||
fuel_type ENUM('e10','e5','b7_standard','b7_premium','b10','hvo')
|
||||
price_pence SMALLINT UNSIGNED — price × 100 (e.g. 15990 = 159.90p, never float)
|
||||
price_effective_at DATETIME — price_change_effective_timestamp from API
|
||||
price_reported_at DATETIME — price_last_updated from API
|
||||
recorded_at DATETIME — when this row was last upserted
|
||||
|
||||
PRIMARY KEY (station_id, fuel_type)
|
||||
```
|
||||
|
||||
### `station_prices`
|
||||
|
||||
Append-only price history. One row per price change per station+fuel. Partitioned monthly on `price_effective_at`. Covers the last 12 months (hot table).
|
||||
|
||||
```
|
||||
id BIGINT UNSIGNED AUTO_INCREMENT
|
||||
station_id VARCHAR(64) FK → stations.node_id
|
||||
fuel_type ENUM('e10','e5','b7_standard','b7_premium','b10','hvo')
|
||||
price_pence SMALLINT UNSIGNED
|
||||
price_effective_at DATETIME
|
||||
price_reported_at DATETIME
|
||||
recorded_at DATETIME
|
||||
|
||||
PRIMARY KEY (id, price_effective_at)
|
||||
INDEX (station_id, fuel_type, price_effective_at)
|
||||
INDEX (price_effective_at)
|
||||
PARTITION BY RANGE (YEAR(price_effective_at) * 100 + MONTH(price_effective_at))
|
||||
```
|
||||
|
||||
**Deduplication:** only insert a new row if `price_pence` differs from the most recent stored value for that `(station_id, fuel_type)`. This prevents duplicates on full refreshes when prices haven't changed.
|
||||
|
||||
### `station_prices_archive`
|
||||
|
||||
Identical schema to `station_prices` but no partitioning. Rows older than 12 months are moved here by a monthly scheduled command. Used only for trend graphs and historical comparisons — never queried by the scoring engine.
|
||||
|
||||
```
|
||||
(same columns as station_prices — no partition)
|
||||
INDEX (station_id, fuel_type, price_effective_at)
|
||||
INDEX (price_effective_at)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Relationships
|
||||
|
||||
```
|
||||
stations.node_id ←── station_prices_current.station_id
|
||||
←── station_prices.station_id
|
||||
←── station_prices_archive.station_id
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Service responsibilities
|
||||
|
||||
**`FuelPriceService`**
|
||||
1. Fetch/cache OAuth token
|
||||
2. Incremental poll every 15 min: GET `/pfs/fuel-prices`, upsert `station_prices_current`, insert into `station_prices` where price changed
|
||||
3. Full refresh daily: iterate all batches of `/pfs` (upsert `stations`) and `/pfs/fuel-prices` (same price logic)
|
||||
4. Call `StationTaggingService` to set `is_supermarket` and normalise `brand_name`
|
||||
5. Dispatch `PricesUpdatedEvent` after each poll
|
||||
|
||||
**`StationTaggingService`**
|
||||
- Matches `trading_name` against known supermarket brands (case-insensitive)
|
||||
- Sets `is_supermarket = 1` and normalises `brand_name`
|
||||
|
||||
**Scheduled archive command**
|
||||
- Runs monthly
|
||||
- Moves rows from `station_prices` where `price_effective_at < NOW() - 12 months` into `station_prices_archive`
|
||||
- Drops the corresponding old partition from `station_prices`
|
||||
|
||||
---
|
||||
|
||||
## Open questions / adjustable later
|
||||
|
||||
- Exact partition pre-creation strategy (how many months ahead to create partitions)
|
||||
- Whether `station_prices_archive` needs its own partitioning if it grows very large
|
||||
- Additional fuel types if the API introduces new ones (extend ENUM in migration)
|
||||
Reference in New Issue
Block a user