146 lines
4.6 KiB
Markdown
146 lines
4.6 KiB
Markdown
# LLM Prediction Providers
|
|
|
|
The oil price direction prediction supports multiple LLM backends behind a shared interface. The active provider is selected via environment variable. All providers return the same response shape and fall back to EWMA if not configured or if the API call fails.
|
|
|
|
## Selecting a Provider
|
|
|
|
Set `LLM_PREDICTION_PROVIDER` in `.env`:
|
|
|
|
```
|
|
LLM_PREDICTION_PROVIDER=anthropic # default
|
|
LLM_PREDICTION_PROVIDER=openai
|
|
LLM_PREDICTION_PROVIDER=gemini
|
|
```
|
|
|
|
Each provider needs its own API key. If the key is missing or empty the provider returns `null` and EWMA is used instead.
|
|
|
|
---
|
|
|
|
## Providers
|
|
|
|
### Anthropic (default)
|
|
|
|
**Key:** `ANTHROPIC_API_KEY`
|
|
**Model:** `ANTHROPIC_MODEL` (default: `claude-sonnet-4-6`)
|
|
|
|
Uses **tool use** with a forced `submit_prediction` tool call — no JSON parsing, guaranteed schema. Structured output is enforced at the API level via `tool_choice: { type: "tool", name: "submit_prediction" }`.
|
|
|
|
Two-phase prediction flow:
|
|
|
|
1. **Context phase** — multi-turn web search (`web_search_20250305` tool) for recent oil/geopolitical news (up to 5 iterations, `pause_turn` loop)
|
|
2. **Submission phase** — once searches are complete, forces a `submit_prediction` tool call with the full conversation context
|
|
|
|
If the context phase fails, falls back to a single-turn basic prediction (tool use only, no web search).
|
|
|
|
```php
|
|
// Structured output schema (enforced by Anthropic)
|
|
'input_schema' => [
|
|
'type' => 'object',
|
|
'properties' => [
|
|
'direction' => ['type' => 'string', 'enum' => ['rising', 'falling', 'flat']],
|
|
'confidence' => ['type' => 'integer', 'minimum' => 0, 'maximum' => 85],
|
|
'reasoning' => ['type' => 'string'],
|
|
],
|
|
'required' => ['direction', 'confidence', 'reasoning'],
|
|
],
|
|
```
|
|
|
|
`PredictionSource`: `llm_with_context` (web search succeeded) or `llm` (basic fallback).
|
|
|
|
---
|
|
|
|
### OpenAI
|
|
|
|
**Key:** `OPENAI_API_KEY`
|
|
**Model:** `OPENAI_MODEL` (default: `gpt-4o-mini`)
|
|
|
|
Uses `response_format: json_schema` with `strict: true`. The schema is sent to the API and the response is guaranteed to match it.
|
|
|
|
```php
|
|
'response_format' => [
|
|
'type' => 'json_schema',
|
|
'json_schema' => [
|
|
'name' => 'oil_prediction',
|
|
'strict' => true,
|
|
'schema' => [
|
|
'type' => 'object',
|
|
'properties' => [
|
|
'direction' => ['type' => 'string', 'enum' => ['rising', 'falling', 'flat']],
|
|
'confidence' => ['type' => 'integer'],
|
|
'reasoning' => ['type' => 'string'],
|
|
],
|
|
'required' => ['direction', 'confidence', 'reasoning'],
|
|
'additionalProperties' => false,
|
|
],
|
|
],
|
|
],
|
|
```
|
|
|
|
Response is extracted from `choices.0.message.content` (a JSON string) and decoded.
|
|
|
|
`PredictionSource`: `llm`
|
|
|
|
---
|
|
|
|
### Gemini
|
|
|
|
**Key:** `GEMINI_API_KEY`
|
|
**Model:** `GEMINI_MODEL` (default: `gemini-2.0-flash`)
|
|
|
|
Uses `responseMimeType: application/json` and `responseSchema` in `generationConfig`. The API key is passed as a query parameter.
|
|
|
|
```php
|
|
'generationConfig' => [
|
|
'responseMimeType' => 'application/json',
|
|
'responseSchema' => [
|
|
'type' => 'OBJECT',
|
|
'properties' => [
|
|
'direction' => ['type' => 'STRING', 'enum' => ['rising', 'falling', 'flat']],
|
|
'confidence' => ['type' => 'INTEGER'],
|
|
'reasoning' => ['type' => 'STRING'],
|
|
],
|
|
'required' => ['direction', 'confidence', 'reasoning'],
|
|
],
|
|
],
|
|
```
|
|
|
|
Response is extracted from `candidates.0.content.parts.0.text` (a JSON string) and decoded.
|
|
|
|
`PredictionSource`: `llm`
|
|
|
|
---
|
|
|
|
## Confidence Caps
|
|
|
|
All providers cap confidence at **85** regardless of what the model returns. EWMA is capped at **65**.
|
|
|
|
---
|
|
|
|
## EWMA Fallback
|
|
|
|
`OilPriceService::generatePrediction()` always runs EWMA first and stores its result. The LLM provider runs after; its result is stored and returned if non-null. If the provider returns null (key missing, API error, malformed response), EWMA is returned instead.
|
|
|
|
```
|
|
generatePrediction()
|
|
├── generateEwmaPrediction() → always stored
|
|
└── provider->predict()
|
|
├── on success → stored and returned (LLM wins)
|
|
└── on null → EWMA returned
|
|
```
|
|
|
|
---
|
|
|
|
## Adding a New Provider
|
|
|
|
1. Create `app/Services/LlmPrediction/YourProvider.php` implementing `OilPredictionProvider`
|
|
2. Add a case to the `match` in `AppServiceProvider::register()`
|
|
3. Add key/model config to `config/services.php` and document the `.env` vars
|
|
|
|
The interface requires one method:
|
|
|
|
```php
|
|
public function predict(Collection $prices): ?PricePrediction;
|
|
```
|
|
|
|
Return `null` on any failure — the orchestrator handles the fallback.
|