Case study
Ask the Filings
Question answering over SEC 10-K filings that cites its sources and refuses when it should.
Try it: replay a real run
Pick a question from the eval and watch each step the system took: search, rerank, the answer with its citations, the number check, or a refusal.
Replayed from the eval run on Oct 14, 2026. No model is called on this page.
AnsweredGraded correct in the run.
RefusedQuestions the filings cannot answer.
MissedAn answerable question it declined.
Source: run 20261014-120024 (Oct 14, 2026). Settings: Hybrid search plus reranker, top 8 passages, company detected from the question. Latency and cost are what that run recorded, and each replay plays at that recorded speed (only the total was timed, so the split across steps is an estimate). The run logged only the final citations. The keyword, meaning, merged and reranked lists were recomputed on 2026-10-15 with the same database, models and settings. For every question here the recomputed top 8 contains every passage the run cited, and its reciprocal rank matches the run.
Pick a question. The replay walks through each step the system took for it in the eval run.
Problem
Most retrieval demos look good and fail quietly. I wanted a system whose quality I could measure, that never states a number it can’t point to.
Constraints and approach
- Every answer cites the filing passages it came from.
- When the filings do not hold the answer, it refuses instead of guessing.
Hybrid search (Postgres full-text plus pgvector embeddings, merged with reciprocal rank fusion) and a cross-encoder reranker, with guardrails before and after the model and an eval that decides every design choice.
Guardrails
| Where | What it does |
|---|---|
| Before the model | Rejects empty questions and questions over 500 characters. |
| In the prompt | Retrieved filing text is marked as data, never instructions. |
| Model output | Each request is labeled as a filing question, investment advice, a prediction or off-topic. Anything but a filing question gets a fixed refusal written in code. |
| After the model | An answer without a valid citation is withheld. Numbers not found in the cited passages are shown as a warning. |
| Demo page | 20 questions per session. |
Results
The eval has 144 questions[1]Source 1The 144 eval questions (evals/questions.jsonl)Open source 1 (opens in a new tab)All sources over 10 companies’ 10-K filings, two fiscal years each[2]Source 2The 20 pinned filings, with accession numbers (filings.json)Open source 2 (opens in a new tab)All sources, downloaded from SEC EDGAR[3]Source 3SEC EDGAR, where the 10-K filings are downloaded fromOpen source 3 (opens in a new tab)All sources: 117 questions the filings can answer and 27 they cannot. The numbers below are from the results table in the repo[4]Source 4Results table in the Ask the Filings READMEOpen source 4 (opens in a new tab)All sources.
- Answerable questions correct
- 111 of 117
- Likely range 89 to 98% (95% confidence)
- Wrong answers
- 0
- The other 6 said the answer was not found
- Main figure right, checked in code
- 65 of 67
- The 2 misses are refusals
- Answers with every number in the cited passages
- 112 of 112
- Unanswerable questions refused
- 26 of 27
- Guardrail tests passed
- 21 of 21
- Cost per question
- $0.007
- Median latency
- 5.5 s
Retrieval ablation
I measured each search setup on its own. Keyword search beats the embedding model here, because bge-small does not handle numbers and line items well. The cross-encoder reranker on top of hybrid search does most of the work.
Eval setting: Company detected from the question
| Search setup | Retrieval recall |
|---|---|
| Embeddings only | 0.60 |
| Keywords only (Postgres full-text) | 0.68 |
| Hybrid (both, merged with RRF) | 0.74 |
| Hybrid + reranker (default) | 0.89 |
| Hybrid + reranker, no company detection | 0.77 |
Refusals on unanswerable questions
| Type | Example | Refused |
|---|---|---|
| Company not loaded | What was Intel’s gross margin in 2025? | 6 of 6 |
| Year not loaded | What was Apple’s total net sales in fiscal 2015? | 6 of 6 |
| Prediction | What will NVIDIA’s total revenue be in fiscal 2030? | 5 of 5 |
| Not disclosed | How many iPhone units did Apple sell in fiscal 2025? | 4 of 4 |
| False premise | Why did Apple discontinue the iPhone in fiscal 2025? | 5 of 6 |
Sources
- [1]The 144 eval questions (evals/questions.jsonl) (opens in a new tab)Back to where it is cited
- [2]The 20 pinned filings, with accession numbers (filings.json) (opens in a new tab)Back to where it is cited
- [3]SEC EDGAR, where the 10-K filings are downloaded from (opens in a new tab)Back to where it is cited
- [4]Results table in the Ask the Filings README (opens in a new tab)Back to where it is cited
Links and stack
- Python
- PostgreSQL
- pgvector
- Gemini API
- Cross-encoder reranker
- Docker