pytest-triage: LLM triage for GitHub issues

01

Problem

Since January 2025, 247 of pytest’s 484 new issues (51%) never got a label. Triage is the first thing to slip when volunteers are busy.^{[1][1] pytest’s issue tracker on GitHub (opens in a new tab)Source 1pytest’s issue tracker on GitHubOpen source 1 (opens in a new tab)All sources}

I wanted to see how much of that triage a low-cost model could take on, and to know how often it is wrong.

02

Constraints and approach

It never writes to GitHub. Every suggestion is scored against the labels the maintainers gave.
I limited the model to fixed lists (the 11 labels, the issues it was shown) and check every pick again in code.
Every choice is made on one year of data, then tested once on a later year.
CI cannot call the model, so it guards quality another way: a changed prompt, schema or model fails the build until it is measured again, and every push re-scores 256 frozen predictions and fails on any drop over 0.02^{[5][5] The regression gate, in the pytest-triage README (opens in a new tab)Source 5The regression gate, in the pytest-triage READMEOpen source 5 (opens in a new tab)All sources}.

Three low-cost LLM calls plus local hybrid search per issue, with fixed-choice outputs, injection guardrails and a trained classifier that decides which labels are shown. It runs in read-only shadow mode and never writes to GitHub, and a CI gate fails any prompt or model change until it is measured again.

How one issue flows through it

A fixed pipeline rather than a free-roaming agent, so cost and latency are predictable and each step can be measured on its own. Search runs locally; only the three model calls cost money.

Pipeline steps for one issue
Step	What it does
Guard	Secrets and file paths are redacted, and the issue text is fenced and marked as data, never instructions.
Assess (1 call)	Suggests labels and missing information, with the five most similar labeled past issues as examples.
Expand (1 call)	Writes three root-cause queries for the duplicate search.
Search (local)	BM25 and embeddings over older issues, merged with reciprocal rank fusion: the issue’s own top 8 plus the queries’ top 8.
Verify (1 call)	Decides which candidates describe the same problem.
Policy check	An issue flagged for injection goes to a human with no suggestions. Otherwise a label is shown only when a trained classifier agrees with it at 0.90 or more.

03

How it is evaluated

The data is split by time, the way the tool would be used. Few-shot examples and duplicate candidates always come from issues opened before the one being triaged, so nothing leaks from the future.

Data splits by time
Split	Issues	Used for
Train and examples	Before 2024, 4,459 labeled	Few-shot examples, duplicate candidates, the classifier
Dev	2024, 256 issues	Every choice: prompt, model, thresholds
Test	2025 onward, 237 issues	Scored once per final choice
Shadow	Opened since Oct 14, 2026	Predicted within about a day, scored later

Every comparison gets a paired bootstrap confidence interval (2,000 resamples). With 11 to 44 examples per label, gaps under about 0.03 macro-F1 are noise, and I would rather say so than report them.

04

Results

Scored on 2025 issues the maintainers had already triaged. The full tables are in the repo README^{[2][2] Results at a glance in the pytest-triage README (opens in a new tab)Source 2Results at a glance in the pytest-triage READMEOpen source 2 (opens in a new tab)All sources}.

Label macro-F1 (best method without an LLM: 0.46): 0.58
Shown labels matching maintainers: 82%; It shows a label only when confident: on 18% of issues
Known duplicates flagged: 7 of 15
Injection attacks that got through: 0 of 104
Cost per issue: $0.0015
Spent building and evaluating: $2.64

Label macro-F1 by predictor2025 test set, 11 labels. Higher is better. From the README’s predictor table.

Label macro-F1 on the 2025 test set for each predictor
Setup	Macro-F1
Keywords from past titles (no LLM)	0.35
kNN vote of similar issues (no LLM)	0.38
Logistic regression (no LLM)	0.46
LLM, first prompt	0.47
LLM, current prompt	0.58

Every predictor is scored on the same 237 issues and 11 labels^{[3][3] Label results by predictor in the pytest-triage README (opens in a new tab)Source 3Label results by predictor in the pytest-triage READMEOpen source 3 (opens in a new tab)All sources}.

Results on the 2025 test set
What	Result	Compared with
Labels, 11 labels (macro-F1)	0.58	0.46 for the best method without an LLM (+0.115, 95% CI +0.06 to +0.16)
Labels shown by default	82% precise, on 18% of issues	0.46 precision when every suggested label is shown
type: regression (F1)	0.71	0.47 for the best method without an LLM
Needs more information (F1)	0.42	0.24 for flagging every issue
Duplicates found	7 of 15	5 of 15 for search alone; 3 of 30 false alarms
Injection attacks that reached a maintainer	0 of 104 runs (19 hand-written attacks, in variants)	1 of 104 on the first run, before I fixed what it found

The LLM clearly beats every method that needs no model calls, but on its own it suggests too many labels. So I trained a classifier to act as a filter: by default a label is shown only when it agrees, and then four in five shown labels match the maintainers’, on fewer issues. Anything sent to the person who opened the issue, such as a request for missing details, has to be 70% precise first. Nothing clears that yet, so those suggestions stay in shadow mode.

What did not work

Ideas that did not make it in
Tried	Outcome
A bigger model	0.51 against 0.49 on dev with the same prompt, at 2.7× the cost per call, and worse at spotting missing information.
Ensembling the LLM and the classifier	About +0.01 on dev and slightly worse on test. The classifier works better as a filter on what to show.
Local cross-encoder rerankers for duplicates	8 of 15 originals against 10 for the current pool, at 6 to 65 seconds per issue.

05

Sources

06

Links and stack

Code on GitHub Live results

Python
Gemini API
BM25
Embeddings
scikit-learn
GitHub Actions