How FeatureKit is evaluated

FeatureKit changes shared behaviour only from evidence: a failing eval case, a recorded trial, or feedback from a real product repo. This page explains how those evals work and what their results do and don’t show. It is written for the website and reviewed against FeatureKit 0.14.1. The rule it summarises is Improving FeatureKit.

How a case works

Each case pairs a short prompt with a scaffolded fixture repository: a small Python items service, or Click or SQLAlchemy at pinned commits. Every case comes from a real failure: an agent coding without an approved plan, implementing from a plan, trusting stale project knowledge, resuming a half-finished plan, or reviewing an untested path.

evals/run.py runs each case through claude plugin eval in two arms with the same model, tools and turn limits:

Bare. The agent loads an empty plugin.
FeatureKit. The agent loads the FeatureKit plugin plus the instruction block that /setup-featurekit writes into a product repo.

Graders are regex checks on the files the agent leaves behind and LLM-judged rubrics on its final answer. Where a case has acceptance tests, they are re-run independently on a copy of each kept workspace. Cost, turns, tokens and time come from the run traces. Cost is the agent’s own cost as the host reports it; the judge’s cost is tracked separately.

Bare agent compared with FeatureKit

The last full-suite comparison is FeatureKit 0.11.0 against a bare agent: five cases, 13 runs per arm.

Case	Checks passed, bare	Checks passed, FeatureKit	Mean cost, bare	Mean cost, FeatureKit
Plan gate (no approved plan)	0 / 4	4 / 4	$0.125	$0.091
Implement an approved deletion	6 / 6	5 / 6	$0.177	$0.213
Resume at slice 2	9 / 9	8 / 9	$0.205	$0.224
Review an untested path	5 / 6	3 / 6	$0.177	$0.293
Stale project knowledge	4 / 12	10 / 12	$0.203	$0.291
Total	24 / 37	30 / 37	$0.181 per run	$0.227 per run

What this supports:

FeatureKit stops for a plan. Across every FeatureKit run of the plan-gate case (12 of 12, over five result sets), the agent wrote no production code before a plan was approved. The bare agent wrote code in both of its runs. This is a designed difference: the bare agent is never told to plan.
It corrects stale project knowledge. FeatureKit fixed a wrong project doc in 2 of 3 runs, the bare agent in 0 of 3.
It costs more. FeatureKit used more turns (229 against 153), more time and about 25% more cost per run. It was cheaper only in the plan-gate case, where it stops before writing code. Without that case the totals are 26/33 against 24/33, within noise.
It is not better at everything. It scored lower on the review case and slightly lower on implement and resume.

No full-suite comparison has been run for 0.12 to 0.14.1. Later releases were checked on the cases each change affected.

Simplifying made it better

FeatureKit 0.10 included JEV routing: an optimiser service, prompt hooks, task capsules and an attempt ledger. On the same five cases it scored 17 of 37. One cause: it routed an explicit “implement the approved plan” request to a skill that only you could start, so the agent stopped and asked you to type the command. Its implementation runs passed the acceptance checks 0 of 6 times, against 6 of 6 for a bare agent.

0.11.0 removed JEV routing completely, after a calibration showed every route matched the floor the caller had already supplied (11 of 11), and it made the implementation skills loadable by the agent. The same suite then scored 30 of 37. See the changelog.

Evals catch regressions in FeatureKit itself

The 0.14.0 small-fix exemption is an example. Its first wording still passed the plan-gate case 3 of 3 times, but cost per run rose from $0.088–0.094 to $0.184–0.204. In every run the agent searched the repo for an exemption before stopping. Pointing the rule at the instructions it had already loaded brought two runs back to $0.093 and $0.096. The third cost $0.158, inside the earlier spread. Correctness held at 3 of 3 throughout.

Cost and routing findings

These come from paired trials recorded before 0.11.0. They shaped today’s rules but are not benchmarks.

Delegating didn’t lower total cost in these trials. A frontier session handing a medium task to a cheaper in-session worker cost the same ($1.158 against $1.161) and took longer. Small tasks cost more in both single runs. So FeatureKit works directly by default and runs a whole slice on a cheaper model only when the slice warrants it.
Escalating with a failure packet was cheaper than restarting. In three paired runs with an injected failure, handing the stronger model a summary of the failed attempt cut its median cost from $0.225 to $0.161, with one file read instead of five.

The knowledge index

The optional index for large Python codebases was tested over 36 runs in an improvement episode.

Hooks delivered it reliably. It reached the agent in 13 of 13 hooked runs, against 2 of 15 when a skill told the agent to use it and 3 of 8 with an always-on trigger line.
Cost fell on impact-analysis tasks. Cost per run fell about 18% across four tasks and was lower on all four. The baselines came from separate runs, so treat this as medium confidence.
Correctness didn’t measurably improve. Checks passed went from 17–18 of 33 to 21 of 33, within run-to-run variance.

What the results don’t show

That FeatureKit makes an agent cheaper overall. On the full suite it costs more, because it plans, verifies and reviews.
That the results apply to every model, host or codebase. They come from Sonnet on Claude Code with Python fixtures.
Figures from single runs, or from mechanisms FeatureKit no longer ships.