A Self-Improving Harness for QC Skills
How we turn human decisions on our quality checks into better written rules, using the WikiSkill research loop, and why we released our production version as the open-source skillwiki package.
By Vikash Ranjan, Co-Founder & Chief Technical Officer · Published 2026-09-14 · 10 min read
Checklists go stale, and nobody notices which rule is wrong
Our quality checks for generated fashion images run on written rules. A rule says what to look at, what counts as a defect, and what to ignore. People wrote those rules from experience, and the rules were good on the day they were written. Then the inputs changed, the reviewers found new kinds of mistakes, and the rules stayed where they were.
We had plenty of signal about what was going wrong. Every image we assess can later receive a human decision, and every decision that disagrees with ours is evidence that some rule is missing, too broad, or simply wrong. What we did not have was a safe way to turn that evidence into a better rule. Letting a model rewrite the rules directly would produce confident edits with no record of why, and no way to tell a real improvement from noise.
This post is about the loop we built to close that gap, the research it is based on, what we had to change to run it in production, and why we released the result as an open-source package that does not depend on our product.
The research: WikiSkill
The design comes from WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution by Tang, Rashtchian, Ferng, Tomkins, Juan and Vu at Google Research. The paper separates three things that are usually mixed together:
- Raw traces. What actually happened on each task. Once written, a trace is never changed.
- The wiki. A persistent knowledge base of patterns: root causes, workarounds, what tends to go wrong and why. It is never reset and never rolled back.
- Skills. The short instructions the agent reads while doing the work. This is the only layer the agent sees.
Each round, a maintainer reads a small sample of failing and passing traces and updates the wiki. A proposer reads the wiki, the traces and the history of past proposals, and suggests exactly one change to one skill: create it, patch it, or retire it. A gate then measures the changed skill set on a validation set and keeps it only if the score improves. A rejected skill is thrown away. The wiki keeps what was learned either way, including a record of the rejection, so the proposer does not repeat it.
Three rules from the paper mattered most to us, and we kept all three unchanged:
- The agent doing the work never reads the wiki. The paper tested this directly: giving the agent the wiki made its results worse. The wiki is for the maintainer and the proposer.
- One change per round. If two skills change at once, nobody can say which one earned the score.
- The wiki is never rolled back, even when the skill it motivated is rejected.
Why we could not run it as written
The paper describes a benchmark setup: three directories on one disk, a fresh run of the training tasks every round, text traces, and a start from zero skills. Each of those broke against a real deployment.
- No shared disk. Our workers run on a cluster. Two of them must not fork the wiki by writing at the same time.
- Fresh rollouts cost money. Re-running every task every round against paid model APIs is the most expensive part of the loop, and we already run those tasks in production.
- Our traces have images. You cannot diagnose why a garment image was judged wrongly from text alone.
- We do not start empty. We start from human-written checklists that must not be silently rewritten by a model.
- Labels change. A human decision can be corrected later. A wiki claim that rests on a corrected label must say so.
- The strict gate is noisy. The package’s own design notes record why its paired gate exists: on a test set of a few hundred tasks, accepting any rise in the average score accepts random noise about half the time, and every wrong acceptance raises the bar the next candidate must beat.
None of that is a criticism of the paper. It is the normal distance between a method that works and a method that runs for months unattended.
skillwiki: the same loop, built for production
skillwiki is our independent reimplementation. It contains none of the paper authors’ code and none of our product. The core needs no other packages to run. The loop, the roles, the prompts’ structure, the sampling sizes, the patch operations and the stopping rules follow the paper; the paper’s file layout is still available, but as an export of what is stored rather than the storage itself.
What we changed, and why, in the order we needed it:
| Paper | skillwiki | Why |
|---|---|---|
| Files on one disk | Write-once JSON records, each linked to the previous one by a checksum, behind a small storage interface with in-memory, file, PostgreSQL and object-store back ends | Two workers cannot split the history; the store refuses a write if someone else wrote first |
| Re-run every task each round | Any trace source, including production runs that later received a human decision | Those runs are already paid for and follow the real input distribution |
| Text traces | Traces carry images; the proposer fetches them per trace, and each image travels in one turn only | Image defects need the image; sending every image on every turn makes cost grow very fast |
| Start from zero skills | Starting skills you bring, with a protected flag the proposer cannot patch or retire | Human checklists are the baseline, and some rules are not the model’s to change |
| Accept if the mean improves | You choose the gate. A statistical gate that compares the two skill sets task by task ships alongside the paper’s simple rule | The gate is where a wrong decision compounds |
| Native tool calls | Both: a plain-text tool format that works with any provider, and native tool calling for models that support it | Some model APIs only accept plain text |
| Evidence never changes | The host can report traces whose labels were corrected; every citation of them is marked stale, never deleted | The reasoning trail must survive a retracted label |
| Bounded by round count only | A budget on model calls, output tokens, evaluations and wall-clock time, checked before every paid call | A misbehaving loop against a paid API should stop itself |
| One flat skill set | Layered skill sets, outer to inner, each editable or read-only; an inner layer overrides an outer rule instead of editing it | Shared rules must be specialised per workspace without rewriting them for everyone |
| The wiki only grows | An optional pruner merges patterns with the same root cause and retires dead ones, under hard rules it cannot override | A wiki that grows for months eventually exceeds any context window |
| One uninterrupted process | Checkpoints per round, so an interrupted run finishes the round instead of repeating it | Deploys and worker restarts happen |
Smaller additions follow the same pattern. A candidate identical to one already rejected is refused before anyone pays for its validation. Each trace records which skills the agent actually had in front of it, so the proposer can see that a rule was present on most failing tasks and few passing ones. Trace text is redacted before any model reads it. A command line shows, exports and diffs the wiki, because write-once storage should not make it harder to read than a folder. One workspace can inherit another’s wiki, with the inherited evidence marked as coming from elsewhere.
How it plugs into a production system
The package deliberately does not run your agent, store your traces, decide what counts as a label, or publish a skill set. Those stay with the host. The module owns the loop; the host owns everything the loop reads and everything that has consequences.
In our deployment the pieces are filled in like this. Configuration values are not public.
- Trace source. Assessments served with the current skill set that later received a human decision. An assessment whose decision disagrees with ours is a failing trace; one that agrees is a passing trace. The images travel with the trace.
- Stores. The wiki and every candidate skill set are write-once, checksummed records in our own database, next to the human decisions they cite. When a decision is later corrected, the record that cited it is marked, not rewritten.
- Evaluator and gate. A candidate skill set is replayed on a validation set: labelled cases kept aside for testing. It is compared with the current set on the same cases. The gate applies our own acceptance policy on top of the score, including cost.
- Budget and release. Every model call goes through our metering, which can stop a run. A winning skill set is evaluated once more on a separate final set before it can be published, and publication stays a controlled release with its own review.
What building it taught us
The gate decides everything. If the maintainer and proposer do a poor job, the skills still cannot get worse, because the gate throws out bad proposals. If they do a great job but the gate is noisy, the loop still fails, because every wrong acceptance raises the bar the next proposal has to clear. So we spent more time on the gate than on the prompts.
The model will write things the parser did not expect. A skill is a small text file with a header. The proposer was told the header format and still, occasionally, wrote a list where a value was expected, or a field name the renderer could not write back. Early on, that crashed the run and lost the round’s wiki update. Now the parser treats model-written text as untrusted: a bad header is an error the proposer sees in its own loop and fixes, and fields that grant protection or move a skill between layers are honoured only when the host, not the model, wrote them.
The proposer needs to know which rules were in play. Our agent retrieves a few skills per task rather than loading all of them. Without a record of which ones it had, the proposer edited blind. With the record, the obvious candidate to patch is the rule that was present on most failures and few passes.
Budget first. The proposer reads, thinks and reads again, and every step is a paid model call. It is the first thing a new adopter runs, and it needs a spending ceiling before it needs anything else.
Check the host’s copy, not the model’s. The host usually reshapes what the proposer writes before storing it: it assigns ids, checks the format, and folds an override into the rule it replaces. The gate must score the version the host will actually store, not the model’s draft. The module now checks and keeps the host’s copy, and warns if the wording differs from what the proposer wrote. Scoring one version and shipping another is a quiet way to be wrong.
What we claim, and what we do not
The loop is built and verified end to end against real storage, with the model calls scripted so that every branch is exercised: acceptance, rejection, a rejection streak, an interrupted round, a corrected label, an inherited wiki, a pruned wiki. The module’s own test suite runs before every release, and ours runs on every change.
We are not claiming a measured quality gain from live runs yet. The only published results so far are the paper’s, not ours. When we have our own, it will come with the validation and final-set evidence the release process requires.
Open source
We are releasing our implementation of WikiSkill as open source. The package is called skillwiki, it is licensed under Apache-2.0, and the code is at github.com/vikm2o/skillwiki. The core needs no other packages to run. Adapters for PostgreSQL, S3, GCS and the Anthropic and OpenAI APIs are optional extras. The README has a sixty-second example that runs the whole loop with a scripted model on a toy task, and a document that lists every place the package follows or departs from the paper, with the reason.
Two things it will not do, on purpose:
- It does not run your agent. You bring the traces, however you produce them. That is what keeps the wiki away from the agent, and what keeps the module free of your execution, cost and safety decisions.
- It does not decide what a label is or when a skill set may ship. The gate is yours to define, and acceptance in the loop is not release.
If you run agents on written instructions and you have any signal about when they are wrong, the loop applies. Issues and pull requests are welcome, and the paper-differences document is the place to argue with a design choice.