The Signal · From the workshop
Prompt Lab: letting a model fix my prompt
An open, model-agnostic take on automatic prompt optimization. On a real six-ticket test it lifted a small model from 0% to 73% in one round, then two more rounds made it worse.
This is a project I built. Code is MIT-licensed and open for review.
What it is
Prompt Lab runs a prompt template on a handful of test cases, has a judge model score every answer, asks an optimizer model why the weak ones failed, rewrites the prompt and repeats until the scores stop improving. It is a Python library and command line tool, plus a browser version at readsigil.com/lab that uses your own key. It works with any OpenAI-compatible endpoint, and text inside <protect> tags is never rewritten. The loop is inspired by AWS Bedrock Advanced Prompt Optimization. This is an independent project with no affiliation.
The test
Six made-up customer messages, each with an expected label: a category (billing, how_to, bug, feature_request, security) and a priority (low to urgent). The starting prompt was deliberately lazy: “Classify this customer message”, plus one protected line, “Never promise refunds or timelines.” The rubric asked for bare JSON with the exact labels. The target was allam-2-7b on Groq, a small 7B model. openai/gpt-oss-120b was both optimizer and judge. The run finished on 2026-10-15.
What happened
Round 0: 0%. The small model wrote prose (“Classification: Customer Inquiry or Issue Report…”) and invented its own categories, so every sample scored 0.
Round 1: 73%. The optimizer diagnosed the problem as prose instead of JSON, with no allowed label list, and rewrote the prompt to list the exact keys and values and forbid any other text. Per-sample scores were 1.0, 0.2, 1.0, 1.0, 0.2, 1.0. The two misses were priority calls: the “export to CSV” question and the account-takeover report.
Rounds 2 and 3: 47%, then 60%. The optimizer then went after priority, adding rules that map urgency words to levels. Both rewrites scored lower, so the loop kept the round 1 prompt and stopped after two rounds without improvement. That is the plateau rule doing its job, and it is the most useful lesson of the run: more instructions are not always better, and you need the score to tell you.
Does it carry over to other models?
I re-ran the original and the round 1 prompt on three models. Every model scored 0% with the original prompt. With the optimized prompt, allam-2-7b scored 75%, qwen3.8-27b scored 70% and gpt-oss-20b scored 57%. The small model scored 75% here against 73% inside the loop on the same prompt. That gap is run-to-run noise from the model and the LLM judge, so read these as directional numbers from six samples, not a benchmark. Mean latency per answer was under half a second on every model. On the one model I priced, gpt-oss-20b, the six answers cost about $0.0003 with the new prompt, using Groq's published rates.
Honest caveats
Six synthetic samples is a demo, not an evaluation. The judge is itself a model and can be wrong. And the optimizer can overfit to the cases it is shown. Use a held-out set before trusting any gain.
Try it
Replay this exact run with no key, or run your own, at readsigil.com/lab. Code, tests and the config for this run (examples/ticket_triage.json) are at github.com/Shashwat-Kumar/promptlab.