Measured, not demonstrated
A demo is a claim about the good case. We are interested in the distribution, particularly its tail.
Lab
Where ideas become technology.
The research position of the LLP, stated with a status against each item. Including the things we looked at and decided against, which is the half of a radar that is usually missing.
How to read this
- Adopted
- In use in something we have shipped.
- Trialling
- Running now, with a date on the decision.
- Assessing
- Read and reasoned about. Not yet built with.
- Held
- Looked at and deliberately not taken up.
Adopted
04 entries-
Schema-validated model output
Every model call is bounded by a schema and validated before storage. Malformed output is retried or discarded. This is the single change that moved model work from demo to production.
-
Native Telugu analysis
Script, idiom and political register handled directly rather than through an English translation layer. Translation-first pipelines lose exactly the signal the platform exists to measure.
-
Vector retrieval over owned content
Separate accounts of one event group by meaning, which keyword matching does not achieve at any threshold.
-
Sources as database rows
Adding a publication is an insert, not a deploy. This is why coverage expansion is measured in days.
Trialling
03 entries-
Per-call cost attribution
Spend traced to the feature that caused it rather than to the month it landed in. Useful the first time a model price changes.
-
Evaluation sets that predate the prompt
Held-out data fixed before prompt work begins, so improvement cannot be measured against a target that moved.
-
Tool-using loops with hard budgets
Explicit spend ceilings and termination conditions. The open question is what the loop should do when it hits one.
Assessing
03 entries-
Smaller models for classification
Most of our classification work does not need a frontier model. The question is where the accuracy cliff actually sits for Telugu.
-
Structured extraction without a model
A meaningful fraction of extraction is regular enough for a parser. Every call avoided is a call that cannot fail or cost anything.
-
Multi-region data residency as configuration
Technically settled; the open part is the commercial and legal shape it has to take per market.
Held
03 entries-
Fine-tuning as a default
Held. For our current problems, retrieval plus a schema beats a fine-tune on both cost and on the ability to explain a wrong answer.
-
Autonomous multi-agent pipelines
Held. Attractive in a demonstration, and the failure mode is a plausible wrong answer produced expensively with no single place to look.
-
A shared database as an integration point
Held. It is the fastest way to integrate two services and the most expensive way to separate them later.
How the lab works
Kill criteria first
Every research thread is written down with the conditions under which we stop. Without them a thread does not end, it just gets quieter.
Publish the rejections
What we chose not to build says more about the judgement than what we did.