Jevaluate System

Jevaluate rates whether open-source projects use TypeSafe's Jev model well. The ratings are public, so every LLM rater has to read the rubric the same way. Code checks every rule a script can check. The rater reports only what it observes, and code derives the rest. An eval with blinded labels tests the rubric before any version is frozen.

Five design rules

• Code enforces, text explainsEvery rubric rule has a row in enforcement.md naming the function that refuses a wrong answer, or why the rule stays judgment. A lint fails on a rule with no row or a function that doesn't exist.
• Ask for what is observedThe rater records type, the call line, each decision with its stakes as read, and the verdict. Code derives kind and every n.a. People and models both miss exception lists. Code doesn't.
• Evidence over claims"Calls Jev: yes" must cite the file and line of the call, and check confirms it is a call. A README, a docs page or a project name never counts.
• Stakes change the rowsRules whose test depends on stakes (confidence gating, option order, templated data, untrusted text) get one row per stakes level. The rater sees only the rows its project's decisions need.
• Refuse, don't warnScripts stop the workflow: a rating over 200k tokens without approval, a missing step log, routing changed after the facts were served, an eval run on uncommitted rubric changes.

Components and workflows

The harness runs three workflows. A rubric change passes the eval before a git tag freezes it. A rating round sends each project to one rater per model and takes their verdict-changing disagreements to the owner. Publishing exports reviewed ratings in one pull request.

Routing and code enforcement

Routing decides which facts apply and how a failure counts, so code checks it hardest. Each script window on the chart refuses a wrong answer or fills in a value the rater never enters. The rubric doesn't rate guides or jev-replacements yet: they get verdict 1, not yet rated, and their facts, scores and stakes are n.a.

Scroll sideways to see the whole chart.

The eval that gates every rubric version

A rubric change runs the first three instruments in order, each passing before the next. The fourth checks a second rater. Pass bars are set before the run. Each rule is scored on its own, never just an average of all scores.

1. 1

   Materials check

   Is what the grader reads consistent?

   1. lint_materials.pyAllowed values agree across files. No eval-case names in rubric examples. Every rule has an enforcement row.
   2. Fresh reviewerBriefed as a grader, reads the materials cold.
   3. Grader viewThe owner reviews exactly what a grader receives, step by step, after any change that moves instructions into code.
2. 2

   Comprehension quiz

   Does a grader understand the rules on invented projects?

   1. blind writerInvented projects11 from a model that never saw the rubric, and 5 lookalikes a Claude session wrote for a rubric change. Some mislead on the surface. Any overlap with rubric wording is rewritten.
   2. Owner's key, 16 projectsAnswered first, observable fields only. A script derives the rest and refuses contradictory answers.
   3. 8 tuning, 8 held outThree drawn at random (seed 29) after the key was final, plus the five lookalikes. Two of those no longer count as held out on stakes: their key changed after a run.

   FULL RUN, 2026-09-30, ONE SCALE

   Negative control: vocabulary only

   8 of 11 scenarios, 73%

   Real instructions, 3 runs each

   33 of 33 answers

   050%100%

   80% give-away line: the negative control must score under it

   Negative control: A grader given only the allowed values must score under 80%, or the scenarios give their answers away. Its three misses are the three rules the vocabulary can't supply.

   Pass bar: Every tuning case right in 3 of 3 runs, held-out in 2 of 3.

   Result, 2026-09-30: Every rule tag passed, held-out 3 of 3, and no wording fix was needed.

3. 3

   Judgment eval

   Does a grader route real projects the way the owner does?

   1. Golden dataset19 real open-source projects, keyed. 6 of them cover the gaps: one claims Jev and never calls it, one is an honest stub named after Jev, two are high stakes, two very high.
   2. pick_files.pyFixed file ruleRoot README, every file with a hosted call, then Jev or decision hits up to 4 files. Code before docs and tests, data last. 3,000 characters each.
   3. Provable labelsEvery gold answer must be backed by the packet the grader sees. 13 checked: 12 provable, 1 weak, 0 changed.
   4. blind labelersLabel provenanceNew cases: labels sealed until the owner answers from evidence cards. check_labels.py refuses a label that cites a doc as a call; build_cards.py refuses any verdict word on a card.

   Graders under test: Claude Sonnet and GPT-6 Sol in Codex, on identical inputs.

   The call detector is tested too: every miss or false call found on real repos is now a regression test.

4. 4

   Rater agreement

   Do two raters agree, and which is accurate?

   1. Tuning sample named firstProjects whose disagreements fed wording are reported apart, and second.
   2. rater_agreement.pyRaw agreement and Cohen's kappa per field, held-out projects first.
   3. blind ownerAnswer keyA seeded draw of agreed and split facts. The owner answers before seeing either rater.
   4. Reliability and accuracyAgreement and kappa measure inter-rater reliability. Accuracy needs the key: two raters can agree and both be wrong. Wording is tuned for both, never for agreement over accuracy.
5. FOR STAGES 2-3

   Reproducibility

   Can a score be traced to the exact rubric and key?

   1. run_eval.pyPre-flight refuses uncommitted rubric changes or a branch behind main.
   2. StampEvery quiz and eval run records the commit. Eval runs also fingerprint the answer key.
   3. score.pyJudgment eval only: refuses to score if the key changed after the run.
6. FOR STAGES 2-3

   Report

   What does the rubric get wrong?

   1. Per ruleEach rule tag scored alone. An average hides the rule that fails.
   2. One fix per missed ruleWording fixes come only from tuning misses and never name a held-out case.
   3. StopThe owner sees each fix as a before-and-after. The eval never edits the rubric.

Who is blind to what

Each blind guards against one risk: leakage, contamination, anchoring or skimming ahead. Two roles can't be blind, so rules fence in what each can touch.