Time to a classification
Median request latency · lower is faster
Connection + TLS + processing + response parsing.
Warmups and failed responses stay out of the latency statistics.
OPEN CLASSIFIER WORKBENCH
Classification,
measured.
Compare the time to an answer—and whether it’s right.
Repeatable cases, explicit models, every attempt recorded.
THE EXPERIMENTS
Results on the bench
Loading published experiments…
Measured calls—
Models—
Correct / valid—
Run date · UTC—
| Model / route | Median | Mean | p95 | Correct | Errors |
|---|
A sample, not a leaderboard. These runs use fixed dataset samples from one machine; accuracy depends on the task and labels, while latency also depends on network conditions and routing.
LABELS CHECKED AGAINST ANSWERS
How accurate is the classification?
Accuracy measures agreement with the dataset’s expected labels; the model’s confidence is a separate number.
Accuracy—
Macro F1—
Correct / valid—
Wrong labels—
Per-class performance
Recall is the fraction of a class correctly identified; precision is the fraction of its predictions that are correct.
| Class | Correct / valid | Precision | Recall | F1 |
|---|
Inspect the mistakes
Every wrong classification from the selected run is listed below.
| Message | Expected | Predicted |
|---|
Warmups and invalid responses are excluded; errors and skipped calls remain visible above; macro F1 weights every declared class equally, with zero scores for undefined class ratios.
HOW IT WORKS
A fairer starting point.
Keep the experiment visible, so a result can be questioned and repeated.
- 01
One dataset, every model
Each model receives the same text, labels, and classification instructions; expected answers stay out of the request.
- 02
Rotate the order
Shuffle the cases with a recorded seed and counterbalance model order, with warmups reported separately.
- 03
Keep the whole run
Save attempts, errors, returned model names, latency, accuracy, and the exact experiment settings.
YOUR NEXT EXPERIMENT
Change a model.
Keep the test.
Choose an experiment JSON file, point it at your dataset, and run locally with Python.
See the configuration guide# preview the models and request budget
python3 -m jev_bench --dry-run
# run the selected experiment
python3 -m jev_bench \
--config experiments/classifier-models.json