| Measure | CPython | Monty |
|---|---|---|
| First callfresh process, includes runtime load | 18.85 ms | 0.575 ms |
| Next callsame process, new interpreter state | 4.412 ms | 0.049 ms |
| Warm print(1)median per call in a long-lived session | 4.424 ms | 0.018 ms |
Benches
Latest benchmark snapshot
Static aggregate generated from repository result artifacts. Use the linked files for raw measurements and full eval traces.
Python startup history
CPython per run
| Run | First call | Warm print(1) |
|---|---|---|
| 2026-10-07import http.client: 461.12 ms | 18.85 ms | 4.424 ms |
| 2026-10-06import timing not recorded | 21.896 ms | 6.312 ms |
| Gap | Calls now | Recorded |
|---|---|---|
expected 'done'parse error | 23 | 38 |
unterminated command substitutionparse error | 18 | 23 |
busyboxmissing command | 10 | 0 |
tar -unknown option | 9 | 1 |
unexpected end of input in [[ ]]parse error | 8 | 7 |
sudomissing command | 7 | 7 |
python3missing command | 6 | 6 |
grep Iunknown option | 6 | 0 |
gawkmissing command | 5 | 1 |
perlmissing command | 4 | 3 |
(no detail)parse error | 3 | 11 |
unexpected tokenparse error | 3 | 4 |
join ounknown option | 3 | 0 |
nodemissing command | 2 | 2 |
find -lsunknown option | 2 | 2 |
pythonmissing command | 2 | 1 |
unterminated single quoteparse error | 2 | 1 |
mawkmissing command | 2 | 0 |
gzip cunknown option | 2 | 0 |
xargs runknown option | 2 | 0 |
Gap history
Calls hitting a gap
| Replay | Now | Recorded |
|---|---|---|
| 2026-10-075,206 calls, bashkit 0.18.2 | 125 | 319 |
| Spec file (most misses) | bashkit | bash |
|---|---|---|
background74.1% of 27 cases | 20 | 27 |
bugs82.8% of 29 cases | 24 | 29 |
builtin-trap87.9% of 33 cases | 29 | 33 |
brace-expansion94.5% of 55 cases | 52 | 55 |
builtin-meta83.3% of 18 cases | 15 | 18 |
builtin-trap-bash87% of 23 cases | 20 | 23 |
sh-options92.3% of 39 cases | 36 | 39 |
assign-extended94.3% of 39 cases | 33 | 35 |
builtin-read96.9% of 64 cases | 62 | 64 |
builtin-trap-err90.9% of 23 cases | 20 | 22 |
builtin-umask91.7% of 24 cases | 22 | 24 |
fatal-errors60% of 5 cases | 3 | 5 |
Oils spec history
Cases passed vs real bash
| Run | Rate | bashkit / bash |
|---|---|---|
| 2026-10-09Oils 57d3f0d088c3, bashkit 298bf8f9cfd1 | 97.3% | 2,618 / 2,692 |
| 2026-10-08Oils 57d3f0d088c3, bashkit 74d816bc7893 | 95.8% | 2,578 / 2,692 |
| 2026-10-08Oils 57d3f0d088c3, bashkit 38a65ba78b0b | 92.8% | 2,500 / 2,693 |
| 2026-10-08Oils 57d3f0d088c3, bashkit fa242de13904 | 89% | 2,395 / 2,692 |
| 2026-10-08Oils 57d3f0d088c3, bashkit 26a3acc75177 | 86% | 2,316 / 2,693 |
| 2026-10-08Oils 57d3f0d088c3, bashkit 017d9bfb4f3d | 82.1% | 2,211 / 2,692 |
| 2026-10-08Oils 57d3f0d088c3, bashkit 5f04b4810ec2 | 78.8% | 2,120 / 2,692 |
| 2026-10-08Oils 57d3f0d088c3, bashkit bae540476f21 | 77.3% | 2,080 / 2,692 |
| 2026-10-08Oils 57d3f0d088c3, bashkit f86bd7c81ae3 | 74.2% | 1,997 / 2,693 |
| Category | Cases | Last run |
|---|---|---|
| startupSmall commands where interpreter startup dominates runtime. | 4 | 0.016 msbash median: 4.769 ms |
| arithmeticInteger math, substitutions, and expression-heavy shell snippets. | 6 | 0.018 msbash median: 4.313 ms |
| stringsString expansion, pattern handling, and text manipulation. | 8 | 0.018 msbash median: 4.378 ms |
| variablesVariable assignment, lookup, expansion, and environment handling. | 8 | 0.019 msbash median: 4.096 ms |
| arraysIndexed array reads, writes, expansion, and iteration. | 6 | 0.019 msbash median: 4.064 ms |
| pipesPipeline construction, streaming, and command chaining. | 6 | 0.021 msbash median: 5.898 ms |
| subshellCommand substitution and nested shell execution paths. | 6 | 0.024 msbash median: 5.484 ms |
| controlConditionals, loops, case statements, and branching scripts. | 9 | 0.025 msbash median: 3.812 ms |
| Category | Passed | Pass rate |
|---|---|---|
| system_info | 1/2 | 50%tasks passed |
| file_operations | 3/4 | 66.7%tasks passed |
| scripting | 5/7 | 68.6%tasks passed |
| json_processing | 8/8 | 100%tasks passed |
| data_transformation | 6/6 | 100%tasks passed |
| complex_tasks | 6/6 | 100%tasks passed |
| text_processing | 6/6 | 100%tasks passed |
| pipelines | 5/5 | 100%tasks passed |
| Run | Score | Tools |
|---|---|---|
| gpt-5.3-codex2026-05-26 | 93% | 86.8% |
| gpt-5.52026-05-26 | 92.7% | 91.5% |
| claude-opus-4-72026-05-26 | 97.8% | 90.3% |
| claude-sonnet-4-62026-05-26 | 94% | 91% |
| claude-haiku-4-5-202510012026-05-26 | 98.4% | 92.3% |
| claude-sonnet-4-62026-02-28 | 92.5% | 85.1% |
Indexes