Benchmark

10 of 10 tasks solved

Our own check of Omi CLI 0.2.0 on 10 small, realistic repos in JavaScript and Python, run on November 21, 2026. Each task is judged by hidden tests the agent never sees. This measures Omi CLI alone; it is not a comparison with other tools, and 10 tasks is a small sample.

Solved

10 / 10

Hidden tests pass, and nothing the task forbade was changed.

Credits, all tasks

5,27,56,311

About 52,75,631 a task on average.

Time, all tasks

27 min

One attempt each, start to finish, tests included.

Watch any run

The real terminal output of each task, replayed faster. Nothing is edited except colours.

real run
~/js-pagination-bug $ omi -p "Users say the first page of results is missing and the last page never shows. Fix the pagination helpers and make sure the tests pass."
  (The chosen model is busy at Google right now; a backup model answered. You're charged what the backup really cost.)
  ↳ 3 files in .
  ↳ read package.json (8 lines)
  $ npm test  # Run existing tests to see current failures
  ✗ exit 1
  ↳ read src/paginate.js (10 lines)
  ↳ read test/paginate.test.js (5 lines)
  Plan
  ▸ Fix paginate and pageCount functions in src/paginate.js
  ○ Add test cases in test/paginate.test.js to verify pagination logic and pageCount
  ○ Run tests to confirm they pass
  edit src/paginate.js  +2 -2
    1   export function paginate(items, page, size) {
    2     // page is 1-based
    3 -   const start = page * size;
    3 +   const start = (page - 1) * size;
    4     return items.slice(start, start + size);
    5   }
    6
    7   export function pageCount(total, size) {
    8 -   return Math.floor(total / size);
    8 +   return Math.ceil(total / size);
    9   }
   10
  $ npm test  # Verify that the first page test passes
  ✓ exit 0
  ↳ read test/paginate.test.js (5 lines)
  edit test/paginate.test.js  +9 -1
    1   import { test } from "node:test";
    2   import assert from "node:assert/strict";
    3 - import { paginate } from "../src/paginate.js";
    3 + import { paginate, pageCount } from "../src/paginate.js";
    4 +
    5   test("first page", () => assert.deepEqual(paginate([1,2,3,4,5], 1, 2), [1,2]));
    6 + test("second page", () => assert.deepEqual(paginate([1,2,3,4,5], 2, 2), [3,4]));
    7 + test("last page", () => assert.deepEqual(paginate([1,2,3,4,5], 3, 2), [5]));
    8 +
    9 + test("pageCount with remainder", () => assert.equal(pageCount(5, 2), 3));
   10 + test("pageCount division exact", () => assert.equal(pageCount(6, 2), 3));
   11 + test("pageCount empty", () => assert.equal(pageCount(0, 2), 0));
   12 +
   13
  $ npm test  # Verify all new tests pass successfully
  ✓ exit 0
  Plan
  ✓ Fix paginate and pageCount functions in src/paginate.js
  ✓ Add test cases in test/paginate.test.js to verify pagination logic and pageCount
  ✓ Run tests to confirm they pass
◆ I have successfully resolved the issues with the pagination functions and verified them by running the tests.
 
  What changed:
  • src/paginate.js:
    • Fixed paginate start index to be (page - 1) * size instead of page * size (which was causing the first page of size N to start at index N instead of index 0).
    • Fixed pageCount to use Math.ceil(total / size) instead of Math.floor(total / size) to properly account for remainder items and not omit the last page.
  • test/paginate.test.js:
    • Added test cases verifying the correct behavior of the first, second, and last pages.
    • Added test cases for pageCount with remainders, exact division, and empty items.
 
  All tests are now passing!
  59,79,638 credits

Solved · hidden tests pass · took 2.4 min live, replayed faster · benchmark of Nov 21, 2026

Time per task

  • js-pagination-bug145s
  • js-slugify-feature159s
  • js-rename-across-files184s
  • js-async-race166s
  • py-duration-parser97s
  • py-csv-export93s
  • js-three-failures136s
  • py-cli-flag179s
  • js-input-validation292s
  • js-question-no-edit139s

Every task

TaskWhat the agent was askedResultTimeCredits
js-pagination-bugUsers say the first page of results is missing and the last page never shows. Fix the pagination helpers and make sure the tests pass.Solved145s59,79,638
js-slugify-featureAdd and export slugify(text) in src/strings.js: lowercase, strip accents (é → e), turn any run of non-alphanumeric characters into a single hyphen, and trim hyphens from both ends. Add tests.Solved159s78,20,464
js-rename-across-filesRename getUsr to getUser everywhere in the project (definition and every use). Keep everything working.Solved184s41,14,545
js-async-racesaveAll sometimes returns 0 even though items get saved later. Find the bug and fix it so saveAll returns only after every item is saved, keeping the original order of items in saved.Solved166s57,79,283
py-duration-parserparse_duration gives wrong answers for things like '1h30m' and '90s'. Fix it and add tests. Run the tests with python3 -m unittest.Solved97s37,94,938
py-csv-exportAdd to_csv(orders) in orders.py that returns CSV text with a header row 'id,customer,total' and the total as dollars with 2 decimals (e.g. 1250 cents -> 12.50). Customer names may contain commas or quotes, so quote properly (use the csv module). Lines end with \n. Add tests and run them with python3 -m unittest.Solved93s48,04,238
js-three-failuresThe test suite is failing. Fix the code (not the tests) until npm test passes.Solved136s67,80,798
py-cli-flagAdd a --limit N option to todo.py that prints at most N tasks (after the --done filter). N must be a positive integer; argparse should reject 0 or negative values with an error. Show me it works by running it.Solved179s42,38,730
js-input-validationHarden validateSignup: email must look like name@domain.tld (no spaces) and the error is 'invalid email'; password must be at least 10 characters with a letter and a digit, error 'weak password'. Keep the existing 'required' errors (and don't add the other error for a missing field). Add tests.Solved292s77,25,866
js-question-no-editQuestion only, don't change any files: what request timeout does makeClient use when the TIMEOUT_MS environment variable is not set, and which file and line decides it? Reply with the number in milliseconds.Solved139s17,17,811

Method

  • • 10 small repos written for this check: bug fixes, new features, a rename across files, an async race, a multi-failure test suite, a CLI flag, input validation, and one question that must be answered without changing any file. 6 in JavaScript (Node test runner), 4 in Python (unittest), each a fresh git repo.
  • • Each task ran once, with no retries and no human help: omi -p in auto-edit mode, with only the test commands allowed to run without approval (npm test, node --test, python3 -m …, python3 todo.py).
  • • After the agent finished, hidden tests it had never seen were copied in and run. A task counts as solved only if they pass. Tests the task said not to change were put back first; for the question, the answer had to contain the right number and no file could change.
  • • Model: the default (standard). During this run Google rate-limited that model, so every turn was answered by the backup, Gemini 3.5 Flash. Credits are what the run really cost; time includes waiting on the model and running the tests.
  • • The tasks are small and the sample is 10, so this shows Omi CLI handles everyday tasks end to end; it does not show how it does on large codebases or compare it with other tools.

We will rerun this check on every release and add harder tasks. To see how Omi CLI does on your own code, install it: every account gets free credits each month.