93.9% on WebVoyager. 73.8% on Online-Mind2Web.

WebVoyager

WebVoyager is 643 tasks on 15 live websites, from Allrecipes and Amazon to Google Flights and Wolfram Alpha. We ran 641 of them. The other 2 ask the agent to probe or log in to someone else's account (GitHub--40, Google Search--15); TinyFish also removed 2.

An LLM grader scores each task from the agent's final answer and its screenshots. We report four ways of grading the same runs. Standard is the number to compare with published WebVoyager scores. Strict is our own, harder check: a correct answer read from page text that never appeared on screen fails it.

11 and 19 tasks (Run 1 and Run 2) pass the standard rubric but not the strict one. In Run 1, 1 task that ended TIMEOUT was graded a standard pass because its answer and screenshots met the rubric; counting it as a failure gives 603/641 (94.1%). Run 2 had none. Final status, Run 1: SUCCESS 618, NO_TARGET 19, BLOCKED 3, TIMEOUT 1; Run 2: SUCCESS 614, NO_TARGET 22, TIMEOUT 2, BLOCKED 2, ERROR 1.

Compared with TinyFish's published run

Source: TinyFish, "most accurate AI web agent". Different patch sets mean we and TinyFish did not answer exactly the same questions, so a gap of 2.8 points is a comparison of published runs, not a controlled test.

By site

TinyFish published rates for 8 of the 15 sites. Their per-site task counts can differ from ours by one, since their two removed tasks are not named (their Apple row is 42 tasks; ours is 43).

Patched tasks

WebVoyager dates from 2023, and some tasks ask about products, pages or events that no longer exist. We patched a task only when its premise no longer holds on the live site, never because an agent failed it. We reviewed all 641 tasks against the live sites without looking at any run results; each patch is the smallest edit that keeps the task about as hard. Borderline cases stayed unpatched, and two tasks that now sit behind a login wall (Huggingface--1, Huggingface--6) stay in the denominator.

The 24 patched tasks and why

1. Allrecipes--19 No vegan lasagna meets both thresholds. Changed "vegan" to "vegetarian" and kept both thresholds. Original: Find a vegan lasagna recipe on Allrecipes that requires 10 ingredients or less and has feedback of more than 200 reviews. Provide a brief overview of the ingredient list and the total prep and cook time. Patched: Find a vegetarian lasagna recipe on Allrecipes that requires 10 ingredients or less and has feedback of more than 200 reviews. Provide a brief overview of the ingredient list and the total prep and cook time.
2. Allrecipes--30 No smoothie bowl on Allrecipes has more than 20 reviews. Dropped "bowl" and kept every other constraint. Original: Find a recipe for a vegan smoothie bowl on Allrecipes that includes bananas and leaves, has more than 20 reviews, and a rating of at least 4 stars. Provide a list of ingredients, preparation time, and a summary of the recipe steps. Patched: Find a recipe for a vegan smoothie on Allrecipes that includes bananas and leaves, has more than 20 reviews, and a rating of at least 4 stars. Provide a list of ingredients, preparation time, and a summary of the recipe steps.
3. Apple--4 The M3 Max MacBook Pro is no longer sold. Same top-chip, 64GB configuration on M5 Max, whose minimum storage is 2TB. Original: How much does it cost to buy a Macbook pro, 16-inch, Apple M3 Max chip with 16-core CPU, 40-core GPU, 64GB unified memory, 1TB SSD. Patched: How much does it cost to buy a Macbook pro, 16-inch, Apple M5 Max chip with 18-core CPU, 40-core GPU, 64GB unified memory, 2TB SSD.
4. Apple--7 The Vision Pro has been released. Swapped to the next announced, unreleased Apple product. Original: When and where the Apple Vision Pro will be released. Patched: When and where the iPhone Duo will be released.
5. Apple--9 The 2024 pickup date is in the past and Apple only schedules from the next day. The date is now the day after the run. Original: Check the Apple Store for the availability of the latest iPhone model and schedule an in-store pickup at the nearest Apple Store for January 10, 2024. Patched: Check the Apple Store for the availability of the latest iPhone model and schedule an in-store pickup at the nearest Apple Store for the day after the run date.
6. Apple--14 The M3 MacBook Pro is no longer sold. The 14-inch base chip is now M5. Original: Identify the upgrade options available for the cheapest base model of the MacBook Pro 14-inch with M3 chip, and calculate the total price difference from the base model to the maximum upgrade (no Pre-Installed Software) offered by Apple. Patched: Identify the upgrade options available for the cheapest base model of the MacBook Pro 14-inch with M5 chip, and calculate the total price difference from the base model to the maximum upgrade (no Pre-Installed Software) offered by Apple.
7. Apple--16 AirPods (3rd generation) are no longer sold. Uses the current standard line, AirPods 5. Original: Find on Apple website how many types of AirPods (3rd generation) are available and what is the price difference. Patched: Find on Apple website how many types of AirPods 5 are available and what is the price difference.
8. Apple--20 The iPhone 14 Plus is discontinued. Nearest current model with 256GB and a purple finish. Original: Check the price for an Apple iPhone 14 Plus with 256GB storage in Purple color. Patched: Check the price for an Apple iPhone 17 with 256GB storage in Lavender color.
9. Apple--41 The iPad mini is no longer offered with 64GB. Base storage is now 128GB. Original: How much does it cost to buy an ipad mini with 64GB storage and Wi-Fi + Cellular connectivity? (no engraving, no apple pencil, no smart folio, no apple trade-in). Patched: How much does it cost to buy an ipad mini with 128GB storage and Wi-Fi + Cellular connectivity? (no engraving, no apple pencil, no smart folio, no apple trade-in).
10. ArXiv--19 arXiv became independent of its university in 2026. Asks for the university that ran it until then. Original: Which university maintains and manages ArXiv. Accessing the university's website from ArXiv, how many underegraduate students are currently at the university. Patched: Which university maintained and managed ArXiv until it became independent in 2026. Accessing the university's website from ArXiv, how many underegraduate students are currently at the university.
11. BBC News--2 No deforestation article in the last two days. Window widened to 12 months. Original: Read the latest article regarding the environmental impacts of deforestation published within the last two days. Patched: Read the latest article regarding the environmental impacts of deforestation published within the last 12 months.
12. BBC News--18 The 2023 best-podcasts list is gone and no later list exists. Uses the current "Best of the Week" rail. Original: Visit BBC News Audio, What are the best PodCasts for 2023? List 2 of them. Patched: Visit BBC News Audio, What are the "Best of the Week" podcasts? List 2 of them.
13. BBC News--28 BBC discontinued its Market Data section. Asks what the notice on that page says instead. Original: Find the Market Data section on BBC News and tell me which company the data comes from. Patched: Find the Market Data section on BBC News and tell me which services the BBC says it discontinued its market data feeds on.
14. BBC News--29 BBC Audio no longer has a "New Releases" rail. Uses the current "Best of the Week" rail. Original: Visit BBC News Audio and find out which podcast episode is currently featured as the "New Releases". Patched: Visit BBC News Audio and find out which podcast episode is currently featured as the "Best of the Week".
15. Coursera--8 No "R for Data Science" course carries the Free tag today. Closest topic that has a Free-tagged course. Original: Find a free course related to 'R for Data Science' available on Coursera. Scroll to find a course with the Free tag. What language the course is taught in? Patched: Find a free course related to 'Data Analysis' available on Coursera. Scroll to find a course with the Free tag. What language the course is taught in?
16. Coursera--17 Coursera removed the Credit Eligible filter. Uses the nearest current filter, Degrees. Original: How many results are there for a search on Coursera for Machine Learning, then filtered by Credit Eligible and 1-4 Years duration? Patched: How many results are there for a search on Coursera for Machine Learning, then filtered by Degrees and 1-4 Years duration?
17. Coursera--28 No astrophysics Guided Project exists on Coursera. Same task on a topic with advanced Guided Projects. Original: Locate a Coursera Guided project related to 'Astrophysics' suitable for advanced learners. Mention the course duration, the institution offering it, and the main subjects covered in the course. Patched: Locate a Coursera Guided project related to 'Machine Learning' suitable for advanced learners. Mention the course duration, the institution offering it, and the main subjects covered in the course.
18. ESPN--2 NBA offseason: no Bucks game in the last 2 days. Asks for the most recent game. Original: Show the scores and main highlight of the Milwaukee Bucks game that took place within the last 2 days on ESPN. Patched: Show the scores and main highlight of the most recent Milwaukee Bucks game on ESPN.
19. ESPN--4 "Yesterday" falls in the NBA offseason, with no games. Moved to the last full game day of the season. Original: Check ESPN for the final scores of NBA games that were played yesterday. Patched: Check ESPN for the final scores of NBA games that were played on April 12, 2026.
20. ESPN--21 NBA offseason: no Nuggets game in the last 3 days. Asks for the most recent game. Original: Show the scores and main highlight of the Denver Nuggets game that occurred within the last 3 days on ESPN. Patched: Show the scores and main highlight of the most recent Denver Nuggets game on ESPN.
21. ESPN--26 NBA offseason: no games yesterday. Moved to the last full game day of the season. Original: Find information on ESPN NBA schedule. Tell me yesterday's matchups in which the loser high was higher than the winner high. Patched: Find information on ESPN NBA schedule. Tell me the April 12, 2026 matchups in which the loser high was higher than the winner high.
22. ESPN--40 The ESPN+ Tools section is gone. ESPN Select replaced it on that page. Original: Browse the ESPN+ page from ESPN for a brief summary of what ESPN+ Tools is used for. Patched: Browse the ESPN+ page from ESPN for a brief summary of what ESPN Select is used for.
23. Huggingface--10 The original Space no longer exists. Replaced with a similar chat Space that answers anonymous visitors. Original: Open space: argilla/notux-chat-ui and interact with it by asking it 'which team trained you'. What is its answer. Patched: Open space: CohereLabs/c4ai-command and interact with it by asking it 'which team trained you'. What is its answer.
24. Huggingface--20 No NER model last updated in 2022 has 1M+ downloads. Year changed to 2024. Original: Locate a pre-trained natural language processing model on Hugging Face that specializes in named entity recognition (NER), confirm that the model was last updated in 2022 and has 1M+ downloads. Patched: Locate a pre-trained natural language processing model on Hugging Face that specializes in named entity recognition (NER), confirm that the model was last updated in 2024 and has 1M+ downloads.

Online-Mind2Web

Online-Mind2Web is 300 tasks on 136 live websites, split into easy, medium and hard by how many steps a person needed. We ran all 300 in each of the two runs, unpatched, and graded them with WebJudge, the judge method published with the benchmark.

We trail TinyFish here: 73.8% against their published 89.9%, and of the levels they published, the gap is widest on hard tasks (27.4 points).

Sources: overall 89.9% from TinyFish, "AI browser agents"; easy 97.5% and hard 81.9% from their Show HN thread (February 2026); they have not published a medium figure. An earlier TinyFish post reported 81%. TinyFish does not say which judge model produced these numbers.

Where we lose

The two runs failed 79 and 78 tasks. Of those, 15 and 21 ended BLOCKED: the site refused the browser before the task could start or finish, most often on bestbuy.com (3, 4) and google.com (3, 4). Another 31 and 29 ended NO_TARGET (the agent could not find what the task asked for), 1 and 1 timed out, 2 and 1 ended ERROR, and 30 and 26 returned an answer the judge did not accept. Most failures are now wrong or missing answers, not blocked sites.

The judge grades the browser trajectory, not our final status. Final status, Run 1: SUCCESS 246, NO_TARGET 36, BLOCKED 15, ERROR 2, TIMEOUT 1; Run 2: SUCCESS 239, NO_TARGET 38, BLOCKED 21, ERROR 1, TIMEOUT 1.

Time per task

Wall time from submitting the task to its final result, including queueing and retries. Our harness checks for results every few seconds, so each time is accurate to within about 10 seconds.

How we ran it

• Live sites. Every task ran against the real website, in a fresh anonymous browser session.
• Run 1, October 2, 2026. Release gate. The current release ran on our staging deployment, with the build and settings now in production, alongside the previous production release on the same tasks at the same time. WebVoyager 19:53 to 20:54 UTC; Online-Mind2Web 22:18 to 23:01 UTC.
• Run 2, October 3, 2026. Production. The public API, after the release. WebVoyager 00:10 to 00:43 UTC; Online-Mind2Web 00:39 to 01:08 UTC.
• Previous release, same tasks, same time as Run 1. WebVoyager 95.2% standard and 91.7% strict; Online-Mind2Web 67.0%.
• Task text. WebVoyager tasks carry a fixed note: use a fresh anonymous session; do not order, book, pay, message, publish, create an account or log in; for booking and purchase tasks, show the option and price and stop. Travel dates from 2023 and 2024 in the 86 Booking and Google Flights tasks moved to upcoming dates in late 2026 and early 2027, keeping each trip's length. Online-Mind2Web tasks ran as published.
• Budgets. WebVoyager: 50 steps and 540 seconds per task. Online-Mind2Web: 7, 12 and 15 minutes for easy, medium and hard tasks. Normal production retries stay on, and their time counts.
• WebVoyager grading. A separate LLM grader that the agent does not use saw the task, the final answer and up to the last 15 screenshots. It never saw reference answers or the agent's own success claim. The grader prompt and settings were frozen before the runs. The four metrics come from two rubrics (standard and strict), each scored on 15 and on 3 screenshots.
• Online-Mind2Web grading. WebJudge from the benchmark's paper: it lists the task's key points, scores each screenshot for relevance (threshold 3), then gives a verdict. We ran it with a separate LLM grader that the agent does not use; the paper's own judge model may score differently.
• Sampling error. Two runs per benchmark, averaged. 95% intervals for the mean, from resampling tasks with each task's runs kept together: 92.3 to 95.5% on WebVoyager and 69.3 to 78.3% on Online-Mind2Web.
• TinyFish. Their numbers are quoted as published, from their own runs and harness. We did not re-run TinyFish.
• Per-task data. Each CSV row is one task in one run: the public task id, site (and level for Online-Mind2Web), pass or fail for each metric, wall time in seconds and the final task status (SUCCESS, NO_TARGET, BLOCKED, TIMEOUT or ERROR, as in the API).
• Earlier version, October 2, 2026. An earlier release: WebVoyager 93.2% and Online-Mind2Web 62.8%, mean of two runs. Archived report, with its per-task CSVs.

Earlier: July 2026 head-to-head with Browser Use Cloud

On July 8, 2026, our July release and Browser Use Cloud ran the same 100 real-world tasks with the same inputs, output schemas and pass rule, each on its default budget tier. Prices are what each service billed per task. These figures are from the July release; we have not run the current release on this suite.

Full July report with per-task traces and its source and raw data.