AI Agents Complete Less Than 4% of Real SaaS Tasks

What did SaaS-Bench actually test?

SaaS-Bench is a new benchmark for computer-use agents working inside real software-as-a-service environments. The paper, submitted to arXiv on May 15, 2026, includes 106 tasks across 23 deployable SaaS systems and six professional domains.

Many AI benchmarks test short prompts, isolated browser tasks or simplified user interfaces. Finance work is messier. A close checklist, vendor cleanup, reconciliation, forecast update or board report can require several systems, changing data, partial evidence and exception handling.

The benchmark's authors describe the tasks as long horizon workflows with weighted checkpoints for strict task completion and partial progress. In plain English: the agent does not get full credit for looking good at step one if it fails before the work is done.

What does the less-than-4% result mean?

The headline result is blunt: representative LLM-based agents struggled, and the strongest model completed fewer than 4% of SaaS-Bench tasks end to end.

That does not mean every AI agent is useless. It means buyers should separate step accuracy from completed work. An agent may classify a document correctly, draft a note, click the right tab or summarize a table. The harder question is whether it can keep track of the job across a messy sequence and finish without human rescue.

That distinction is where finance leaders can get misled. A vendor demo usually shows a single task, a clean account, a familiar workflow and a scripted path. Real finance work includes duplicate vendors, missing support, permissions gaps, stale data, edge cases and reviewers who need evidence.

Why do AI agents fail at professional workflows?

The SaaS-Bench paper points to limitations in planning, state tracking, cross-application context maintenance and error recovery. Those are exactly the capabilities professional workflows need.

Planning means the agent understands the sequence, not just the next click. State tracking means it remembers what has changed after each step. Cross-application context means it can carry the right facts from one system into another without mixing accounts, dates or entities. Error recovery means it can notice when the screen, data or permission path does not match the expected route.

Finance teams should read that list like an implementation checklist. Month-end close, AP exception handling, account reconciliation, cash forecasting and financial reporting all depend on those capabilities. If the agent cannot recover from a missing invoice, a changed approval status or a data mismatch, it is not automating the workflow. It is creating a new review queue.

Which finance workflows are most exposed?

The most exposed workflows are the ones vendors like to demo as end-to-end automation: close management, reconciliations, AP exception cleanup, multi-system reporting and account maintenance.

These workflows have two traits. First, they span systems. A task may start in email, move through a document store, touch an ERP, require a bank or payment system and finish in a checklist or report. Second, they require judgment when something does not line up. That is where a polished demo can overstate readiness.

Think about a NetSuite close workflow that starts with a Slack message, uses a Google Drive support folder, checks a Stripe payout report and ends with a controller signoff. Or a QuickBooks Online cleanup where the agent must match a Bill.com payment, a bank feed transaction and a vendor invoice with a slightly different name. Or a Sage Intacct reporting task where the model has to pull the right entity, class and month before drafting variance commentary. None of those workflows are hard because the next button is hidden. They are hard because the state keeps changing.

Short, bounded workflows are better first pilots. Receipt categorization, invoice field extraction, single-system variance drafting, duplicate-payment flags and document-request tracking are easier to measure. The buyer can still use AI. The difference is that the pilot scope matches the current capability instead of the sales deck.

How should finance teams change AI agent pilots?

Finance teams should measure end-to-end task completion, review burden and exception handling before expanding an AI agent contract.

Start with a real baseline. How long does the workflow take today? How many errors, exceptions and reviewer interventions happen in a normal month? Then run the agent on actual work, not a sample workflow designed by the vendor. Track what completed without rescue, what needed review, what failed silently and what created extra cleanup.

Do not let the pilot stop at "the model got the field right." For finance, the useful question is whether the controller can close the file faster with the same or better evidence. If the agent saves 20 minutes of preparer time but adds 25 minutes of reviewer cleanup, the pilot did not improve capacity. If it completes 80% of simple invoices but fails every exception, the team still owns the hard work.

The pilot scorecard should include:

• End-to-end task completion rate, not just step accuracy.
• Human review minutes per completed task.
• Exception rate and failure reasons.
• Evidence retained for audit, tax, compliance or management review.
• Whether the workflow improved after 30, 60 and 90 days.

Deloitte's Finance Trends 2026 context shows finance departments are already using AI at scale. That makes measurement more important, not less. A tool can be widely deployed and still weak at the specific workflow your team wants to automate.

What should you ask before buying an AI agent?

Ask three questions before buying or expanding an AI agent: Can it complete our actual workflow, can it handle exceptions, and can we verify what it did?

Use direct language with vendors:

• Show the task completion rate on our workflow, with our data, across a full month.
• Show what happens when the document is missing, the vendor name changes or the approval state is wrong.
• Show the evidence trail a reviewer, auditor or controller can inspect later.

Also ask for the rollback plan. If the agent misclassifies a payment, changes a status, loses context between applications or creates a duplicate record, who notices and how fast can the team unwind it? In finance operations, recovery is part of the product. A tool that works only when nothing breaks is not ready for critical workflow ownership.

If a vendor can only answer with a demo, keep the pilot small. If the agent performs well on bounded tasks, expand carefully. If it cannot complete real work without senior staff cleaning up after it, the budget belongs somewhere else.