Judge a marketing AI agent on four layers: data access, output quality, action execution and business effect. One demo proves at most one, and none of the sixteen benchmarks below covers all four. Run all four on your own data with the 12-question scorecard below.
Each of the four layers needs its own test
An agent can write a confident, well-structured answer about last month's cost per acquisition (CPA) that is wrong because it read the wrong column. A rubric or a judge model grades the writing. It catches the wrong column only when it can see the source data or the right answer, so the output test needs one of the two.
Execution tests start where output tests stop: a perfect budget recommendation can land on the wrong campaign, and only a read of the system state afterwards shows it. The agent's report of what it did is a claim to verify by reading the system.
The business effect needs a third test. A month of correct pauses and budget moves can leave CPA unchanged when every change sat on a campaign that carries 2% of spend, to take one example. A controlled read of the business number, such as a holdout, shows it. That read takes weeks.
What each test proves, and what it misses
| Test | Question it answers | Evidence it needs | What it cannot see |
|---|---|---|---|
| Data access | What can it read and change, and is the data fresh? | A read-back of the connection: sources, permissions and the last load time | Whether the answer built on that data is right |
| Output quality | Is the answer right, and is it readable? | A known answer, or a rubric and a judge | Whether the agent can act, or whether acting helps |
| Action execution | Did it change the right thing, and nothing else? | A check of the system state after the run | Whether the change was worth making |
| Business effect | Did the number move, against a control? | A holdout and weeks of data | Which step of the agent's work caused it |
Marketing questions come in eight kinds, and the risk rises down the list
Before you pick a benchmark, write down what you will ask the agent. Marketing questions fall into eight kinds. Four of the kinds only read data, and the other four draft, change, watch or confirm.
The eight kinds of question, the question each one answers, and an example
| Kind | The question it answers | Example |
|---|---|---|
| Describe | What happened? | Which campaigns spent the most last week? |
| Explain | Why did it happen? | Why did the cost per lead rise in September? |
| Forecast | What will happen? | Where will CPA land next month at this spend? |
| Choose | Which option? | Which three ad sets should we pause first? |
| Prepare | Draft the thing. | Write the Q4 budget proposal with the numbers filled in. |
| Execute | Change the system. | Move 20% of the budget from the losing ad set to the winning one. |
| Monitor | Keep checking. | Tell me when any campaign's CPA runs 30% above its 14-day average. |
| Verify | Confirm it worked. | Did the budget move land, and did CPA respond? |
The guide behind this post carries three example cards for each kind, 24 in all. Each card lists its inputs, an acceptance check and its limits. Start each request with the account, the period and the objective. Check access, definitions and freshness first. Then ask the question and say what to deliver, what counts as a pass and where the answer stops. Three cards are printed in full with example values filled in. They are the ones the walkthrough and the scorecard use. The table after them prints the question, the check and the limit for the other 21, and their inputs stay in the guide.
Q04, an Explain card. Why did cost per lead increase?
- Inputs. Clicks, spend, leads, settings and change history.
- Request, example values filled in. Why did cost per lead increase? Context: our Meta and Google Ads accounts, March to April 2026, cost per lead. Check access, definitions and freshness first. Deliver: A breakdown of the change and tested explanations. Acceptance: Every explanation has evidence. Unproven causes are labelled. Limitation: Changes happening together do not establish causation. For now, read and prepare only. Do not change external systems.
Q16, an Execute card. Create the approved campaign in a paused state.
- Inputs. Approved brief, account, permissions and budget constraints.
- Request, example values filled in. Prepare a plan for this task: Create the approved campaign in a paused state. Context: the Google Ads account, the approved Q4 launch brief, a paused state and the agreed budget. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: Created objects with confirmed settings. Verification: Objects are read back. Delivery has not been activated. For now, read and prepare only. Do not change external systems.
Q24, a Verify card. Did the change actually persist?
- Inputs. Exact object, expected fields and operation response.
- Request, example values filled in. Did the change actually persist? Context: the same Google Ads account, the week after the change, the budget move and its cost per lead. Check access, definitions and freshness first. Deliver: Confirmation of the intended object state. Acceptance: There is a readback. An unknown result does not trigger a blind retry. Limitation: A dispatched command is not a confirmed change. For now, read and prepare only. Do not change external systems.
| Kind | Question | Acceptance check | Limit |
|---|---|---|---|
| Describe | Q01. How much did we spend, and what did we get? | Totals reconcile with details. Sources and freshness are visible. | Different attribution windows need an explicit reconciliation. |
| Describe | Q02. Which channels generated qualified meetings? | Meetings are deduplicated. Channel assignment rules are visible. | An attributed meeting does not prove incremental channel impact. |
| Describe | Q03. Which ad formats perform best? | Sample sizes, segments and comparability are visible. | Different audiences can explain differences in average performance. |
| Explain | Q05. Where are potential customers dropping out? | Conversion delays are considered. Cohorts have comparable maturity. | A missing event can be a data problem. |
| Explain | Q06. Did performance fall, or are the data delayed? | Source, load and calculation use the same period. | An empty table does not establish zero business activity. |
| Forecast | Q07. Will we exceed the budget by month end? | Calculation date, assumptions and alternative scenarios are visible. | Extending the current pace is a conditional projection. |
| Forecast | Q08. How many meetings could we generate by period end? | Known volume is separated from future assumptions. | A different lead mix can invalidate the historical conversion rate. |
| Forecast | Q09. What would change if we increased the budget? | No linear increase in leads is promised without response evidence. | Average cost per lead is not marginal cost per additional lead. |
| Choose | Q10. Where should we reallocate part of the budget? | Keeping the current allocation is an option. Advice is separate from action. | Tool access does not authorize spending changes. |
| Choose | Q11. Which hypothesis should we test next? | Hypotheses are testable and success is defined in advance. | An idea list alone does not establish expected lift. |
| Choose | Q12. Which campaigns should we investigate first? | Priority is explained. High spend is not treated as a cause. | Investigation priority does not authorize automatic pausing. |
| Prepare | Q13. Prepare a brief for the next campaign. | Requirements cite sources. Missing information is labelled. | The quality of a brief still needs subject-matter review. |
| Prepare | Q14. Write ad variants from the approved brief. | No invented facts. The tone and format match the requirements. | Copy is proven by a click-through or reply test against the control, not by the draft. |
| Prepare | Q15. Prepare a campaign measurement plan. | Every metric has a source, calculation and owner. | The plan still needs implementation and verification on actual events. |
| Execute | Q17. Update the exclusion audience from the CRM (customer records). | Included and excluded records are checked. No extra changes. | Support and permissions are checked for the specific connection. |
| Execute | Q18. Update verified contact fields. | The right records changed and unrelated records did not. | Identify a contact by its exact key, an email or a CRM id, and treat a name match as a candidate. |
| Monitor | Q19. Alert me when spend deviates from plan. | Alerts are delivered, duplicates controlled and check failures visible. | Monitoring counts as working after a deliberately failed check produces a visible alert. |
| Monitor | Q20. Warn me when data arrive late. | Source delay, load delay and report delay are distinguished. | Silence proves nothing. Confirm monitoring with a test alert. |
| Monitor | Q21. Flag signs of ad fatigue. | Periods are comparable. A signal is separated from a proven cause. | Read a click-through drop together with frequency and conversion rate before calling it fatigue. |
| Verify | Q22. Why does the report disagree with the ad platform? | The discrepancy is quantified and localized. | Different metric definitions are not automatically a load error. |
| Verify | Q23. Do we have enough evidence to name a test winner? | Cohort comparability and uncertainty are assessed. | A spend threshold is not a test of statistical significance. |
A wrong Describe answer costs a meeting. A wrong Execute action costs money before anyone reads the report.
Sixteen public benchmarks, and none measures business effect
Four of the sixteen benchmarks below were built for marketing work. Two are Mark-Bench and AD-Bench. The other two are xbench's marketing track and the AIMultiple set. The other twelve were built for customer service, software, office work or tool use. Each gets one of three verdicts in the last column: screen only, run your own, or not evidence. Every number is printed as its source showed it on 20 October 2026.
Public benchmarks mapped to the three scored layers, read on 20 October 2026
| Benchmark | What it grades, and how | Test it covers | Limit for your stack | Verdict |
|---|---|---|---|---|
| Mark-Bench (Optimizely, September 2026) | Marketing deliverables such as briefs, plans and copy. Each scenario has a brief, input files and a rubric of 12 to 30 criteria. Large language model (LLM) judges, Claude and Gemini, score the output. The launch article counts 297+ tasks and scenarios across 17 functions, with 3,564 criteria. | Output quality | It never acts in a system. The judge is a model, and the cost figures compare the vendor's own test setup, the software around the model, with a raw model. | Screen only |
| AD-Bench (Tencent, February 2026, revised June 2026) | 225 real marketing-analysis requests from a production advertising platform, in three difficulty levels. An LLM judge, Gemini-3.1-Pro, grades the answer and whether the agent's tool calls match an expert's. Best model Claude-Opus-4.7 at Pass@1 76.9%, falling to 61.4% on the hardest level. | Output quality on Describe and Explain questions | Any change to the platform. The judge is also one of the models under test, which the authors flag. | Screen only: a public test of Describe and Explain questions, the same kinds as the cards above |
| AutomationBench (Zapier, April 2026) | Cross-app workflows in sales, marketing, operations, support, finance and HR. 600 public tasks, 100 per domain, plus a harder private set. Code compares the final system state with the expected state using assertion-based rubrics. A task passes only if every assertion passes. Top of the leaderboard (v1.0.6): Gemini 4 Argon (High) at about 51% on Zapier's private, harder task set, leaderboard read on 20 October 2026 (Zapier says run-to-run variance is typically within 1%). | Action execution | It grades the end state of a task on its own set of apps. The quality of a deliverable and your own accounts are outside it. | Run your own: copy its grader design |
| GDPval (OpenAI, October 2025) | 1,320 one-shot work tasks across 44 occupations. The 220-task open subset is graded by blinded expert pairwise comparison. The automated grader agrees with human experts 65.7% of the time. Humans agree with each other 70.8%. | Output quality | Anything interactive. None of the 44 occupations is a marketing one. | Not evidence for marketing |
| tau-bench (Sierra, June 2024) | Customer service agents talking to a simulated user in retail and airline settings. Graded by comparing the database state after each task with the expected outcome. Introduced pass^k, the share of tasks an agent passes all k times. | Action execution, with a reliability measure | Its tasks are customer-service chats in retail and airline settings, so no ad account or CRM (customer records) is touched. Its authors now mark the original task sets outdated. | Screen only: borrow pass^k |
| BFCL (Berkeley Function Calling Leaderboard) V4, updated 12 April 2026 | Whether a model calls functions accurately. Two checks: the call's syntax tree against possible answers, and executable runs. Overall accuracy is the unweighted average of the sub-categories. | The first step of execution: the call goes out, the result stays unchecked | Whether the right thing changed in the system afterwards. | Screen only |
| MCP-Universe (August 2025) | 231 tasks run through 11 live MCP (Model Context Protocol) servers, the open-standard connections that give an agent tools and data. GPT-5 at 43.72% success, as published. | Action execution through tools | The servers belong to the benchmark, and no business number is read: a task says whether a call reached the right tool. | Screen only: it shows what tool use reached on those tasks |
| MCP-Bench (August 2025) | 104 tasks across 28 live MCP servers and 250 tools. | Action execution through tools | Your servers and your permissions. Any business number. | Screen only |
| MCPMark (September 2025) | 127 tasks on live MCP servers. Best model, gpt-5-medium, as published: 52.56% pass@1, the share passed in one try, and 33.86% pass^4, the share passed all four times. | Action execution through tools, with a reliability measure | Your servers and your permissions. Any business number. | Screen only: borrow pass^4 |
| LiveMCPBench (August 2025) | 95 tasks on 70 live MCP servers with 527 tools. | Action execution through tools | Your servers and your permissions. Any business number. | Screen only |
| CRMArena-Pro (Salesforce, May 2025) | Nineteen expert-validated tasks across sales, service and configure-price-quote processes, in single-turn and multi-turn settings. Its abstract reports leading agents at around 58% single-turn success, dropping to approximately 35% in multi-turn settings, and near-zero inherent confidentiality awareness. | Output quality on CRM reads and multi-turn conversations | A Salesforce sandbox: no ad platform, and none of your own records. | Run your own: copy the confidentiality test |
| TheAgentCompany (December 2024) | 175 workplace tasks, counted only when every checkpoint passes, with cost printed per task instance next to every score. | Execution on general office work | Its tasks are software-company office work, such as code, admin and HR tickets, so no ad account or CRM is touched. | Not evidence |
| OSWorld 2.0 (leaderboard read on 20 October 2026) | 108 long computer-use workflows. Claude Opus 4.8 at 20.6%, as published. | Execution by computer use | Its tasks are desktop operations on an operating system, so no ad platform or CRM is touched. | Not evidence |
| Vending-Bench (February 2025) | A simulated vending business scored on net worth, each experiment run five times. | A simulated business number | A simulated shop with simulated money: no ad spend and no real profit and loss. | Not evidence |
| xbench, marketing track (tested May 2025) | 50 influencer-search tasks graded by a model judge. | Output quality | Influencer search only, graded by a model: no system is changed and no business number is read. | Screen only |
| AIMultiple agentic marketing benchmark | Three workflows built on its own pages. Its authors publish one score per model and say no repeated runs sit behind it. | Output quality | Three workflows on the authors' own pages, run once each: no system is changed and no spread between runs is reported. | Not evidence |
Google and Microsoft each ship an agent evaluator, both marked preview when read on 20 October 2026. Google's agent evaluation checks whether an agent's tool calls match the sequence of calls an expert would make, exactly or in order. Microsoft Foundry's agent evaluators score intent resolution, tool call accuracy and task adherence from 1 to 5. They test the first step of execution on your own traces. None of the sixteen benchmarks above does that.
None of the sixteen measures business effect. Vending-Bench comes closest, and its dollars are simulation money. The two execution benchmarks built on business systems, AutomationBench and tau-bench, both check the system state after the run against a fixed expected outcome. Copy that grader design. The scores themselves come from tasks outside marketing.
How three public skill libraries group marketing work
A skill is a folder with a SKILL.md file inside. The file opens with a name and a description. Then it gives the instructions the agent follows for one kind of work, such as a brand review. Anthropic developed the Agent Skills specification and released it as an open standard. It sets the limits: a name of up to 64 characters and a description of up to 1,024 characters. The agent loads only the name and description at the start, a few lines each, and reads the full instructions when a task matches them. Agent Skills specification.
Three public libraries show how skill authors group marketing work. The counts are folder counts from the GitHub API (application programming interface) on 20 October 2026.
Three public skill libraries, counted on 20 October 2026 and run on 22 October
| Library and skill count | How it groups them | Our run on 22 October 2026 |
|---|---|---|
| marketingskills, 52 skills | Nine categories: Conversion Optimization, Content & Copy, SEO & Discovery, Paid & Distribution, Measurement & Testing, Retention, Growth Engineering, Strategy & Monetization, Sales & RevOps | 9 of 10 copywriting tests passed on Sonnet 5.5 in one run on 22 October 2026, 68 of 69 assertions, a self-check since Sonnet 5.5 graded its own output. The miss wanted facts the brief never gave, so it answered with fill-in placeholders. All 52 skills ship an evals file, and the 69 assertions we ran check structure and wording only. |
| Anthropic marketing plugin, 8 skills | One folder per slash command: draft-content, campaign-plan, brand-review, competitive-brief, performance-report, seo-audit, email-sequence, plus content-creation | brand-review: 10 findings, 6 pinned to a sentence. seo-audit: 55 items, 22 pinned to the page. brand-review ran with no voice guide and no link access. seo-audit's rankings, backlinks and speed items came from the page text alone, since no rank tracker was connected. |
| anthropics/skills, 19 skills | Four sets: Creative & Design, Development & Technical, Enterprise & Communication, Document Skills. | A Word file in 46 seconds, 5 tables, 116 bullets and 40 links kept. It keeps the content and drops the behaviour: the 8 collapsible question cards flattened into headings. |
Six common marketing skills and the question kind each one answers
| Skill | Question kind | Why |
|---|---|---|
| Performance report | Describe | It states what happened. |
| Competitive brief | Describe, with an Explain inside it | It states what rivals do and why it matters. |
| Campaign plan | Prepare | It drafts something a person will approve. |
| Email sequence | Prepare | It drafts a sequence with send timing, for an approver to edit. |
| Content draft | Prepare | It drafts a post or a page, for an approver to edit. |
| Brand review | Choose | It scores a draft against a rubric. |
That covers four of the eight kinds: Describe, Explain, Prepare and Choose. Forecast, Execute, Monitor and Verify fall outside these libraries, since the last three need a live connection to a system. A skill can only tell the agent to use whatever connection its host provides.
A skill library tells you which deliverables an author has practised, and what the agent can do in your accounts shows only once you connect one. The one library with test prompts gives you prompts and expected outputs, which a person or a test setup you build has to run. The front matter of a skill holds a name, a description and an argument hint, and no score. Grade a skill by running it.
Why cost per lead rose: three causes the numbers point to
We chose "Why did cost per lead increase?" because that number rose 3.9x between March and April 2026 in our own accounts. It is card Q04 above. We ran it once, on our own ad accounts, read-only, on 19 October 2026, so treat the ratios below as one reading. The figures compare March with April 2026.
Data access. In our run the agent named its data limits before answering. Spend was loaded through 19 October, and the newest lead was from 18 October. One account had lost its connection in August, and the agent flagged it. It did not estimate that account's numbers. The test needs the ad platforms, the site forms and the CRM through one connection with set permissions, read-only for this question.
Output quality. The answer passed on four counts. It agreed on definitions first: what counts as a lead, which period, which currency, which timezone. "How many leads did we get in March" had four honest answers in our accounts, depending on which form and which attribution you count. The smallest was at most a third of the largest, so the agent had to name the one it used. It reconciles the numbers: one platform's spend matched the warehouse, and the gap on another was traced to a second account whose connection had broken. Causes it could not test stay labelled as untested. The agent proposed three causes and checked each against the data. None was tested against a control.
The three causes and the evidence behind each, March to April 2026
| Cause | Evidence |
|---|---|
| A paused source | The biggest lead source had all its campaigns paused in March. |
| Dropped campaigns | Two cheap mid-funnel campaigns were dropped that month. |
| Spend with no leads | One platform took half of all spend in both months and brought no leads at all. |
In that run, comparing March with April, the mix shift alone works out to a 3.2x factor. The remaining source's own cost per lead rose 1.2x on top. Both are estimates from that one run, built on each source's March and April spend and lead counts, and no control tested them. The two rounded factors multiply to 3.84, and the unrounded figures give 3.9x. The next step was written down with its verification and left for a person to approve.
Action execution. The agent proposed one action, pausing the empty campaign, with a way to verify it, and stopped. This question writes nothing by design: its card says read and prepare only. The write is tested on an Execute card, such as Q16 above: create the approved campaign in a paused state. After the run you read the platform back through the API. The campaign exists, its status is paused, its budget is the agreed number, and the count of campaigns rose by exactly one. Then check that nothing else changed. If the agent reports success and the read-back disagrees, the read-back wins. Pass it on five fresh briefs, one after the account's data has changed. If all five pass, the Execute kind reaches "checked now". A clean sheet still leaves the failure rate at most about 45%. That is the highest rate that fits zero failures in five independent runs, at 95% confidence and a steady failure rate. The formula is 1 minus the fifth root of 0.05. "Works repeatedly" is earned over the weeks of runs that follow.
Business effect. The business effect here is a plan. The read starts once a person acts on the answer. Pause the empty campaigns, hold back a matched set, and compare cost per lead once the number has settled. The scorecard below gives the sample sizes for this read. That read measures the decision. The agent is measured by running the same question through your current process as the control and comparing the two. In our accounts the four answers above moved with the attribution setting, so platform-reported conversions on their own cannot separate the agent's effect from the market's. Our run covered data access and output quality. The agent proposed the pause, and the business effect read starts once a person runs it.
Six checks for any leaderboard score
- AutomationBench ranks models on a private, harder set. Zapier ships 600 public tasks. Its AutomationBench README says scores you measure locally may not match the official leaderboard 1:1.
- The grader can be code, people or a model. AutomationBench compares the final environment state with the expected state using assertion-based rubrics. GDPval used blinded human experts and its paper (October 2025) reports that its automated grader agreed with them 65.7% of the time, against 70.8% between two humans. Mark-Bench uses Claude and Gemini as judges. When a model did the grading, ask for its agreement rate with people. GDPval paper.
- One run hides reliability. Zapier reports run-to-run variance typically within 1%. That is the spread of the average. It says nothing about one task's reliability. The reliability number is pass^k, the share of tasks an agent passes every time in k tries. Sierra's tau-bench paper found that function-calling agents such as gpt-4o succeeded on fewer than half the tasks. Its blog (20 June 2024) puts the retail pass^8 score at about 25%, which the post calls a 60% drop from its pass^1 score. MCPMark prints both numbers for one model, gpt-5-medium, as published: 52.56% pass@1, the share passed in one try, and 33.86% pass^4, the share passed all four times. A weekly task runs 52 times a year. Its pass^52 is the share of tasks it passes on all 52 runs, and that share cannot be higher than pass^4 for the same tasks. Our weekly task has no pass^52 yet. MCPMark's published pass^4 comes from other tasks, so it does not give our number.
- Versions drift. The tau-bench repository, read on 20 October 2026, says its airline and retail tasks are outdated. BFCL is on V4, last updated 12 April 2026. Mark-Bench's own pages disagree with each other: the launch article counts 297+ tasks across 17 functions, the press release 285 tasks across 15. Quote the count from the page you link, with its date. A score without a version and a date cannot be compared with another score.
- A model row can hide a fallback model. Zapier's leaderboard, read on 20 October 2026, notes that some rows use default fallbacks: a step the model refuses is routed to the provider's own fallback model. Its footnote for the "Claude Fable 5.1 (with Opus 5 Fallback)" row says Opus 5 handled 260 of 657 tasks, about 40%. Those completions count toward the score. TheAgentCompany prints cost per task instance next to every score. Ask for both the fallback policy and the cost per completed task. Zapier leaderboard.
- Mark-Bench's test setup belongs to its vendor. Its cost claim compares Optimizely's own test setup with a raw model. A vendor's setup can also go away: OpenAI's deprecations page, read on 21 October 2026, says Agent Builder shuts down on 30 November 2026. Keep your own 12 questions and your own log. OpenAI deprecations.
MCP sets what the agent can read and change
MCP, the Model Context Protocol, is the open standard that connects an AI application to outside systems. A server exposes three things: resources, prompts and tools. Resources are context and data, prompts are templated workflows, and tools are functions the model can execute. The version cited here is the specification dated 2026-07-28.
A tool definition may carry annotations, optional properties that describe how the tool behaves. The tools page says: "clients MUST consider tool annotations to be untrusted unless they come from trusted servers." And hosts must obtain explicit user consent before invoking any tool. The tool list, read with the account's own permissions, shows what the agent may change.
Before any Execute question, list the tools the server exposes. Count the ones that change a budget, a status or an audience in one call. Run your 12 questions once under a read-only connection and once under the full one, and compare what fails. In our reasoning, not a measured result, a full-rights run can miss failures that appear only under a narrower connection, so rerun the same 12 questions on the narrow one. The rules for who may change what belong in a written policy. Our guide to AI agent governance covers that side.
What Improvado MCP gives the agent
We offer one such connection, Improvado MCP. Its docs, read on 21 October 2026, describe one connection with set permissions and read and write tools. Its tools pull live numbers from the ad and analytics platforms. They run SQL (database queries) on the ClickHouse warehouse in your workspace and query BigQuery, Snowflake and Redshift directly. It is our product and the docs are our own description of it. Run it through the same 12-question scorecard as any other connection before you trust a number from it. The walkthrough above used only read tools, so the write path still needs its own Execute run. Start on the read tools: a Describe question on your own data, with the answer pulled by hand first, needs no spend and no holdout.
A 12-question scorecard you can run this quarter
Pick five questions from the cards above and run each one twice as a pilot. The person who scores an answer must not be the person who ran the agent. A question that passes both runs goes into the full scorecard below.
Four readiness levels label how far an agent is proven on a question.
Readiness levels for one question kind on one system
| Level | What it means | How you get there |
|---|---|---|
| Described | The vendor says the agent can do it. | Read the docs. Nothing else. |
| Historically checked | It answered correctly on past data with known answers. | Run it on last quarter's questions. Compare to what actually happened. |
| Checked now | It answered correctly on live data this week. | Run the same questions on current data. Verify against the source system. |
| Works repeatedly | It has passed the same check several times, including after a data change. | Schedule the check. Keep a log. |
How the 12 questions split by block and kind
| Block | Question kinds |
|---|---|
| Four reads | One Describe, one Explain, one Forecast, one Choose |
| Four that draft or change | Two Prepare, two Execute |
| Four that repeat a check | Two Monitor, two Verify |
We ran one of the twelve questions, Q04, through data access and output quality.
- Pick 12 questions from the three blocks in the table above. Spread them over the ad accounts, the CRM, the site and the warehouse.
- Write the known answer for each Describe question before you run the agent, pulled from the source system by hand. For Forecast, ask about a past month, give the agent only the data up to that month's start, and score against what happened. For Explain, pick a change whose cause is on record, such as a budget cut or a tracking outage logged at the time.
- Run each question five times. The spread across those runs is the reliability read, as the pass^k check above explains.
- Score reads on correctness first, readability second. Score Execute on system state after the run, and give the agent's summary no weight.
- For the one business number you care about, set a holdout or another control. Read it once the outcome has had time to land and the sample is large enough. A two-sided read for a 30% change at 95% confidence with 80% power needs about 200 leads per arm. That assumes leads arrive at random, so the count sets its own noise and no baseline rate is needed. The arithmetic: 1.96 plus 0.84, squared, times 2.3 (the two arms' rates, 1 and 1.3, added), divided by 0.3 squared, gives 200. At 50 leads a week per arm that is four weeks. A 20% change needs about 430 per arm on the same assumptions, nine weeks at that volume. At 15 leads a week the same reads take 13 and 29 weeks. The control is what makes it an effect test.
- Label each question with its readiness level. Ship only what reaches "checked now" for reads and "works repeatedly" for writes. Until a write reaches "works repeatedly", it stays behind a human approval.
Frequently asked questions
Is a high benchmark score enough to let an agent change campaigns?
A benchmark score covers one layer at most, output quality or execution, and your business number sits outside it. Letting an agent change campaigns needs the execution test on your own accounts. Run the change in a paused or sandbox state, read the system back, and confirm nothing else moved. Repeat it before you remove the human approval step.
How many questions do we need to test an agent properly?
Twelve covers the eight kinds in three blocks: four reads, four that draft or change, four that repeat a check. Reliability comes from the runs, so run each five times. Report the lowest and highest score across the five runs next to the average.
What is the difference between a skill and an MCP server?
A skill is a set of instructions and files that tells the agent how to do one kind of work, such as a brand review. An MCP server is a connection that gives the agent data or actions from a system. A skill tested on fixed inputs proves the instructions. The agent needs its own run on your data, through a connection or an export. Our list of MCP servers for marketing data compares the connections.
Should we trust the agent's own report that it finished a task?
Trust it after a read-back confirms the system state, through the same connection the agent used or a second one. tau-bench compares the database state after each task with the expected outcome.
How long does the business effect test take?
Weeks, and the count depends on weekly volume and the lift you need. Pick one number and set a holdout or another control. A 30% change at 95% confidence and 80% power needs about 200 leads per arm. At 50 leads a week, that is four weeks. A 20% change at the same confidence and power needs about 430 per arm, nine weeks at that volume.
