How to Evaluate Marketing AI Agents: Four Layers

Judge a marketing AI agent on four layers. Data access asks what it can read and change, through which connection, and whether it says when the data is stale. Output quality asks whether its answer is right and readable. Action execution asks whether it changed the right thing in the right system, and nothing else. Business effect asks whether the number you care about moved, against a control. One demo can prove at most one of them. None of the sixteen public benchmarks below covers all four.

A benchmark score tells you how an agent did in someone else's sandbox. It cannot tell you whether it moved the right budget in yours. Sixteen public benchmarks and three skill libraries map to the three scored layers below, and one section covers data access. A scorecard at the end runs all four on your own data in a quarter.

The four layers of the test

An agent can write a confident, well-structured answer about last month's cost per acquisition (CPA) that is wrong, because it read the wrong column. A rubric or a judge model grades the writing. It catches the wrong column only when it can see the source data or the right answer, so the output test needs one of the two.

A perfect budget recommendation can land on the wrong campaign. Execution tests catch that only when someone checks the system state after the run. The agent's own report of what it did proves nothing.

Flawless execution for a month can move nothing, because the lever it pulled never mattered. Only a controlled read of the business number tells you that, and that read takes weeks.

What each test proves, and what it misses

The eight kinds of question a marketing agent gets

Before you pick a benchmark, write down what you will ask the agent. Marketing questions fall into eight kinds. The first four only read data. The fifth drafts. The sixth changes a system. The last two check on the system afterwards: one keeps watching, one confirms the change landed.

1. Describe. What happened? "Which campaigns spent the most last week?"
2. Explain. Why did it happen? "Why did the cost per lead rise in September?"
3. Forecast. What will happen? "Where will CPA land next month at this spend?"
4. Choose. Which option? "Which three ad sets should we pause first?"
5. Prepare. Draft the thing. "Write the Q4 budget proposal with the numbers filled in."
6. Execute. Change the system. "Move 20% of the budget from the losing ad set to the winning one."
7. Monitor. Keep checking. "Tell me when any campaign's CPA runs 30% above its 14-day average."
8. Verify. Confirm it worked. "Did the budget move land, and did CPA respond?"

The guide behind this post has three example cards for each kind, 24 in all. Each card carries its inputs, an acceptance check, its limits, and a request you can copy into any agent. Open a kind to see its three.

Describe: three example cards

Q01. How much did we spend, and what did we get?

• Inputs. Account, dates, currency and the definition of a result.
• Acceptance check. Totals reconcile with details. Sources and freshness are visible.
• Limits. Different attribution windows need an explicit reconciliation.
• Request to copy. How much did we spend, and what did we get? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Spend and results for an agreed period. Acceptance: Totals reconcile with details. Sources and freshness are visible. Limitation: Different attribution windows need an explicit reconciliation. For now, read and prepare only. Do not change external systems.

Q02. Which channels generated qualified meetings?

• Inputs. CRM, lead source, statuses and event history.
• Acceptance check. Meetings are deduplicated. Channel assignment rules are visible.
• Limits. An attributed meeting does not prove incremental channel impact.
• Request to copy. Which channels generated qualified meetings? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A channel comparison using one definition of a meeting. Acceptance: Meetings are deduplicated. Channel assignment rules are visible. Limitation: An attributed meeting does not prove incremental channel impact. For now, read and prepare only. Do not change external systems.

Q03. Which ad formats perform best?

• Inputs. Assets, format labels, spend and conversions.
• Acceptance check. Sample sizes, segments and comparability are visible.
• Limits. Different audiences can explain differences in average performance.
• Request to copy. Which ad formats perform best? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A format comparison with the volume of observations. Acceptance: Sample sizes, segments and comparability are visible. Limitation: Different audiences can explain differences in average performance. For now, read and prepare only. Do not change external systems.

Explain: three example cards

Q04. Why did cost per lead increase?

• Inputs. Clicks, spend, leads, settings and change history.
• Acceptance check. Every explanation has evidence. Unproven causes are labelled.
• Limits. Changes happening together do not establish causation.
• Request to copy. Why did cost per lead increase? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A breakdown of the change and tested explanations. Acceptance: Every explanation has evidence. Unproven causes are labelled. Limitation: Changes happening together do not establish causation. For now, read and prepare only. Do not change external systems.

Q05. Where are potential customers dropping out?

• Inputs. Transition events and comparable lead cohorts.
• Acceptance check. Conversion delays are considered. Cohorts have comparable maturity.
• Limits. A missing event can be a data problem.
• Request to copy. Where are potential customers dropping out? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A loss map between agreed funnel stages. Acceptance: Conversion delays are considered. Cohorts have comparable maturity. Limitation: A missing event can be a data problem. For now, read and prepare only. Do not change external systems.

Q06. Did performance fall, or are the data delayed?

• Inputs. Latest source records, load status and report records.
• Acceptance check. Source, load and calculation use the same period.
• Limits. An empty table does not establish zero business activity.
• Request to copy. Did performance fall, or are the data delayed? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A freshness check and location of the gap. Acceptance: Source, load and calculation use the same period. Limitation: An empty table does not establish zero business activity. For now, read and prepare only. Do not change external systems.

Forecast: three example cards

Q07. Will we exceed the budget by month end?

• Inputs. Spend, budget limit, calendar and planned changes.
• Acceptance check. Calculation date, assumptions and alternative scenarios are visible.
• Limits. Extending the current pace is a conditional projection.
• Request to copy. Will we exceed the budget by month end? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A spend forecast under a stated spending pace. Acceptance: Calculation date, assumptions and alternative scenarios are visible. Limitation: Extending the current pace is a conditional projection. For now, read and prepare only. Do not change external systems.

Q08. How many meetings could we generate by period end?

• Inputs. Inbound history and time to a qualified meeting.
• Acceptance check. Known volume is separated from future assumptions.
• Limits. A different lead mix can invalidate the historical conversion rate.
• Request to copy. How many meetings could we generate by period end? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Scenarios that account for volume and conversion delay. Acceptance: Known volume is separated from future assumptions. Limitation: A different lead mix can invalidate the historical conversion rate. For now, read and prepare only. Do not change external systems.

Q09. What would change if we increased the budget?

• Inputs. Constraints, past budget changes and any available response curve.
• Acceptance check. No linear increase in leads is promised without response evidence.
• Limits. Average cost per lead is not marginal cost per additional lead.
• Request to copy. What would change if we increased the budget? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Conditional scenarios with explicit uncertainty. Acceptance: No linear increase in leads is promised without response evidence. Limitation: Average cost per lead is not marginal cost per additional lead. For now, read and prepare only. Do not change external systems.

Choose: three example cards

Q10. Where should we reallocate part of the budget?

• Inputs. Objective, limits, response evidence and learning constraints.
• Acceptance check. Keeping the current allocation is an option. Advice is separate from action.
• Limits. Tool access does not authorize spending changes.
• Request to copy. Where should we reallocate part of the budget? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Allocation options and the cost of each trade-off. Acceptance: Keeping the current allocation is an option. Advice is separate from action. Limitation: Tool access does not authorize spending changes. For now, read and prepare only. Do not change external systems.

Q11. Which hypothesis should we test next?

• Inputs. Past results, weak points and available resources.
• Acceptance check. Hypotheses are testable and success is defined in advance.
• Limits. An idea list alone does not establish expected lift.
• Request to copy. Which hypothesis should we test next? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A ranked set of hypotheses and ways to evaluate them. Acceptance: Hypotheses are testable and success is defined in advance. Limitation: An idea list alone does not establish expected lift. For now, read and prepare only. Do not change external systems.

Q12. Which campaigns should we investigate first?

• Inputs. Metric changes, spend volume and data freshness.
• Acceptance check. Priority is explained. High spend is not treated as a cause.
• Limits. Investigation priority does not authorize automatic pausing.
• Request to copy. Which campaigns should we investigate first? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: An investigation queue based on possible loss and confidence. Acceptance: Priority is explained. High spend is not treated as a cause. Limitation: Investigation priority does not authorize automatic pausing. For now, read and prepare only. Do not change external systems.

Prepare: three example cards

Q13. Prepare a brief for the next campaign.

• Inputs. Product context, brand rules, past assets and findings.
• Acceptance check. Requirements cite sources. Missing information is labelled.
• Limits. The quality of a brief still needs subject-matter review.
• Request to copy. Prepare a brief for the next campaign. Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A reviewable brief with an objective, audience, message and constraints. Acceptance: Requirements cite sources. Missing information is labelled. Limitation: The quality of a brief still needs subject-matter review. For now, read and prepare only. Do not change external systems.

Q14. Write ad variants from the approved brief.

• Inputs. Approved brief, format limits and allowed evidence.
• Acceptance check. No invented facts. The tone and format match the requirements.
• Limits. Good copy does not guarantee more sales.
• Request to copy. Write ad variants from the approved brief. Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Drafts in the required format with supported claims. Acceptance: No invented facts. The tone and format match the requirements. Limitation: Good copy does not guarantee more sales. For now, read and prepare only. Do not change external systems.

Q15. Prepare a campaign measurement plan.

• Inputs. Objective, user journey, available systems and data.
• Acceptance check. Every metric has a source, calculation and owner.
• Limits. The plan still needs implementation and verification on actual events.
• Request to copy. Prepare a campaign measurement plan. Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Events, metric definitions and validation checks. Acceptance: Every metric has a source, calculation and owner. Limitation: The plan still needs implementation and verification on actual events. For now, read and prepare only. Do not change external systems.

Execute: three example cards

Q16. Create the approved campaign in a paused state.

• Inputs. Approved brief, account, permissions and budget constraints.
• Acceptance check. Objects are read back. Delivery has not been activated.
• Limits. A prepared campaign and a live campaign are different outcomes.
• Request to copy. Prepare a plan for this task: Create the approved campaign in a paused state. Context: [account or object], [objective], [constraints]. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: Created objects with confirmed settings. Verification: Objects are read back. Delivery has not been activated. For now, read and prepare only. Do not change external systems.

Q17. Update the exclusion audience from CRM.

• Inputs. Unambiguous selection rules and an agreed source.
• Acceptance check. Included and excluded records are checked. No extra changes.
• Limits. Support and permissions are checked for the specific connection.
• Request to copy. Prepare a plan for this task: Update the exclusion audience from CRM. Context: [account or object], [objective], [constraints]. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: A change to the authorized audience membership only. Verification: Included and excluded records are checked. No extra changes. For now, read and prepare only. Do not change external systems.

Q18. Update verified contact fields.

• Inputs. Exact contact key, value provenance and allowed fields.
• Acceptance check. The right records changed and unrelated records did not.
• Limits. A matching name alone is not enough to identify a contact.
• Request to copy. Prepare a plan for this task: Update verified contact fields. Context: [account or object], [objective], [constraints]. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: Confirmed changes to the intended records. Verification: The right records changed and unrelated records did not. For now, read and prepare only. Do not change external systems.

Monitor: three example cards

Q19. Alert me when spend deviates from plan.

• Inputs. Plan, schedule, threshold, recipient and expiry condition.
• Acceptance check. Alerts are delivered, duplicates controlled and check failures visible.
• Limits. One successful read does not establish functioning monitoring.
• Request to copy. Prepare monitoring for this request: Alert me when spend deviates from plan. Context: [account], [threshold], [recipient], [expiry]. Show sources, rules, schedule and failure handling. Verification after activation: Alerts are delivered, duplicates controlled and check failures visible. For now, read and prepare only. Do not create schedules or send messages.

Q20. Warn me when data arrive late.

• Inputs. Expected update frequency and acceptable delay.
• Acceptance check. Source delay, load delay and report delay are distinguished.
• Limits. No alert does not establish that monitoring is healthy.
• Request to copy. Prepare monitoring for this request: Warn me when data arrive late. Context: [account], [threshold], [recipient], [expiry]. Show sources, rules, schedule and failure handling. Verification after activation: Source delay, load delay and report delay are distinguished. For now, read and prepare only. Do not create schedules or send messages.

Q21. Flag signs of ad fatigue.

• Inputs. History of frequency, clicks, conversion and delivery conditions.
• Acceptance check. Periods are comparable. A signal is separated from a proven cause.
• Limits. A lower click-through rate alone does not prove creative fatigue.
• Request to copy. Prepare monitoring for this request: Flag signs of ad fatigue. Context: [account], [threshold], [recipient], [expiry]. Show sources, rules, schedule and failure handling. Verification after activation: Periods are comparable. A signal is separated from a proven cause. For now, read and prepare only. Do not create schedules or send messages.

Verify: three example cards

Q22. Why does the report disagree with the ad platform?

• Inputs. One account, period, currency and attribution definition.
• Acceptance check. The discrepancy is quantified and localized.
• Limits. Different metric definitions are not automatically a load error.
• Request to copy. Why does the report disagree with the ad platform? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A staged reconciliation and any unexplained remainder. Acceptance: The discrepancy is quantified and localized. Limitation: Different metric definitions are not automatically a load error. For now, read and prepare only. Do not change external systems.

Q23. Do we have enough evidence to name a test winner?

• Inputs. Assignment, metric, dates, stopping rules and data loss.
• Acceptance check. Cohort comparability and uncertainty are assessed.
• Limits. A spend threshold is not a test of statistical significance.
• Request to copy. Do we have enough evidence to name a test winner? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A review of design, sample and the permitted conclusion. Acceptance: Cohort comparability and uncertainty are assessed. Limitation: A spend threshold is not a test of statistical significance. For now, read and prepare only. Do not change external systems.

Q24. Did the change actually persist?

• Inputs. Exact object, expected fields and operation response.
• Acceptance check. There is a readback. An unknown result does not trigger a blind retry.
• Limits. A dispatched command is not a confirmed change.
• Request to copy. Did the change actually persist? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Confirmation of the intended object state. Acceptance: There is a readback. An unknown result does not trigger a blind retry. Limitation: A dispatched command is not a confirmed change. For now, read and prepare only. Do not change external systems.

Cross those eight kinds with six subjects: ads, funnel, creatives, measurement, budget, and CRM (customer records) with its processes. That gives 48 cells. You do not need all 48. Twelve questions, spread over the six subjects and all eight kinds, cover reads, drafts, writes and checks.

Risk rises down the list. A wrong Describe answer costs a meeting. A wrong Execute action costs money before anyone reads the report. So the tests below weight execution checks above writing quality for the last three kinds.

What the public benchmarks measure

Sixteen public benchmarks get cited when someone sells a marketing agent. Four were built for marketing work. The rest were built for customer service, software, office work or tool use, and they still get quoted. Each gets a verdict from a fixed set: screen only, run your own, or not evidence. Every number is printed as its source showed it on 20 October 2026.

Public benchmarks mapped to the three scored layers, read on 20 October 2026

Cloud vendors now ship evaluators of their own. Google's agent evaluation checks whether an agent's tool calls match the sequence of calls an expert would make, exactly or in order. Microsoft Foundry's agent evaluators score intent resolution, tool call accuracy and task adherence from 1 to 5. Both were marked preview when read on 20 October 2026. They test the first step of execution on your own traces, which no public leaderboard can.

None of the sixteen measures business effect. Vending-Bench comes closest, and its dollars are simulation money. The two execution benchmarks built on business systems, AutomationBench and tau-bench, both check the system state after the run against a fixed expected outcome. That design is worth copying. The scores are not worth quoting.

Skill libraries and what their taxonomies tell you

A skill is a folder with a SKILL.md file inside. The file starts with a name and a description. The rest is the instructions the agent follows for one kind of work, such as a brand review. Anthropic developed the Agent Skills specification and released it as an open standard. It sets the limits: a name of up to 64 characters and a description of up to 1,024 characters. The agent loads only the name and description at the start, a few lines each, and reads the full instructions when a task matches them. Agent Skills specification.

Three public libraries show how skill authors group marketing work. The counts are folder counts from the GitHub API (application programming interface) on 20 October 2026.

Three public skill libraries, counted on 20 October 2026

Read the skill names against the eight question kinds.

Six common marketing skills and the question kind each one answers

That covers four of the eight kinds: Describe, Explain, Prepare and Choose. No skill here forecasts. Execute, Monitor and Verify need a live connection to a system, and no skill in these libraries carries one. A skill can only tell the agent to use whatever connection its host provides. The libraries sort skills by the marketer's job and ignore what each skill may change.

So a skill library tells you which deliverables an author has practiced. It says nothing about what the agent can do in your accounts. And a skill in these libraries ships with no proof that it works. Even the one library with test prompts gives you prompts and expected outputs, which a person or a harness you build has to run. The front matter of a skill holds a name, a description and an argument hint, and no score. Grade a skill by running it. Its file shows nothing about results.

One question, end to end

Take one Explain question and walk it through all four layers. The question is one paid media teams ask an agent sooner or later: "Why did cost per lead increase?" It is the first Explain card above. We ran it on our own ad accounts on 19 October 2026, read-only, as a reference. The figures below come from that run and compare March with April 2026.

Data access. The agent needs the ad platforms, the site forms and the CRM through one governed connection, read-only for this question. Before it answers, it says what it will read and when each source was last loaded. In our run, spend was loaded through 19 October and the newest lead was from 18 October. One account had lost its connection in August. The agent said so instead of guessing at its numbers.

Output quality. A passing answer does four things. It agrees on definitions first: what counts as a lead, which period, which currency, which timezone. In our run, "how many leads did we get in March" had four honest answers, depending on which form and which attribution you count. The smallest was under a third of the largest. The agent had to name the one it used. It reconciles the numbers. In our run, one platform and the warehouse agreed to within two cents. The gap on another was traced to a second account whose connection had broken. It labels unproven causes. The agent found three tested causes in our run. The biggest lead source had all four of its campaigns paused by 9 March. Two cheap mid-funnel campaigns were dropped on 11 and 13 March. One platform took half of all spend in both months and brought no leads at all. That mix explained a 3.2x rise in cost per lead. The decay in the one source left explained 1.2x more. Together that is 3.9x, and the report showed 3.9x. The two factors are rounded. The total comes from the cost-per-lead figures themselves. It changes nothing: the next step was written down with its verification and left for a person to approve.

Action execution. This question writes nothing, by design. Its card says read and prepare only. The agent proposed one action, pausing the empty campaign, with a way to verify it, and stopped. The write is tested on an Execute card, such as the first one above: create the approved campaign in a paused state. After the run you read the platform back through the API. The campaign exists, its status is paused, its budget is the agreed number, and the count of campaigns rose by exactly one. Then the check that is easy to skip: nothing else changed. If the agent reports success and the read-back disagrees, the read-back wins. Pass it five times on five fresh briefs, one of them after the account's data has changed, and the Execute kind earns "works repeatedly".

Business effect. Nothing to measure yet. The effect test starts when a person acts on the answer, with a holdout or an incrementality test and enough weeks of data for the number to settle. That test measures the decision. The agent is measured by running the same question through your current process as the control and comparing the two. Platform-reported conversions and last-click attribution cannot show the effect on their own, because they move with attribution settings as much as with the market. An agent that answers this question correctly every time has passed three layers out of four. Its business effect is still unmeasured.

How to read a leaderboard without fooling yourself

Six questions sort a useful score from a rumour. Each has a public example.

1. Which task set? Zapier ships 600 public tasks but ranks models on a private, harder set. The AutomationBench README says scores you measure locally may not match the official leaderboard 1:1. A vendor quoting a public-set score is quoting a different test. AutomationBench README.
2. Who graded? Code, people or a model. AutomationBench compares the final environment state with the expected state using assertion-based rubrics. GDPval used blinded human experts and its paper (October 2025) reports that its automated grader agreed with them 65.7% of the time, against 70.8% between two humans. Mark-Bench uses Claude and Gemini as judges. When a model did the grading, ask for its agreement rate with people. GDPval paper.
3. How many runs? Zapier reports run-to-run variance typically within 1%. That is the spread of the average. It says nothing about one task's reliability. The reliability number is pass^k, the share of tasks an agent passes every time in k tries. Sierra's tau-bench paper found that function-calling agents such as gpt-4o succeeded on fewer than half the tasks, and fewer than a quarter on pass^8 in retail. Its blog (20 June 2024) puts the retail pass^8 score at about 25%, a 60% fall from the single-attempt score. MCPMark prints both numbers for one model, gpt-5-medium, as published: 52.56% pass@1, the share passed in one try, and 33.86% pass^4, the share passed all four times. For a weekly task, the second number is the one you will live with.
4. Which version? The tau-bench repository now says its airline and retail tasks are outdated. BFCL is on V4, last updated 12 April 2026. Mark-Bench's own pages disagree with each other: the launch article counts 297+ tasks across 17 functions, the press release 285 tasks across 15. Quote the count from the page you link, with its date. A score without a version and a date cannot be compared with another score.
5. What is in the model row? Zapier's leaderboard notes that some rows use default fallbacks: a step the model refuses is routed to the provider's own fallback model. Its footnote for the "Claude Fable 5.1 (with Opus 5 Fallback)" row says Opus 5 handled 260 of 657 tasks, about 40%. Those completions count toward the score. TheAgentCompany prints cost per task instance next to every score. Ask for both the fallback policy and the cost per completed task. Zapier leaderboard.
6. Who owns the harness? Mark-Bench's cost claim compares Optimizely's own harness with a raw model. OpenAI's deprecations page, read on 21 October 2026, says Agent Builder shuts down on 30 November 2026. Its Evals platform goes read-only on 31 October 2026 and shuts down on 30 November 2026. A vendor's tooling can be withdrawn while your tests still depend on it. Keep your own 12 questions and your own log. OpenAI deprecations.

Where MCP fits: the data the agent reads

MCP, the Model Context Protocol, is the open standard that connects an AI application to outside systems. A server exposes three things: resources, prompts and tools. Resources are context and data, prompts are templated workflows, and tools are functions the model can execute. The version cited here is the specification dated 2026-07-28.

For evaluation, two lines in that specification matter more than any feature. First, a tool definition may carry annotations, optional properties that describe how the tool behaves. The tools page says: "clients MUST consider tool annotations to be untrusted unless they come from trusted servers." Second, hosts must obtain explicit user consent before invoking any tool. So a connection does not tell you what the agent may change. The tool list, read together with the account's own permissions, does.

That is the permission test to run before any Execute question. List the tools the server exposes. Count the ones that change a budget, a status or an audience in one call. Run your 12 questions once under a read-only connection and once under the full one, and compare what fails. A benchmark run with full rights says nothing about an agent on a narrower connection. The rules for who may change what belong in a written policy. Our guide to AI agent governance covers that side.

What Improvado MCP gives the agent

We offer one such connection, Improvado MCP. Its docs, read on 21 October 2026, describe one governed connection with read and write access to 1,400+ platforms. Its tools pull live numbers from the ad and analytics platforms. They run SQL (database queries) on the ClickHouse warehouse in your workspace, and query BigQuery, Snowflake and Redshift directly. Start every evaluation on the read tools. A Describe question on your own data, with the answer pulled by hand first, needs no spend and no holdout. Run it before any demo. If your question is which numbers to trust before any agent touches them, start with AI marketing analytics.

A 12-question scorecard you can run this quarter

Start small. Pick five questions from the cards above and run each one twice. Keep the judge apart from the doer: the person who scores an answer is never the person who ran the agent. A question that passes both runs goes into the full scorecard below.

Four readiness levels describe how far an agent has been proven on a given question. They are labels, and they carry no score.

Readiness levels for one question kind on one subject

Run the scorecard like this.

1. Pick 12 questions in three blocks of four. Four reads: one Describe, one Explain, one Forecast, one Choose. Four that draft or change: two Prepare, two Execute. Four that repeat: two Monitor, two Verify. Spread them over the six subjects.
2. Write the known answer for each Describe question before you run the agent, pulled from the source system by hand. For Forecast, ask about a past month, give the agent only the data up to that month's start, and score against what happened. For Explain, pick a change whose cause is on record, such as a budget cut or a tracking outage logged at the time.
3. Run each question five times. Agents vary between runs, and one pass proves nothing.
4. Score reads on correctness first, readability second. Score Execute on system state after the run, and give the agent's summary no weight.
5. For the one business number you care about, set a holdout or another control. Read it once the outcome has had time to land and the sample is large enough. Four weeks is a rule of thumb for paid media, with no source behind it. The control is what makes it an effect test.
6. Label each question with its readiness level. Ship only what reaches "checked now" for reads and "works repeatedly" for writes. Until a write reaches "works repeatedly", it stays behind a human approval.

Frequently asked questions

Is a high benchmark score enough to let an agent change campaigns?

No. A benchmark score covers one test at most, output quality or execution. Your business number sits outside it. Letting an agent change campaigns needs the execution test on your own accounts. Run the change in a paused or sandbox state, read the system back, and confirm nothing else moved. Repeat it before you remove the human approval step.

How many questions do we need to test an agent properly?

Twelve is enough to start: four reads, four that draft or change, four that repeat a check, spread over the six subjects. Run each five times. The spread between runs tells you as much as the average.

What is the difference between a skill and an MCP server?

A skill is a set of instructions and files that tells the agent how to do one kind of work, such as a brand review. An MCP server is a connection that gives the agent data or actions from a system. A skill tested on fixed test inputs proves the instructions. It proves nothing about the agent on your data. For that you also need the data, through a connection or an export. Our list of MCP servers for marketing data compares the connections.

Should we trust the agent's own report that it finished a task?

Not on its own. The execution test reads the system state after the run, through the same connection the agent used or a second one. AutomationBench compares the final system state with the expected state using assertion-based rubrics. tau-bench compares the database state after each task with the expected outcome. An agent's own account of its work is not evidence.

How long does the business effect test take?

Weeks. Pick one number and set a holdout or another control. Read it once the outcome has had time to land and the sample is large enough to show the change you care about. Four weeks is a rule of thumb for paid media numbers, with no source behind it. The smaller the lift you need to see, the longer the read. A test with no control cannot tell the agent's effect from the market's.