How to Evaluate Marketing AI Agents: Three Tests

Judge a marketing AI agent on three separate tests. Output quality asks whether its answer is right and readable. Action execution asks whether it changed the right thing in the right system, and nothing else. Business impact asks whether the number you care about moved, against a control. None of the sixteen public benchmarks in this guide covers all three.

A benchmark score tells you how an agent did in someone else's sandbox. It does not tell you whether it moved the right budget in yours. A single number hides which of the two you have. This guide maps sixteen public benchmarks and three skill libraries to the three tests, then gives you a scorecard to run all three on your own data in a quarter.

Three tests, not one score

Start with the failure each test catches.

An agent can write a confident, well-structured answer about last month's CPA that is wrong, because it read the wrong column. A rubric or a judge model grades the writing. It catches the wrong column only when it can see the source data or the right answer, so the output test needs one of the two.

An agent can produce a perfect budget recommendation and then apply it to the wrong campaign. Execution tests catch that only if someone checks the system state after the run, not the agent's own report of what it did.

An agent can execute flawlessly for a month and move nothing, because the lever it pulled never mattered. Only a controlled read of the business number tells you that, and that read takes weeks, not minutes.

What each test proves, and what it misses

The eight kinds of question a marketing agent gets

Before you pick a benchmark, write down what you will ask the agent. Marketing questions fall into eight kinds. The first four only read data. The fifth drafts. The sixth changes a system. The last two repeat a check over time.

1. Describe. What happened? "Which campaigns spent the most last week?"
2. Explain. Why did it happen? "Why did the cost per lead rise in September?"
3. Forecast. What will happen? "Where will CPA land next month at this spend?"
4. Choose. Which option? "Which three ad sets should we pause first?"
5. Prepare. Draft the thing. "Write the Q4 budget proposal with the numbers filled in."
6. Execute. Change the system. "Move 20% of the budget from the losing ad set to the winning one."
7. Watch. Keep checking. "Tell me when any campaign's CPA runs 30% above its 14-day average."
8. Verify. Confirm it worked. "Did the budget move land, and did CPA respond?"

Cross those eight kinds with six subjects: ads, funnel, creatives, measurement, budget, and CRM and processes. That gives 48 cells. You do not need all 48. Twelve questions, spread over the six subjects and all eight kinds, cover reads, drafts, writes and checks.

Risk rises down the list. A wrong Describe answer costs a meeting. A wrong Execute action costs money before anyone reads the report. That is why the tests below weight execution checks more heavily than writing quality for the last three kinds.

What the public benchmarks measure

Sixteen public benchmarks get cited when someone sells a marketing agent. Four were built for marketing work. The rest were built for customer service, software, office work or tool use, and they still get quoted. The table maps each one to the three tests and says what it cannot show. Each gets a verdict from a fixed set: screen only, run your own, or not evidence. Every number is printed as its source showed it on 20 October 2026.

Public benchmarks mapped to the three tests, read on 20 October 2026

Two smaller marketing sets belong in the same column. The marketing track of xbench is 50 influencer-search tasks graded by a model judge, tested in May 2025. AIMultiple's agentic marketing benchmark is three workflows built on its own pages, and its authors say they do not report repeated runs, seeds, confidence intervals or error bars. Both grade a deliverable. Neither touches a system.

Cloud vendors now ship evaluators of their own. Google's agent evaluation scores whether an agent's tool calls match a reference trajectory, the sequence of calls an expert would make, exactly or in order. Microsoft Foundry's agent evaluators score intent resolution, tool call accuracy and task adherence from 1 to 5. Both are marked preview. They test the first step of execution on your own traces, which no public leaderboard can.

Two things stand out. First, none of the sixteen measures business impact. Vending-Bench comes closest, and its dollars are simulation money. Second, the two execution benchmarks built on business systems, AutomationBench and tau-bench, both grade the state of the system after the run and never the agent's own report. That is the design to copy, not the score to quote.

Skill libraries and what their taxonomies tell you

A skill is a folder with a SKILL.md file inside. The file starts with a name and a description. The rest is the instructions the agent follows for one kind of work, such as a brand review. Anthropic developed the Agent Skills specification and released it as an open standard. It sets the limits: a name of up to 64 characters and a description of up to 1,024 characters. The agent loads only the name and description at the start, about 100 tokens per skill, and reads the full instructions when a task matches them. Agent Skills specification.

Three public libraries show how skill authors group marketing work. The counts are folder counts from the GitHub API on 20 October 2026.

Three public skill libraries, counted on 20 October 2026

Read the skill names against the eight question kinds. A performance report is a Describe. A competitive brief is a Describe with an Explain inside it. A campaign plan, an email sequence and a draft are Prepare. A brand review is a Choose: it scores a draft against a rubric. That covers four of the eight kinds. Execute, Watch and Verify need a live connection to a system, and no skill in these libraries carries one. A skill can only tell the agent to use whatever connection its host provides. The libraries sort skills by the marketer's job, never by what the skill may change.

So a skill library tells you which deliverables an author has practised. It says nothing about what the agent can do in your accounts. And a skill in these libraries ships with no proof that it works. Even the one library with test prompts gives you prompts and expected outputs, which a person or a harness you build has to run. The front matter of a skill holds a name, a description and an argument hint, not a score. Grade the run, not the file.

One question, end to end

Take one Execute question and walk it through all three tests. This is a constructed walk-through, not a logged run, and the numbers are placeholders you replace with your own. The question is one every paid media team asks an agent sooner or later: "Create the agreed campaign in the ad platform, in paused state."

Output quality. Before the agent touches the platform, it returns the campaign it intends to create: name, objective, daily budget, audience, three ads with their copy. You grade that draft against the agreed plan with a rubric of a dozen criteria, the way Mark-Bench grades a brief. Every field matches or the run fails here. A draft that passes earns the question the label "checked now" for the Prepare kind, and nothing more.

Action execution. The agent creates the campaign. You do not read its summary. You read the platform back through the API or the interface. The campaign exists. Its status is paused. Its budget equals the agreed number. The count of campaigns in the account rose by exactly one. Then the check most teams skip: nothing else changed. No other campaign's status moved. No budget elsewhere changed. If the agent reports success and the read-back disagrees, the read-back wins. Pass this five times on five fresh briefs and the Execute kind earns "works repeatedly". Pass it once and it earns "checked now".

Business impact. There is none to measure yet. The campaign is paused. That is the point of asking for a paused campaign: it separates the execution test from the impact test. The impact test starts when a person switches the campaign on, with a holdout and enough weeks of data for the number to settle, and it measures the campaign, not the agent. To measure the agent, run the same brief through your current process as the control and compare the two. An agent that creates correct paused campaigns every time has passed two tests out of three, and that is enough to let it keep doing exactly that.

How to read a leaderboard without fooling yourself

Six questions sort a useful score from a rumour. Each has a public example.

1. Which task set? Zapier ships 600 public tasks but ranks models on a private, harder set. Its README says scores you measure locally may not match the official leaderboard 1:1. A vendor quoting a public-set score is quoting a different test. AutomationBench README.
2. Who graded? Code, people or a model. AutomationBench scores the final environment state with no model judge. GDPval used blinded human experts and reports that its automated grader agreed with them 65.7% of the time, against 70.8% between two humans. Mark-Bench uses Claude and Gemini as judges. When a model graded, ask for its agreement rate with people. GDPval paper.
3. How many runs? Zapier reports run-to-run variance typically within 1%. That is the spread of the average, not the reliability of one task. The reliability number is pass^k, the share of tasks an agent passes every time in k tries. Sierra's tau-bench paper found that function-calling agents such as gpt-4o succeeded on fewer than half the tasks, and fewer than a quarter on pass^8 in retail. Its blog puts that pass^8 drop at 60% from the single-run score. MCPMark prints both numbers for one model, gpt-5-medium: 52.56% pass@1, the share passed in one try, and 33.86% pass^4, the share passed all four times. For a weekly task, the second number is the one you will live with.
4. Which version? The tau-bench repository now says its airline and retail tasks are outdated. BFCL is on V4, last updated 12 April 2026. Mark-Bench's own pages disagree with each other: the launch article counts 297+ tasks across 17 functions, the press release 285 tasks across 15. Quote the count from the page you link, with its date. A score without a version and a date cannot be compared with another score.
5. What is in the model row? Zapier's leaderboard labels some rows "default fallbacks": a step the model refuses is routed to the provider's own fallback model. Its footnote for the Claude Fable 5.1 row says Opus 5 handled 260 of 657 tasks, and those completions count toward the score. TheAgentCompany prints cost per task instance next to every score. Ask for both the fallback policy and the cost per completed task. Zapier leaderboard.
6. Who owns the harness? Mark-Bench's cost claim compares Optimizely's own harness with a raw model. OpenAI's Agent Builder launched in October 2025 and shuts down on 30 November 2026. Its Evals platform goes read-only on 31 October 2026 and shuts down on 30 November 2026. A vendor's tooling can change under you in little over a year. Keep your own 12 questions and your own log. OpenAI deprecations.

Where MCP fits: the data the agent reads

MCP, the Model Context Protocol, is the open standard that connects an AI application to outside systems. A server exposes three things: resources, which are context and data; prompts, which are templated workflows; and tools, which are functions the model can execute. The current protocol version is 2026-07-28. MCP specification.

For evaluation, two lines in that specification matter more than any feature. First, a tool may carry hints such as readOnlyHint and destructiveHint, flags that say whether the tool only reads or may delete. The specification calls them hints, and its words are: "clients MUST consider tool annotations to be untrusted unless they come from trusted servers." Second, hosts must obtain explicit user consent before invoking any tool. So a connection does not tell you what the agent may change. The tool list, read together with the account's own permissions, does.

That is the permission test to run before any Execute question. List the tools the server exposes. Count the ones that change a budget, a status or an audience in one call. Run your 12 questions once under a read-only connection and once under the full one, and compare what fails. A benchmark run with full rights says nothing about an agent on a narrower connection. The rules for who may change what belong in a written policy; our guide to AI agent governance covers that side.

What Improvado MCP gives the agent

Improvado MCP is one such connection. The docs describe one governed connection with read and write access to the platforms it connects. Its tools run live queries against platform APIs, run SQL against the ClickHouse warehouse in your workspace, and query BigQuery, Snowflake and Redshift directly. Improvado MCP docs. Start every evaluation on the read tools. A Describe question on your own data, with the answer pulled by hand first, needs no spend and no holdout. Run it before any demo. If you are still choosing connections, our list of MCP servers for marketing data compares them. If your question is which numbers to trust before any agent touches them, start with AI marketing analytics.

A scorecard you can run this quarter

Four readiness levels describe how far an agent has been proven on a given question. Use them as labels, not scores.

Readiness levels for one question kind on one subject

Run the scorecard like this.

1. Pick 12 questions in three blocks of four. Four reads: one Describe, one Explain, one Forecast, one Choose. Four that draft or change: two Prepare, two Execute. Four that repeat: two Watch, two Verify. Spread them over the six subjects.
2. Write the known answer for each Describe question before you run the agent, pulled from the source system by hand. For Forecast, ask about a past month, give the agent only the data up to that month's start, and score against what happened. For Explain, pick a change whose cause is on record, such as a budget cut or a tracking outage logged at the time.
3. Run each question five times. Agents vary between runs, and one pass proves nothing.
4. Score reads on correctness first, readability second. Score Execute on system state after the run, never on the agent's own summary.
5. For the one business number you care about, set a holdout or another control and read it once the outcome has had time to land and the sample is large enough; four weeks is the usual floor for paid media. The control is what makes it an impact test.
6. Label each question with its readiness level. Ship only what reaches "checked now" for reads and "works repeatedly" for writes.

Frequently asked questions

Is a high benchmark score enough to let an agent change campaigns?

No. A benchmark score covers one test at most, output quality or execution, never your business number. Letting an agent change campaigns needs the execution test on your own accounts. Run the change in a paused or sandbox state, read the system back, and confirm nothing else moved. Repeat it before you remove the human approval step.

How many questions do we need to test an agent properly?

Twelve is enough to start: four reads, four that draft or change, four that repeat a check, spread over the six subjects. Run each five times. The spread between runs tells you as much as the average.

What is the difference between a skill and an MCP server?

A skill is a set of instructions and files that tells the agent how to do one kind of work, such as a brand review. An MCP server is a connection that gives the agent data or actions from a system. A skill tested on fixtures proves the instructions, not the agent on your data. For that you also need the data, through a connection or an export.

Should we trust the agent's own report that it finished a task?

Not on its own. The execution test reads the system state after the run, through the same connection the agent used or a second one. AutomationBench scores only the final state of the systems the agent touched. tau-bench compares the database state after each task with the expected outcome. An agent's own account of its work is not evidence.

How long does the business impact test take?

Weeks, not minutes. Pick one number, set a holdout or another control, and read it once the outcome has had time to land and the sample is large enough to show the change you care about. Four weeks is a common floor for paid media numbers, not a guarantee. A test with no control cannot tell the agent's effect from the market's.