Contexts

Eval Context

Explains the _eval context for defining reusable evaluation scenarios and batches for a template. Useful for regression testing AI conversation flows by running simulated users against stubs and scoring the results.

The _eval context is where you define evaluation scenarios for a template. A scenario describes a simulated user, what that user should try to do, and how the result should be judged. When a scenario runs, a scenario agent plays the user and talks to a real stub of your template. Stubber then checks the stub's data and scores the conversation.

The _eval context is inherited from the Adam template, so every template has it.

# Which flows you can evaluate

Evals work with any flow that has a main conversational LLM agent which talks to users through text (chat) messages.

The scenario agent talks to your flow over Agent Link. Stubber enables a default Agent Link setup that ties into any flow, so your agent can reply to the scenario agent without any extra configuration. If your agent already chats on a channel such as WhatsApp or webchat, you don't need to change it to run evals.

Agent Link uses debounce: when the scenario agent sends several messages in quick succession, they're grouped so your agent responds once, rather than starting a separate LLM loop for each message. This keeps eval conversations close to how a real user's burst of messages would be handled.

# Passing data the scenario agent can't see

Sometimes your flow sends information to the user outside the chat, which the scenario agent never receives. For example, your flow might send a one-time PIN (OTP) by SMS and then ask the user to type it into the chat. A real user would read the SMS, but the scenario agent has no phone, so it can't complete the step.

To handle this, add a notification to the target flow that sends the same information to the eval stub over Agent Link. Give it a condition on stub.data._stubber_eval.active = true, so it only sends during an eval run and never to real users.

Add it to the same action as the SMS notification, so the scenario agent gets the OTP at the moment a real user would. The scenario agent can then read the PIN from the message and type it back, as described in its prompt.

# Managing evals

The recommended ways to work with evals are the Evals page in the Editor and Flowpilot.

# The Evals page

Open the Evals page in the Editor at:

        https://editor.stubber.com/<your-org-domain>/evals

      

From the Evals page you can create scenarios, run them and view the results of each run.

# Flowpilot

Flowpilot can create, edit and run both scenarios and batches. Ask it in the Flowpilot chat on your template, for example "Add an eval scenario where a customer asks to cancel their order" or "Create a batch that runs all the greeting scenarios three times".

# Working with the _eval context directly

The rest of this page explains how evals work and how they're stored in the _eval context. That's useful for writing better scenarios, reviewing what Flowpilot creates, and reading results. It also covers working with the _eval context directly, for example to set up schedules.

# Definition versus execution

Evals keep what to test apart from the result of testing it.

A scenario definition is a declarative test specification. It describes the conversation, limits, fixtures and evaluation criteria. On its own it doesn't create a conversation or produce a result.

This split lets you version, review and reuse the same scenarios without mixing configuration with the results of individual runs.

# Defining scenarios

Scenarios you create on the Evals page or with Flowpilot are saved to the template data at eval.scenarios. Each key is the scenario's name, written in snake_case, and each scenario tests one behavior on its own.

To edit them directly instead, use the data_object annotations on the _eval canvas, which expose both eval.scenarios and eval.batches.

        {
  "greeting_scenario": {
    "statement": "The agent can respond helpfully to a greeting.",
    "prompt": "You are Kelly. Greet the agent. After it replies, say you were testing and say goodbye.",
    "summary_prompt": "Summarize the conversation concisely and mask sensitive data.",
    "rubrics": {
      "agent_responded_helpfully": {
        "rubric": "The agent gave Kelly a helpful, appropriate greeting.",
        "threshold": 0.7
      }
    },
    "checks": {
      "stubref_exists": {
        "check_type": "jsonata",
        "check": "$exists(stub.stubref)"
      }
    },
    "scenario_number": 1,
    "limits": {
      "max_interactions": 30,
      "max_messages": 100,
      "max_time_in_seconds": 600,
      "max_time_no_response_in_seconds": 120
    },
    "roles": {
      "user": "Agent",
      "assistant": "Kelly",
      "system": "Instructions"
    },
    "fixtures": {
      "stub_data": {
        "customer_name": "Kelly"
      }
    }
  }
}

      

# Scenario fields

# Writing a good scenario prompt

The scenario agent should behave like a real user, not a tester who knows how the flow works.

• Describe who the user is and what they want, not how the target flow is built.
• Have the user give information only when it would naturally come up.
• Ask for steps in order rather than everything at once.

# Checks or rubrics

Use checks for concrete outcomes you can read from the stub's data, such as whether a field was saved or a state was reached. They're deterministic, so they give the same answer every time.

Use rubrics for conversational quality, such as tone, helpfulness or whether the agent asked the right questions. A rubric passes when its score reaches the threshold.

# Fixtures

Use fixtures.stub_data only for data the target process needs before the conversation starts, for example a customer record the flow expects to exist.

# Using files in a scenario

Some flows ask the user for files, such as an ID document or a proof of address. To test these, give the scenario agent the files it should send:

1. Upload each test file to Stubber, and note its fileuuid. See Files for how files are stored.
2. In the scenario's prompt, list each file's fileuuid and what it's for.
3. Tell the scenario agent to send a file only when the target agent asks for it.

        {
  "submit_documents_scenario": {
    "statement": "The agent collects the customer's ID document and proof of address.",
    "prompt": "You are Kelly, applying for a new account. When the agent asks for documents, send them one at a time, only when asked. Your files are:\n- 3f1c2a9e-7b4d-4e8a-9c1f-2d6b8a0e5f31: your ID document\n- 8142ce6c-f3cd-526c-a5d3-747799ed2b1a: your proof of address (a utility bill)"
  }
}

      

The other scenario fields are left out of this example for brevity.

Use synthetic test documents, never real customer files. Describing what each file is for lets the scenario agent pick the right one, just as a real user would.

# Giving the scenario agent extra actions

By default, the scenario agent can only chat with the target agent. If a scenario needs it to do more, such as looking up test data or checking something outside the conversation, you can give it extra actions to use as tools:

1. Add a state called scenario-running to the _eval context.
2. Add the actions the scenario agent needs to that state, and enable AI on each one. Give each action a clear name and description, since that's how the agent decides when to use it.
3. Mention the actions in the scenario's prompt, saying when the scenario agent should use them.

For example, if your flow emails the customer a reference number and asks for it back, you could add an AI-enabled check_test_inbox action that reads the latest email sent to the test address. The prompt then tells the scenario agent when to use it:

        {
  "email_reference_scenario": {
    "statement": "The agent verifies the customer using the reference number it emailed them.",
    "prompt": "You are Kelly. Ask the agent to update your delivery address. When the agent says it has emailed you a reference number, use the check_test_inbox action to read the email, then give the agent the reference number."
  }
}

      

The other scenario fields are left out of this example for brevity.

Actions in the scenario-running state are available to the scenario agent in every scenario on the template, so name and describe them so they won't be used by mistake in scenarios that don't need them.

# Defining batches

A batch groups scenarios so they can be run together, for example as a regression suite. Batches live in the template data at eval.batches, with snake_case keys. You can ask Flowpilot to create and edit batches for you.

        {
  "core_regression": {
    "description": "Runs the core conversation regression scenarios.",
    "run": {
      "greeting_scenario": {
        "run_count": 2
      },
      "account_help_scenario": {
        "run_count": 3
      }
    }
  }
}

      

A batch doesn't copy scenario definitions. Each key in run references a scenario in eval.scenarios, and run_count (a positive whole number) sets how many times that scenario runs.

Running a scenario more than once samples the agent's behavior, which shows whether it handles the scenario consistently rather than passing by chance.

# Running evals

The simplest way to run scenarios and batches is from the Evals page or Flowpilot.

To start runs from the _eval context directly, use one of these three actions:

Use a custom run for quick investigations. If the scenario turns out to be worth keeping, add it to eval.scenarios so it becomes part of your maintained scenarios.

# Starting a run automatically

You can also start a run as soon as an eval stub is created, by passing one of these parameters in the stub data when you create the stub. The _create action reads them and triggers the matching start action on the new eval stub.

These are runtime parameters for a single stub. Don't add them to scenario or batch definitions.

# Start a named scenario

Set auto_start_scenario to the key of a scenario in eval.scenarios:

        {
  "auto_start_scenario": "greeting_scenario"
}

      

The stub triggers

# Start a one-off custom scenario

Set auto_start_custom_scenario to true and pass the full scenario definition in custom_scenario:

        {
  "auto_start_custom_scenario": true,
  "custom_scenario": {
    "statement": "The agent handles a one-off test request.",
    "prompt": "Ask the agent for help with your test request.",
    "rubrics": {},
    "checks": {}
  }
}

      

The stub triggers

# Batch and scheduled runs

Batches start automatically without these parameters. When a batch runs, Stubber creates a separate eval stub for each planned scenario run, and each of those stubs starts its own scenario. Scheduled batch runs work the same way: each time the schedule fires, the new stub starts the batch automatically.

Stubber manages these starts itself, so you don't need to set anything on the stubs it creates.

# Scheduling batches

You can run a batch on a recurring schedule, for example every weekday morning, to catch regressions without starting runs by hand. The schedule lives on an eval stub and is separate from the batch definition in eval.batches.

# Set up a schedule

1. Create an eval stub, or use an existing one in the active state.
2. Trigger the
   schedule_batch_run
   action.
3. Select the batch to run. The list shows the batches defined in eval.batches.
4. Choose when it runs, using one of these formats:
   • Day of week and time: select a day_of_week (0 for Sunday through 6 for Saturday) and enter a 24-hour time_of_day such as 08:30.
   • Crontab expression: enter a five- or six-field crontab expression. For example, 30 8 * * 1-5 runs at 08:30 Monday to Friday.
5. Set max_count, the maximum number of times the schedule starts the batch. It must be a whole number of at least 1, and defaults to 40.

Schedules use your organization's timezone.

# When the schedule fires

The stub you set the schedule up on becomes the schedule stub. It only manages the schedule. Each time the schedule fires:

1. The schedule stub triggers
   trigger_batch_run
   .
2. That creates a new eval stub for this run of the batch.
3. The new stub starts the batch automatically.
4. The batch creates a stub for each scenario run, as it does when you run a batch by hand.

Every scheduled run gets its own stubs and results, so review each run's results separately. A schedule firing doesn't mean its scenarios passed.

# End a schedule

Trigger

# What happens during a run

1. A target stub is created from the template being evaluated, with any fixtures.stub_data placed at stub.data.
2. The scenario agent follows the scenario's prompt, playing the user.
3. The scenario agent and the target stub's agent talk until the conversation finishes or a limit is reached.
4. The checks are evaluated against the target stub's data.
5. The rubrics are scored against the conversation, or its summary when a summary_prompt is set.
6. Feedback actions update the run when it starts, when the conversation ends, and when evaluation is complete.

# Reading results

You can view the results of each run on the Evals page. A run's results include the conversation transcript, check outcomes, rubric scores and whether a limit ended the conversation.

When a run fails, review the transcript before changing anything. The fix might belong in the template's flow, or the scenario itself might be unclear or unrealistic.

# Keeping evals maintained

• Write statements about outcomes you can observe.
• Keep prompts realistic and independent of how the flow is built, so they still work after the flow changes.
• Prefer checks for data and state outcomes, and rubrics for conversation quality.
• Update batches as you add or retire scenarios, so it's always clear what your regression set covers.