DoorDash · Offline simulation and evaluation flywheel for an LLM support chatbot

A Evidence level

A Simulation and Evaluation Flywheel to Develop LLM Chatbots

Publisher: DoorDash Engineering · Company engineering blog · Direct case-level source

Claim origin: Company engineering team disclosure · Independent verification: No · Accessed: 2026-09-19

Evidence level measures whether a source can be located and reviewed; it does not mean vendor-reported claims were independently audited.

Deep case | Key delivery chain is substantially documented 12 / 12

Business context2 / 2

Transformation workflow2 / 2

Technical workflow2 / 2

Human roles & governance2 / 2

Measured outcomes2 / 2

Source traceability2 / 2

All six dimensions meet the current completeness threshold.

DoorDash turns real failures into scenarios, simulation, evaluation, regression tests, A/B release, and production monitoring—allowing teams to improve an LLM system without using customers as unstructured test traffic.

Business problem

After DoorDash replaced deterministic support decision trees with an LLM conversation system, a prompt change could improve one scenario while degrading another. Direct production testing would expose customers to risk; manual replay was slow and narrow; and edge cases such as fraud, high-value refunds, and extreme delays were especially difficult to validate.

Solution

DoorDash built connected offline simulation and evaluation systems. A pipeline extracts behavioral scenarios from historical production transcripts; an LLM plays a customer who can clarify, push back, and escalate; a mock layer blends production and test data while simulating delivery, refund, and order-tool responses; and calibrated LLM-as-a-judge evaluators score hallucination, tone, and classification. Changes pass regression gates before A/B deployment, and the same evaluations monitor production.

Technical architecture & production workflow

Step 01

Historical production transcripts generate behavioral test scenarios

→

Step 02

LLM customer simulator produces clarification, pushback, and escalation behavior

→

Step 03

Hybrid mocks combine production and test data for tool and backend responses

→

Step 04

Support chatbot calls simulated delivery, refund, and order tools across multi-turn conversations

→

Step 05

Calibrated LLM-as-a-judge plus more than 50 quality evaluations

→

Step 06

Regression gates, A/B deployment, and the same evaluations in production monitoring

Key technology & infrastructure components

Scenario-generation pipelineLLM customer simulatorHybrid mock dataTool-call simulationLLM-as-a-judgeRegression testingA/B experimentationProduction monitoring

Human roles & accountability

Domain experts calibrate automated evaluators against expert judgment. Engineering and data teams use failure cases to change prompts, context, or tools. A change enters the standard A/B process only after guardrails remain stable, and production monitoring checks whether the offline improvement persists.

FDE delivery actions

Derive realistic behavior from historical transcripts instead of hand-writing every script
Simulate both user behavior and backend tool responses
Calibrate automated evaluators against domain experts
Turn each production failure into a permanent regression evaluation
A/B deploy only after guardrails pass and then verify offline/online correlation

Reusable delivery patterns

Non-deterministic agents need behavioral simulation, not only single-turn tests
An LLM evaluator must be calibrated against expert judgment
Test environments must cover tool responses and business state
Production failures should become permanent regression tests

Business outcomes & delivery results

DoorDash reports that one major change reduced hallucinations by 90% in simulation and that the improvement carried into production. Iteration fell from days to hours; the system runs more than 200 multi-turn conversations in under five minutes; and the suite contains more than 50 evaluations.

Evidence boundaries & verification notes

The 90% reduction was first measured in simulation. DoorDash states that the improvement carried into production but does not publish the absolute baseline.
More than 200 conversations in under five minutes is test throughput, not production support throughput.

Primary source: A Simulation and Evaluation Flywheel to Develop LLM Chatbots ↗