--- name: al-653-test-support-agent-updates-on-simulated-customers-first-r description: "Nubank (Nov 5, 2026) reports that its AI Research team cut average AI support-agent update cycles from 21.2 days to 4.4 days by testing new versions against synthetic customers in large-scale simulation before any live rollout. The paper “Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale” describes multi-turn conversations with synthetic customers that have different profiles and needs. Simulation guided changes to a Card Management agent (address updates and customer-requested reissues); offline, one version cut unnecessary human handoffs from 22.4% to 16.0%. In a live A/B test against a Card Delivery agent, the simulation-built agent showed a 4.9-percentage-point higher self-service rate and a 36.7-point higher transactional NPS. The team also screened 29 open-weight model configurations across more than 16,000 simulated conversations before refining a Qwen setup that later raised self-service 8.8 points and cut p95 latency 25% in a live…" --- # Test support-agent updates on simulated customers first: run multi-turn chats with synthetic buyers, keep only the versions that cut unnecessary handoffs, then A/B the winner with real people Nubank (Nov 5, 2026) reports that its AI Research team cut average AI support-agent update cycles from 21.2 days to 4.4 days by testing new versions against synthetic customers in large-scale simulation before any live rollout. The paper “Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale” describes multi-turn conversations with synthetic customers that have different profiles and needs. Simulation guided changes to a Card Management agent (address updates and customer-requested reissues); offline, one version cut unnecessary human handoffs from 22.4% to 16.0%. In a live A/B test against a Card Delivery agent, the simulation-built agent showed a 4.9-percentage-point higher self-service rate and a 36.7-point higher transactional NPS. The team also screened 29 open-weight model configurations across more than 16,000 simulated conversations before refining a Qwen setup that later raised self-service 8.8 points and cut p95 latency 25% in a live A/B. Nubank notes self-service alone does not prove every issue was resolved, and that simulators must be validated before teams rely on them. The prompt below gives any team a smaller version of that screen-before-you-serve loop. ## Inputs to collect first Ask the user for anything below that they haven't already given you. Don't guess missing facts. - E.g. card delivery status, address updates, returns - Prompt edit, new model, new tool, new handoff rule - E.g. fewer unnecessary human handoffs, no drop in satisfaction… - CHECK ## Instructions Do the task below with the user's inputs in place of each [bracket]. Help me screen a support-agent change on simulated customers before real ones see it. Agent job: [e.g. card delivery status, address updates, returns]. Change I'm testing: [prompt edit, new model, new tool, new handoff rule]. Success criteria set in advance: [e.g. fewer unnecessary human handoffs, no drop in satisfaction, latency under X]. 1) Invent 20 synthetic customers with different profiles and needs (frustrated, first-time, policy edge case, unclear ask, escalation-seeker). For each, write a multi-turn chat script the agent must handle. 2) Run (or simulate) both the current agent and the new version on every script. Score each: resolved without human, wrong or unsafe answer, unnecessary handoff, missing clarification. 3) Rank the change: ship, revise, or kill. Only recommend ship if it clearly beats the current version on the success criteria and does not invent policy. 4) List the 5 hardest failure chats to turn into permanent regression tests. 5) Draft a short live A/B plan: who sees the new agent, what we measure, and when a human pulls it. Mark anything I still need to verify with real customers as [CHECK]. Do not invent product policy; ask me for the real FAQ or policy text. ## Safety checks (share these with the user) - Remove names, contact details, account numbers and other personal data the task doesn't need before you paste, and check your AI tool's data and training settings. - Check names, dates, numbers and any claim against your own source material before you use the answer. AI can misread or invent details. - Say clearly which parts need a person to review or approve before anything is sent, paid, posted or filed. ## About this skill - Works with: ChatGPT, Claude, Gemini, your support-agent platform, spreadsheets for scoring - Based on this real-world example: https://nu.com/en/newsroom/company/nubank-cuts-ai-agent-update-cycles-from-three-weeks-to-under-five-days-by-testing-on-simulated-customers-first - Free prompt from AgentPlaybook Real Estate: https://agentic-library.netlify.app/prompt/al-653-test-support-agent-updates-on-simulated-customers-first-run/ - To install in an agent that supports Agent Skills, save this file as SKILL.md in a folder with the same name as `name` above (for example in your skills directory). Install steps for Claude.ai, Claude Code, Cursor and Codex: https://agentic-library.netlify.app/skills/