From Months to Under a Day: Rebuilding Tier-3 Escalation with Agentic AI

A recurring Tier-3 support problem, cases that sat unresolved for months, was traced to a broken operating model, more than a shortage of engineering talent. Ownership was fragmented, routing was manual, knowledge was locked in people’s heads, and every investigation started from scratch.

We redesigned the workflow end-to-end and layered in an agentic AI system that could investigate code, propose fixes, test them, and push to production behind a human approval gate. Mean time to resolution (MTTR) dropped from months to under one day.

Executive summary

A recurring Tier-3 support problem (cases that sat unresolved for months) was traced to a broken operating model, more than a shortage of engineering talent. Ownership was fragmented, routing was manual, knowledge was locked in people’s heads, and every investigation started from scratch.

We redesigned the workflow end-to-end and layered in an agentic AI system that could investigate code, propose fixes, test them, and push to production behind a human approval gate. Mean time to resolution (MTTR) dropped from months to under one day.

1. The problem

Tier-3 cases (the hard ones that survived Tier-1 and Tier-2) were piling up in a queue nobody owned. Some had been open for months. Engineers would pick one up, spend hours reconstructing context, get pulled away, and the case would sink back into the backlog. Meanwhile customers waited, and the same underlying issues resurfaced under new ticket numbers.

The symptoms were obvious. The causes were not.

2. Diagnosis

We ran a structured diagnostic across the escalation lifecycle and found five compounding failures:

The core insight: the bottleneck was the absence of a system that accumulated context and acted on it, not diagnosis difficulty.

3. Approach

We rebuilt the workflow in two layers: a process redesign and an agentic AI platform on top of it.

3.1 Workflow redesign

• Single owner per case. Every Tier-3 case got a named owner and a backup. No orphaned tickets.
• Skill- and code-aware routing. Cases were routed by service, component, and required expertise, not by round-robin.
• Escalation controls. Aging thresholds, automatic nudges, and a hard rule: anything open past X days triggers a review.
• Measurable SLAs. We instrumented time-in-stage so we could see exactly where cases stalled.

3.2 Building the support agents

We built a set of cooperating agents rather than one monolithic AI support bot. Each agent had a narrow job and a clear contract. The system was implemented as a distributed multi-agent architecture:

• A central Orchestrator Controller (written in Go, running as a StatefulSet on Amazon EKS) managed the overall workflow state using Temporal for durable execution and long-running investigations.
• Individual agents ran as independent Kubernetes Deployments (one Deployment per agent type) with Horizontal Pod Autoscalers. Agents communicated via an internal gRPC mesh and shared a Redis-backed short-term context store plus a vector database (Pinecone) for long-term retrieval.
• Each agent was given a restricted tool surface and explicit input/output schemas so failures remained isolated.

Agent roles:

• Intake & Triage Agent: parsed incoming cases, extracted symptoms, affected services, error signatures, and customer impact.
• Grouping / Deduplication Agent: clustered cases by semantic similarity and shared root cause, so ten tickets became one investigation.
• Diagnostic Agent: pulled logs, traces, metrics, and recent deploys to form a hypothesis.
• Code Investigation Agent: located the relevant repositories, files, and code paths where the issue was manifesting.
• Fix Proposal Agent: suggested code changes, config updates, or rollbacks.
• Test & Verification Agent: built and ran the solution in a sandboxed environment.
• Deployment Agent: prepared the change for production behind a human approval gate.

3.3 Case grouping

Before any investigation started, the grouping agent deduplicated and clustered cases. This alone cut the effective workload dramatically. Many “new” Tier-3 cases were the same root cause wearing a different ticket number. Grouping gave us one investigation per root cause, not one per ticket.

3.4 Knowledge retrieval

We wired the agents into a retrieval layer over:

• Past resolved cases and their fixes
• Runbooks and architecture docs
• Code repositories and commit history
• Incident postmortems

When a new case arrived, the agent surfaced the most relevant prior resolutions and code context before a human ever opened it.

3.5 Code-aware investigation

This was the turning point. Instead of a human grepping through repos, the Code Investigation Agent:

1. Identified the service and component from the case metadata.
2. Traced the failing code path using stack traces, error signatures, and recent changes.
3. Pulled the relevant files and diffs into context.
4. Correlated with recent deploys, config changes, and dependency updates.

The agent explained why the problem occurred, and where, with citations to the exact lines and commits.

3.6 Fix proposal, build, and test

Once the root cause was localized, the Fix Proposal Agent generated candidate solutions:

• Code patches
• Configuration changes
• Dependency pinning or upgrades
• Rollback recommendations

The Test & Verification Agent then built and tested the proposed solution in an isolated environment, running unit tests, integration tests, and targeted reproductions. Only solutions that passed verification advanced.

3.7 Human-in-the-loop production push

Nothing shipped autonomously. The Deployment Agent prepared a fully-formed change (diff, test results, risk assessment, rollback plan) and presented it to a human reviewer. The human approved, modified, or rejected. On approval, the change was pushed to production through the normal CI/CD pipeline.

This preserved accountability while removing the toil. The human became a reviewer and decision-maker, not an investigator and implementer.

3.8 MCP for stakeholder communication

We used the open Model Context Protocol (MCP) standard (2024-11-05 specification) with a set of purpose-built MCP servers. The Orchestrator Controller connected to these servers over stdio and HTTP transports:

• MCP server for Jira / ServiceNow (ticket status, comments, attachments)
• MCP server for Slack and Microsoft Teams (real-time updates and approval buttons)
• MCP server for GitHub / GitLab (PRs, diffs, CI status)
• MCP server for internal status page and PagerDuty

Through these MCP connections the agents could:

• Post real-time case status updates
• Summarize investigation findings in plain language
• Request human approval with full context attached
• Notify stakeholders when a fix was deployed
• Log everything back to the case record automatically

Stakeholders stopped asking “what’s the status?” because the status updated itself.

4. Results

5. Why it worked

Three things made the difference:

1. We fixed the process before automating it. Agents amplify a good workflow and accelerate a bad one. Routing, ownership, and escalation controls came first.
2. We kept humans in the loop where it mattered. Approval gates preserved accountability without preserving toil.
3. We made context cumulative. Grouping, retrieval, and code-awareness meant the system got smarter with every case instead of starting from zero.

6. Transferable lessons

• Diagnose the operating model, beyond the technology. The months-long backlog was an ownership and routing problem before it was an AI problem.
• Group before you investigate. Deduplication is the highest-leverage step in any support workflow.
• Let agents investigate code, beyond tickets. The biggest unlock was code-aware root cause analysis.
• Automate build, test, and prep. Gate the push. Human approval on deployment is non-negotiable.
• Meet stakeholders where they are. MCP-driven updates eliminated the status-chasing overhead entirely.