The "It Works on My Machine" Era is Over

If you have built an AI agent beyond a simple "Hello World" demo, you know the pain. You run the agent, it performs a complex task perfectly. You show it to a client or deploy it, and suddenly it enters an infinite loop, hallucinates a tool argument, or simply refuses to answer.

Debugging traditional software is logic-based: If X, then Y. If it breaks, you check the logic.

Debugging AI agents is probability-based. You are debugging a system that is non-deterministic by design. The stack trace doesn't tell you why the model decided to ignore your system prompt, and a syntax error in your Python code is the least of your worries.

As an Automation Engineer, I’ve spent more time fixing agents than building them. Over time, I’ve moved away from "vibes-based debugging" (tweaking the prompt and hoping) to a systematic engineering approach.

This is my guide on how to debug agents when they inevitably break.

---

The Paradigm Shift: Observability Over Breakpoints

In standard web dev, we use breakpoints. In agent dev, breakpoints are often useless because pausing execution doesn't explain the state of the LLM's reasoning.

You need Tracing. You need to see the entire chain of thought (CoT), the raw prompt sent to the API, the tool output, and the revised context.

The Minimum Viable Logging Strategy

Before you fix a bug, you must capture the state. If you aren't using a tool like LangSmith, Phoenix, or Helicone, you need a custom logger. Here is the bare minimum structure your logs must have for every agent step:

{
  "run_id": "uuid-v4",
  "step": 3,
  "input_messages": [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "I will use the search tool..."},
    {"role": "tool", "content": "Search results: ..."}
  ],
  "llm_config": {
    "model": "gpt-4-turbo",
    "temperature": 0.7
  },
  "raw_output": "...",
  "parsed_output": "...",
  "latency_ms": 1400
}

Builder Tip: Never log just the last message. Log the accumulated context window. 90% of agent failures happen because the context window got polluted with previous errors or massive tool outputs that pushed the instructions out of focus.

---

The Agent Debugging Checklist

When an agent fails, don't just change the prompt immediately. Run through this diagnostic checklist to isolate the failure layer.

1. The Context Overflow (The "Memory Loss" Bug)

Symptom: The agent forgets instructions given at the start or ignores specific constraints.

The Fix: Check your token count. If you are retrieving documents (RAG) or getting long JSON responses from APIs, you might be truncating the system prompt (if you use a sliding window) or simply overwhelming the model's attention mechanism.

• Debug Action: Print the exact string sent to the LLM. Is the instruction actually there? Is it buried under 400 lines of XML?

2. The Tool Schema Failure (The "Fat Finger" Bug)

Symptom: The agent tries to call a tool but fails with ValidationError or passes hallucinated arguments.

The Fix: LLMs are bad at guessing proprietary ID formats. If your tool requires a User ID like usr_8923 and the agent sends 8923, that's a schema issue.

• Debug Action: strict typing with Pydantic. Don't just describe the tool in text; use strict JSON schemas. If the agent fails repeatedly, add a "retry" layer that feeds the validation error back to the agent: "You sent argument X, but it requires format Y. Try again."

3. The Reasoning Loop (The "Spinning Wheel" Bug)

Symptom: The agent calls the same tool 5 times with the same inputs, getting the same error, and refuses to stop.

The Fix: This is a lack of "mental scratchpad." The agent doesn't realize it has already tried that path.

• Debug Action: Inspect the assistant history. Does the prompt explicitly tell the agent: "If a tool fails, do not retry with the exact same arguments"? Most default system prompts lack this safety valve.

---

The "Repro Steps" Template for LLMs

Reproduction is the hardest part of AI engineering. Because temperature > 0 introduces randomness, a bug might happen 1 out of 10 times.

To debug effectively, you must eliminate variables. Use this template when logging a bug:

If you cannot reproduce the bug at Temperature 0, you are dealing with a stability issue (the prompt is too weak to guide the model consistently). If you CAN reproduce it at Temperature 0, you have a logic/instruction issue.

---

Advanced: Debugging via Evaluation (Evals)

Stop fixing bugs one by one. If you fix a prompt to handle Edge Case A, you usually break Edge Case B.

The only way to debug at scale is to build Unit Tests for Agents (Evals).

Instead of manually checking if the output is good, create a dataset of 20 inputs and their expected outcomes. When you tweak your prompt to fix a bug, run the full suite.

Code Example: Simple Assertion

def test_agent_grounding():
    response = agent.run("What is the capital of France?")
    
    # Debugging Hallucinations
    assert "Paris" in response.content
    assert len(response.tool_calls) == 0 # Should not need tools for this
    
    # Debugging Tone
    assert "I apologize" not in response.content # We want a confident agent

For more complex logic, use an LLM-as-a-Judge. Ask GPT-4 to grade the output of your smaller agent.

---

Conclusion: Embrace the Chaos

Debugging agents requires a mindset shift. You are no longer just a coder; you are a teacher and a psychologist for a digital brain.

Your code manages the flow, but your prompts manage the logic. Treat your prompts like code—version control them, test them, and debug them systematically. If you don't have visibility into the intermediate_steps of your agent, you are flying blind.

Start logging today. Your future self will thank you when the agent starts hallucinating at 2 AM.