Model or Scaffolding? Classifying AI Agent Failure Modes

Model or scaffolding?

If you are developing AI agents for production, this article is worth saving.

The authors collected 41 typical failure scenarios and classified them by where, at the junction of which components, the problem arises. Each error is tied to the interaction between the model, the scaffolding, the user, the tools, the memory, or the environment, and also indicates on whose side the cause lies and where it needs to be fixed.

This accurately reflects the real picture: most AI agent problems arise not in the model itself, but at the junction of the model and the surrounding infrastructure.

The approach is also suitable for automatic analysis. In tests on four frontier models, the best AI judge achieved a Cohen’s kappa of 0.76 compared to manual annotation. This makes it possible to automatically classify errors directly in production, rather than sorting them out manually after each incident.

This year, scaffolding engineering has become one of the main factors in AI agent quality, but there has still been no common language to distinguish model errors from scaffolding errors. This work proposes exactly such a system.

Article: https://arxiv.org/abs/2607.28802