Skip to main content

299011 - The Agent's Queue for This Backend Is Full

Agent error code #299011 means the agent has so many changes waiting for one backend type that it stopped taking more, and this one was not attempted.

The agent applies changes for each backend type one at a time, in a queue. The queue holds well over a hundred thousand changes, far more than a login rush or an hourly resync produce, so reaching the limit means the agent has been unable to keep up for a long time. The usual cause is a device answering very slowly, or not at all, while the server keeps sending work for it.

The agent does not forget the change. It keeps the grant or revoke as something it still has to do, and applies it once the queue has room. When it lands, the grant's status on the server is replaced with the real outcome, so a grant that shows this error turns to Granted without anyone acting on it.

Steps to Resolve

Check the Device

  1. Open the agent's page in the admin portal and look at its load. The queue named there is the backend type that is behind
  2. Check that device is answering, and how long it takes. A firewall busy with a commit, or near its session or User-ID limits, can answer slowly enough to hold the queue up
  3. Look in the agent log for "backend worker queue far behind" and "backend worker queue full", which name the backend type and the operation that was running

Spread the Load

If one agent serves many large knocs on the same backend type, move some of those knocs to another agent. Each agent works through its own queue.