Laravel Horizon's Metrics Lie Under Burst Load

Horizon's dashboard is beautiful. It's also one of the most misleading monitoring tools I've leaned on in production — and I say that as someone who genuinely likes the package. The throughput numbers it shows you are rolling averages that smooth out burst load so aggressively you can be in serious trouble before the graph twitches.

I learned this the hard way on an e-commerce client last Black Friday. Queue depth hit 14,000 jobs. Horizon's dashboard showed a perfectly reasonable throughput number the entire time because the burst happened fast and the averages hadn't caught up. By the time the graphs looked bad, orders were already delayed.

What Horizon's Metrics Actually Measure

Horizon tracks throughput and wait time as rolling time-series averages stored in Redis. The default snapshot interval is every minute (Horizon::queryResultsFor(minues: 1) under the hood), and the dashboard plots them over 60-minute and 24-hour windows. That's fine for long-running, steady-state throughput analysis. It is genuinely terrible for detecting a sudden 5x spike in job production.

The wait time metric is the other one that bites people. Horizon measures wait time at job completion — it records how long a job sat in the queue before a worker picked it up. That means if jobs are piling up and none of them have finished yet, your reported wait time is still the average from the last snapshot window. It looks fine. It is not fine.

The throughput number has a similar problem. Horizon divides completed jobs by time elapsed. If your producers suddenly outrun your workers, the denominator keeps growing while completed jobs lag. The ratio can look stable for several minutes into a real incident.

The Queue Depth Is the Number You Actually Need

The single most honest signal during a burst is the raw pending job count per queue — not Horizon's processed/failed counts, just how many jobs are sitting there right now waiting to be touched. Redis gives you this directly.

Horizon does show queue size on the dashboard, but it's not front and center, it doesn't alert, and there's no built-in threshold. Here's how I pull it myself:

<?php

namespace App\Services;

use Illuminate\Support\Facades\Redis;

class QueueDepthMonitor
{
    /**
     * Returns pending job counts keyed by queue name.
     * Works for the default Redis queue driver connection.
     */
    public function depths(array $queues = ['default', 'notifications', 'exports']): array
    {
        $depths = [];

        foreach ($queues as $queue) {
            $key = 'queues:' . $queue; // Horizon prefixes with the connection name by default
            $depths[$queue] = Redis::llen($key);
        }

        return $depths;
    }

    public function isOverThreshold(array $thresholds): bool
    {
        $depths = $this->depths(array_keys($thresholds));

        foreach ($thresholds as $queue => $limit) {
            if (($depths[$queue] ?? 0) >= $limit) {
                return true;
            }
        }

        return false;
    }
}

I wire this into a scheduled command that runs every minute and fires a Slack alert if depth crosses a threshold:

<?php

namespace App\Console\Commands;

use App\Services\QueueDepthMonitor;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Http;

class CheckQueueDepth extends Command
{
    protected $signature = 'queues:check-depth';
    protected $description = 'Alert if any queue depth exceeds configured thresholds';

    public function handle(QueueDepthMonitor $monitor): int
    {
        $thresholds = [
            'default'       => 500,
            'notifications' => 1000,
            'exports'       => 200,
        ];

        if (! $monitor->isOverThreshold($thresholds)) {
            return self::SUCCESS;
        }

        $depths = $monitor->depths(array_keys($thresholds));
        $lines  = [];

        foreach ($depths as $queue => $depth) {
            $flag    = $depth >= ($thresholds[$queue] ?? PHP_INT_MAX) ? ' :rotating_light:' : '';
            $lines[] = "`{$queue}`: {$depth} jobs{$flag}";
        }

        Http::post(config('services.slack.webhook_url'), [
            'text' => "*Queue depth alert* on " . config('app.name') . "\n" . implode("\n", $lines),
        ]);

        return self::SUCCESS;
    }
}

Register it in routes/console.php or the scheduler:

Schedule::command('queues:check-depth')->everyMinute()->withoutOverlapping();

Not glamorous, but it's caught two incidents before they became customer-visible in the past year.

The Supervisor Configuration Horizon Hides From You

Another thing Horizon quietly obscures: the balance strategy interacts with burst load in ways that aren't obvious from the dashboard.

The default auto balance mode adjusts worker counts toward a target queue balance, but it does so gradually. If you have minProcesses: 1 and maxProcesses: 10, Horizon won't immediately spin up 10 workers the second a queue floods. It adds one worker at a time per balance cycle. At moderate burst rates that's fine. At high burst rates — say, a webhook processor getting hammered — you can be three or four balance cycles behind before worker count catches up.

Setting balanceCooldown low (it defaults to 3 seconds) helps, but only within the process range you've defined. If maxProcesses is too conservative, Horizon just... stops adding workers and doesn't tell you loudly.

In config/horizon.php, I'm usually explicit about this for anything latency-sensitive:

'supervisor-webhooks' => [
    'connection' => 'redis',
    'queue'      => ['webhooks'],
    'balance'    => 'auto',
    'minProcesses' => 2,   // never drop below 2 — cold start latency is real
    'maxProcesses' => 20,  // give yourself enough headroom for a real burst
    'balanceCooldown' => 1,
    'tries'      => 3,
    'timeout'    => 60,
],

The minProcesses floor matters more than people think. If Horizon scales you down to 1 worker overnight and a cron fires 800 jobs at 7am, that single worker is doing triage while the balance algorithm slowly wakes up.

The horizon:snapshot Command and Its Frequency

Horizon stores its metrics via php artisan horizon:snapshot, which you're expected to schedule. The docs suggest every five minutes. I've seen teams copy that schedule and never question it — then wonder why their Horizon graphs are too coarse to diagnose anything.

Five minutes is fine for a 24-hour trend view. It's nearly useless for a burst that lasts eight minutes. Run it every minute:

Schedule::command('horizon:snapshot')->everyMinute();

This does add minor Redis write overhead, but on anything running Horizon you've already accepted that Redis is load-bearing infrastructure. The overhead is negligible compared to the diagnostic value.

When I'd Reach for Horizon (and When I Wouldn't)

Horizon is the right default for any Laravel app on Redis queues. The job tagging, the retry UI, the supervisor management — all genuinely useful for day-to-day operations. I'm not arguing against it.

What I am arguing against is using its dashboard as your primary incident detection tool. The metrics are built for observability after the fact, not alerting in the moment. Once you understand that, it fits.

For teams doing significant queue volume — multiple supervisors, mixed queue priorities, jobs with different SLAs — I'd layer in something external. Even a simple Prometheus Redis exporter scraping redis.llen on your queue keys and alerting in Grafana gives you a real-time depth signal that Horizon can't. For healthcare clients especially, where a delayed lab result notification is a compliance conversation, I won't rely on Horizon alone.

For a small app with one queue and light load, the dashboard is fine and you're probably not going to have a burst problem worth worrying about. Don't over-engineer it.

The Redis Key Naming Gotcha

One practical thing that trips people up: the Redis key Horizon uses for a queue depends on your connection name and whether you've set a prefix. By default it's queues:{name}, but if you have a Redis prefix configured in config/database.php under the redis connection options, the actual key in Redis is {prefix}:queues:{name}.

If your llen calls are returning 0 when you know there are jobs queued, check your prefix:

// In tinker, sanity-check what keys actually exist
Redis::keys('*queues*');

That'll show you the real key names. I've burned a half-hour on this twice — once on a client's staging environment that had a prefix set in .env that production didn't, so the monitoring script worked in staging and silently returned wrong numbers in prod.

Closing

Horizon is a solid package and I keep using it. But the dashboard is a rearview mirror, not a windshield. Build the raw depth check, run horizon:snapshot every minute, and set minProcesses high enough that a burst doesn't catch you cold. Trust the queue length. Don't trust the pretty graphs under pressure.