Laravel scheduler failures
Posted inCode Examples & Resources Database Development Tools & Platforms Laravel Programming & Development Recent Updates Software Engineering Fundamentals Tutorials & Guides

Our Cron Jobs Were Quietly Failing for Months — Here’s Why

Posted inCode Examples & Resources, Database, Development Tools & Platforms, Laravel, Programming & Development, Recent Updates, Software Engineering Fundamentals, Tutorials & Guides

Laravel scheduler failures

“Why haven’t we sent any invoice reminders in three months?”

The question came from our finance team during a casual Slack conversation. My stomach dropped. Invoice reminders were automated. They ran every night at 2 AM. I set them up myself six months ago.

I pulled up the Laravel Scheduler logs. Nothing. No errors. No warnings. Just… silence.

I checked our database. We had 847 invoices that should have triggered reminders. Zero reminder emails sent. For ninety-three days.

Our users weren’t getting password reset emails. Daily reports weren’t being generated. Database backups hadn’t run in weeks. And we had no idea because nothing was alerting us.

The worst part? Everything looked fine. The cron was running. The scheduler was configured. The jobs existed in our code. But somewhere between the Linux cron daemon and our Laravel application, things were failing silently.

Here’s how one innocent redirect symbol—> /dev/null 2>&1—and a series of bad assumptions led to months of silent failures.

The Setup: Standard Laravel Scheduler

Our crontab looked like every Laravel tutorial tells you to set it up:

* * * * * cd /var/www/html && php artisan schedule:run >> /dev/null 2>&1

This runs every minute and invokes Laravel’s scheduler, which then decides which jobs to run based on your schedule. Textbook implementation.

Our scheduler in app/Console/Kernel.php looked equally normal:

<?php

namespace App\Console;

use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;

class Kernel extends ConsoleKernel
{
    protected function schedule(Schedule $schedule)
    {
        // Invoice reminders every night at 2 AM
        $schedule->command('invoices:send-reminders')
                 ->dailyAt('02:00');

        // Daily reports at 3 AM
        $schedule->command('reports:generate-daily')
                 ->dailyAt('03:00');

        // Database backup every 6 hours
        $schedule->command('backup:run')
                 ->everySixHours();

        // Clean old sessions daily
        $schedule->command('session:clean')
                 ->daily();

        // Update search indexes every 15 minutes
        $schedule->command('scout:import')
                 ->everyFifteenMinutes();
    }
}

Clean. Organized. Following best practices. And completely failing in production.

The Symptoms: The Silence Was Deafening

The problem with silent failures is they’re silent. We had no idea anything was wrong until someone asked a question.

Here’s what we eventually discovered wasn’t working:

  1. Invoice reminders: 0 sent in 93 days (should be ~2,400)
  2. Daily reports: Missing for 67 days
  3. Database backups: Last successful backup was 41 days ago
  4. Session cleanup: Database had 340,000 old sessions (should be ~5,000)
  5. Search index updates: Elasticsearch was 3 weeks behind

But our monitoring showed green lights. Our APM dashboard showed no errors. Sentry had zero cron-related exceptions.

Everything was “working.”

The Investigation: Following the Trail of Nothing

I started by manually running the scheduler:

cd /var/www/html
php artisan schedule:run

Running scheduled command: Closure at /var/www/html/app/Console/Kernel.php:50

It ran. But nothing happened. No errors. No output. Just… nothing.

I tried running the actual command directly:

php artisan invoices:send-reminders

And there it was:

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted 
(tried to allocate 20480 bytes) in /var/www/html/vendor/laravel/
framework/src/Illuminate/Database/Query/Builder.php on line 2345

The command was crashing. But when run through the scheduler via cron, this error disappeared into the void.

The Root Cause #1: Output Redirection

Let’s look at that crontab again:

* * * * * cd /var/www/html && php artisan schedule:run >> /dev/null 2>&1

That >> /dev/null 2>&1 is the problem. Let me break it down:

  • >> /dev/null – Redirect standard output to nowhere (append mode)
  • 2>&1 – Redirect standard error (2) to standard output (1)
  • Combined effect: All output disappears into the void

This is actually recommended in many Laravel tutorials. The reasoning is:

  • “Prevents cron from sending emails”
  • “Reduces noise”
  • “Keeps things clean”

But it also means:

  • No error messages
  • No debugging information
  • No way to know if things are failing

Every minute, our scheduler was running. Every minute, commands were crashing. Every minute, the errors were being thrown away.

The Root Cause #2: Laravel’s Scheduler Swallows Errors

Even without the output redirection, Laravel’s scheduler has a sneaky behavior. When a scheduled command throws an exception, the scheduler catches it and continues.

Here’s the relevant code from Laravel’s framework:

// Inside Laravel's Schedule class
public function run(Container $container)
{
    foreach ($this->dueEvents($container) as $event) {
        if (! $event->filtersPass($container)) {
            continue;
        }

        try {
            $event->run($container);
        } catch (Exception $e) {
            // Exception is caught but not re-thrown
            // Only logged if withoutOverlapping is used
        }
    }
}

The scheduler gracefully handles exceptions to prevent one failing command from stopping others. Great for resilience. Terrible for visibility.

The Root Cause #3: Memory Limits We Never Set

Our invoice reminder command looked like this:

<?php

namespace App\Console\Commands;

use App\Models\Invoice;
use App\Mail\InvoiceReminder;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Mail;

class SendInvoiceReminders extends Command
{
    protected $signature = 'invoices:send-reminders';
    protected $description = 'Send reminders for overdue invoices';

    public function handle()
    {
        // This was the problem
        $invoices = Invoice::where('status', 'pending')
                          ->where('due_date', '<', now())
                          ->get(); // Loading ALL overdue invoices into memory

        foreach ($invoices as $invoice) {
            Mail::to($invoice->customer->email)
                ->send(new InvoiceReminder($invoice));
        }

        $this->info("Sent {$invoices->count()} reminder emails");
    }
}

We had 847 overdue invoices. Each invoice object loaded its relationships. The entire collection was eating 150MB of memory.

PHP’s default memory limit in our cron environment? 128MB.

Crash. Silent crash.

The Root Cause #4: Wrong User, Wrong Environment

This one was subtle. Our crontab was installed under the www-data user:

sudo crontab -u www-data -e

But our application used an .env file with specific paths and configurations. The cron environment didn’t have:

  • The correct APP_ENV variable
  • Database connection details from .env
  • Proper file permissions for log writes
  • Environment-specific configurations

Commands were running in a broken environment, failing, and we never knew.

The Root Cause #5: Log Rotation Killed Our Evidence

We had logrotate configured for Laravel logs:

# /etc/logrotate.d/laravel
/var/www/html/storage/logs/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 644 www-data www-data
}

Seems reasonable. Keep logs for 7 days, compress old ones.

But here’s what happened:

  1. Command fails and writes to laravel.log
  2. We don’t notice for a few days
  3. Log gets rotated and compressed
  4. Eventually deleted after 7 days
  5. Evidence is gone

By the time we started investigating, the logs from when it first broke were long gone.

The Fix: A Multi-Layered Approach

I fixed this in five stages, each addressing a different failure point.

Fix #1: Proper Output Logging

Changed the crontab to actually log output:

# Create logs directory if it doesn't exist
mkdir -p /var/www/html/storage/logs/cron

# New crontab
* * * * * cd /var/www/html && php artisan schedule:run >> /var/www/html/storage/logs/cron/scheduler.log 2>&1

Now we could actually see what was happening:

tail -f /var/www/html/storage/logs/cron/scheduler.log

[2024-01-15 02:00:01] Running scheduled command: php artisan invoices:send-reminders
[2024-01-15 02:00:03] PHP Fatal error: Allowed memory size...

But this created a new problem—the log file grew infinitely. So I added a wrapper script:

#!/bin/bash
# /var/www/html/bin/run-scheduler.sh

TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
LOG_FILE="/var/www/html/storage/logs/cron/scheduler-$(date '+%Y-%m-%d').log"

echo "[$TIMESTAMP] Starting scheduler run" >> "$LOG_FILE"

cd /var/www/html
php artisan schedule:run >> "$LOG_FILE" 2>&1

echo "[$TIMESTAMP] Scheduler run completed" >> "$LOG_FILE"

Updated crontab:

* * * * * /var/www/html/bin/run-scheduler.sh

This created daily log files that were easier to manage.

Fix #2: Memory-Efficient Command Processing

Rewrote the invoice reminder command to use chunks:

<?php

namespace App\Console\Commands;

use App\Models\Invoice;
use App\Mail\InvoiceReminder;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Mail;

class SendInvoiceReminders extends Command
{
    protected $signature = 'invoices:send-reminders';
    protected $description = 'Send reminders for overdue invoices';

    public function handle()
    {
        $count = 0;

        // Process in chunks of 100 instead of loading all at once
        Invoice::where('status', 'pending')
              ->where('due_date', '<', now())
              ->whereNull('reminder_sent_at')
              ->chunk(100, function ($invoices) use (&$count) {
                  foreach ($invoices as $invoice) {
                      try {
                          Mail::to($invoice->customer->email)
                              ->send(new InvoiceReminder($invoice));

                          $invoice->update(['reminder_sent_at' => now()]);
                          $count++;

                      } catch (\Exception $e) {
                          $this->error("Failed to send reminder for invoice {$invoice->id}: {$e->getMessage()}");
                          report($e); // Send to Sentry
                      }
                  }

                  // Force garbage collection after each chunk
                  gc_collect_cycles();
              });

        $this->info("Sent {$count} reminder emails");
        
        return 0; // Explicit success exit code
    }
}

Key changes:

  • Use chunk() to process in batches
  • Try-catch around each individual send
  • Track which invoices got reminders
  • Explicit error logging
  • Force garbage collection
  • Return proper exit codes

Fix #3: Scheduler Error Handling

Added proper error handling to the scheduler:

<?php

namespace App\Console;

use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;
use Illuminate\Support\Facades\Log;

class Kernel extends ConsoleKernel
{
    protected function schedule(Schedule $schedule)
    {
        // Invoice reminders with error handling
        $schedule->command('invoices:send-reminders')
                 ->dailyAt('02:00')
                 ->onFailure(function () {
                     Log::error('Invoice reminders command failed');
                     $this->notifySlack('Invoice reminders failed!');
                 })
                 ->onSuccess(function () {
                     Log::info('Invoice reminders completed successfully');
                 });

        // Daily reports with timeout
        $schedule->command('reports:generate-daily')
                 ->dailyAt('03:00')
                 ->timeout(300) // 5 minutes max
                 ->onFailure(function () {
                     $this->notifySlack('Daily reports failed!');
                 });

        // Database backup with monitoring
        $schedule->command('backup:run')
                 ->everySixHours()
                 ->withoutOverlapping()
                 ->onFailure(function () {
                     $this->notifySlack('Database backup failed!');
                 })
                 ->sendOutputTo('/var/www/html/storage/logs/backups.log');

        // Session cleanup - less critical
        $schedule->command('session:clean')
                 ->daily()
                 ->runInBackground();

        // Search index with error handling
        $schedule->command('scout:import')
                 ->everyFifteenMinutes()
                 ->withoutOverlapping()
                 ->onFailure(function () {
                     Log::warning('Search index update failed');
                 });
    }

    protected function notifySlack($message)
    {
        // Slack webhook notification
        \Illuminate\Support\Facades\Http::post(config('services.slack.webhook'), [
            'text' => $message,
            'channel' => '#alerts',
            'username' => 'Cron Monitor',
            'icon_emoji' => ':rotating_light:'
        ]);
    }
}

Now every critical command has:

  • onFailure() hooks for alerting
  • onSuccess() hooks for logging
  • Timeout values to prevent hanging
  • Output capture for debugging
  • Overlap prevention for long-running jobs

Fix #4: Environment Variables in Cron

Cron doesn’t inherit your shell environment. Fixed by explicitly sourcing the environment:

# /var/www/html/bin/run-scheduler.sh

#!/bin/bash

# Load environment variables
export $(grep -v '^#' /var/www/html/.env | xargs)

# Or if using PHP dotenv
cd /var/www/html

# Set proper timezone
export TZ='UTC'

# Ensure correct PHP path
PHP_BIN=$(which php)

TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
LOG_FILE="/var/www/html/storage/logs/cron/scheduler-$(date '+%Y-%m-%d').log"

echo "[$TIMESTAMP] Starting scheduler run" >> "$LOG_FILE"
echo "[$TIMESTAMP] Environment: $APP_ENV" >> "$LOG_FILE"
echo "[$TIMESTAMP] PHP Version: $($PHP_BIN -v | head -n 1)" >> "$LOG_FILE"

$PHP_BIN artisan schedule:run >> "$LOG_FILE" 2>&1

EXIT_CODE=$?
echo "[$TIMESTAMP] Scheduler run completed with exit code: $EXIT_CODE" >> "$LOG_FILE"

exit $EXIT_CODE

This ensures:

  • Environment variables are loaded
  • PHP version is logged
  • Exit codes are captured
  • All output is preserved

Fix #5: Monitoring and Alerting

Created a health check command that runs before scheduled tasks:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Cache;

class SchedulerHealthCheck extends Command
{
    protected $signature = 'scheduler:health-check';
    protected $description = 'Check if scheduler is running properly';

    public function handle()
    {
        try {
            // Test database connection
            DB::connection()->getPdo();
            
            // Test cache connection
            Cache::set('scheduler_health_check', now(), 60);
            
            // Update last run timestamp
            Cache::set('scheduler_last_run', now(), 600);
            
            $this->info('Health check passed');
            return 0;
            
        } catch (\Exception $e) {
            $this->error('Health check failed: ' . $e->getMessage());
            report($e);
            return 1;
        }
    }
}

Added to scheduler:

$schedule->command('scheduler:health-check')
         ->everyMinute()
         ->onFailure(function () {
             $this->notifySlack('Scheduler health check failed!');
         });

Then created a monitoring script:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\Cache;

class MonitorScheduler extends Command
{
    protected $signature = 'scheduler:monitor';
    protected $description = 'Monitor if scheduler has run recently';

    public function handle()
    {
        $lastRun = Cache::get('scheduler_last_run');
        
        if (!$lastRun) {
            $this->error('Scheduler has never run!');
            $this->notifySlack('Scheduler has never run!');
            return 1;
        }
        
        $minutesSinceLastRun = now()->diffInMinutes($lastRun);
        
        if ($minutesSinceLastRun > 5) {
            $this->error("Scheduler hasn't run in {$minutesSinceLastRun} minutes!");
            $this->notifySlack("Scheduler hasn't run in {$minutesSinceLastRun} minutes!");
            return 1;
        }
        
        $this->info('Scheduler is running normally');
        return 0;
    }
    
    protected function notifySlack($message)
    {
        \Illuminate\Support\Facades\Http::post(config('services.slack.webhook'), [
            'text' => $message,
            'channel' => '#alerts'
        ]);
    }
}

Run this from an external monitoring service (like AWS CloudWatch or UptimeRobot) every 10 minutes:

# This runs from outside the scheduler to monitor it
*/10 * * * * cd /var/www/html && php artisan scheduler:monitor

The Supervisor Addition

We also added Supervisor to ensure the cron wrapper script stays running and restarts if it crashes:

# /etc/supervisor/conf.d/laravel-scheduler.conf<br>process_name=%(program_name)s command=/var/www/html/bin/run-scheduler-loop.sh autostart=true autorestart=true stopasgroup=true killasgroup=true user=www-data numprocs=1 redirect_stderr=true stdout_logfile=/var/www/html/storage/logs/supervisor/scheduler.log stopwaitsecs=60

And the loop script:

#!/bin/bash
# /var/www/html/bin/run-scheduler-loop.sh

while true; do
    php /var/www/html/artisan schedule:run
    sleep 60
done

This is actually more reliable than cron because:

  • Supervisor restarts it if it crashes
  • Better logging integration
  • Easier to monitor process health
  • No dependency on cron daemon

The Results: From Silence to Visibility

Week 1 after deployment:

  • Caught 3 failing commands within hours
  • Fixed memory issues in 2 commands
  • Discovered 1 command with bad database query
  • Got our first successful backup in 6 weeks

Month 1:

  • Zero silent failures
  • 847 overdue invoice reminders finally sent
  • Daily reports back on track
  • Search indexes fully updated
  • Database backups running reliably

Slack alerts received:

  • Week 1: 47 alerts (as we fixed issues)
  • Week 2: 12 alerts
  • Week 3: 3 alerts
  • Week 4+: 0-1 alerts per week

The real win:

  • We knew immediately when something failed
  • Debugging took minutes instead of days
  • No more “why hasn’t X happened?” questions
  • Sleep better at night

The Lessons Learned

1. Never Redirect to /dev/null in Production

That > /dev/null 2>&1 is useful in development to reduce noise. In production, it’s digital duct tape over your smoke detector.

Always capture output. Disk space is cheap. Lost data isn’t.

2. Scheduler Callbacks Are Essential

Laravel’s onFailure() and onSuccess() callbacks aren’t optional nice-to-haves. They’re critical monitoring infrastructure.

Use them. On every scheduled command.

3. Memory Limits Matter in CLI

Your web requests might handle 128MB fine. Your CLI commands that process thousands of records? Not so much.

Always use chunking for bulk operations. Always.

4. External Monitoring Is Required

You can’t trust a system to monitor itself. Our scheduler was failing, so scheduler-based monitoring would have failed too.

Use external health checks. Paid monitoring services. Anything outside your application.

5. Exit Codes Tell the Truth

Laravel commands should return proper exit codes:

  • 0 for success
  • 1 for failure

And you should check those codes:

public function handle()
{
    try {
        // Command logic
        return 0; // Success
    } catch (\Exception $e) {
        $this->error($e->getMessage());
        return 1; // Failure
    }
}

6. Log Rotation Needs Different Rules

Application logs and cron logs need different retention:

# /etc/logrotate.d/laravel-cron
/var/www/html/storage/logs/cron/*.log {
    daily
    rotate 30        # Keep 30 days, not 7
    compress
    delaycompress
    missingok
    notifempty
    create 644 www-data www-data
}

Keep cron logs longer. You’ll thank yourself during investigations.

The Debugging Checklist

Before deploying any scheduled command, verify:

□ Command has proper error handling
□ Uses chunking for large datasets
□ Has explicit return codes (0 = success, 1 = fail)
□ Includes onFailure() callback
□ Includes onSuccess() callback for critical commands
□ Has reasonable timeout values
□ Logs meaningful messages
□ Handles exceptions gracefully
□ Tested with realistic data volumes
□ Reviewed memory usage under load

For cron setup:

□ Output is captured to a log file
□ Environment variables are loaded correctly
□ Running as correct user with proper permissions
□ Log rotation configured appropriately
□ External monitoring in place
□ Slack/email alerts configured
□ Tested manually before scheduling
□ Documented in runbook

The Scripts We Now Use

Quick Debug Script

#!/bin/bash
# /var/www/html/bin/debug-scheduler.sh

echo "=== Scheduler Debug Information ==="
echo "Current time: $(date)"
echo "Current user: $(whoami)"
echo "Working directory: $(pwd)"
echo ""

echo "=== Environment ==="
grep -E '^(APP_ENV|DB_|REDIS_)' .env
echo ""

echo "=== PHP Version ==="
php -v
echo ""

echo "=== Recent Scheduler Runs ==="
tail -20 storage/logs/cron/scheduler-$(date +%Y-%m-%d).log
echo ""

echo "=== Failed Commands (last 24h) ==="
find storage/logs/cron -name "*.log" -mtime -1 -exec grep -l "Fatal error\|Exception" {} \;
echo ""

echo "=== Disk Space ==="
df -h /var/www/html/storage
echo ""

echo "=== Running schedule:list ==="
php artisan schedule:list

Manual Test Runner

#!/bin/bash
# /var/www/html/bin/test-scheduler.sh

COMMAND=$1

if [ -z "$COMMAND" ]; then
    echo "Usage: ./test-scheduler.sh 'command:name'"
    exit 1
fi

echo "Testing command: $COMMAND"
echo "Started: $(date)"
echo "---"

cd /var/www/html
php artisan $COMMAND --verbose

EXIT_CODE=$?

echo "---"
echo "Finished: $(date)"
echo "Exit code: $EXIT_CODE"

if [ $EXIT_CODE -eq 0 ]; then
    echo "✓ Command succeeded"
else
    echo "✗ Command failed"
fi

Looking Back

Three months of silent failures. Hundreds of missed invoices. Dozens of missing reports. All because we trusted that “working” meant “working correctly.”

The fix wasn’t complicated. It wasn’t even time-consuming—maybe a week of work total. But the impact was massive.

We went from hoping things worked to knowing they worked. From discovering failures weeks later to being alerted within minutes. From “maybe we should check the logs?” to “here’s exactly what failed and why.”

The scariest part? This is probably happening right now in thousands of applications. Cron jobs failing silently. Developers trusting that > /dev/null 2>&1 is fine. Teams assuming their scheduler is working because they set it up once and never looked again.

Check your cron logs. Right now. I’ll wait.

Found anything interesting? Yeah, I thought so.

Leave a Comment

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *