Reliable Delivery & Retries

Install a job queue library

npm install bull redis

We'll use Bull with Redis as the backing store. Make sure Redis is running:

docker run -d -p 6379:6379 redis:7

Create a queue service

Create src/queue/deliveryQueue.ts:

import Queue from 'bull';

export interface DeliveryJob {
  delivery_id: string;
  webhook_id: string;
  event_id: string;
  webhook_url: string;
  webhook_secret: string;
  payload: any;
  attempt: number;
}

const deliveryQueue = new Queue('delivery', {
  redis: {
    host: process.env.REDIS_HOST || 'localhost',
    port: parseInt(process.env.REDIS_PORT || '6379')
  }
});

export default deliveryQueue;

Update POST /events to enqueue delivery jobs

Modify src/routes/events.ts:

import deliveryQueue from '../queue/deliveryQueue';

// In the POST /events handler, after inserting the event:
const eventRow = result.rows[0];

// Find all matching webhooks
const webhooksResult = await pool.query(
  'SELECT * FROM webhooks WHERE $1 = ANY(event_types)',
  [type]
);

// Create delivery records and enqueue jobs
for (const webhook of webhooksResult.rows) {
  const deliveryId = crypto.randomUUID();
  await pool.query(
    'INSERT INTO deliveries (id, webhook_id, event_id, status) VALUES ($1, $2, $3, $4)',
    [deliveryId, webhook.id, eventRow.id, 'pending']
  );

  await deliveryQueue.add({
    delivery_id: deliveryId,
    webhook_id: webhook.id,
    event_id: eventRow.id,
    webhook_url: webhook.url,
    webhook_secret: webhook.secret,
    payload: payload,
    attempt: 0
  });
}

res.status(201).json(eventRow);

Implement a worker to process deliveries

Create src/worker/deliveryWorker.ts:

import crypto from 'crypto';
import axios from 'axios';
import pool from '../db/pool';
import deliveryQueue, { DeliveryJob } from '../queue/deliveryQueue';

const BACKOFF_DELAYS = [1000, 4000, 16000, 64000, 256000]; // ms
const MAX_RETRIES = 5;

function calculateNextRetry(attempt: number): number {
  if (attempt >= MAX_RETRIES) return -1;
  return Date.now() + BACKOFF_DELAYS[attempt];
}

function signPayload(payload: any, secret: string): string {
  const body = JSON.stringify(payload);
  return crypto
    .createHmac('sha256', secret)
    .update(body)
    .digest('hex');
}

deliveryQueue.process(async (job) => {
  const { delivery_id, webhook_url, webhook_secret, payload, attempt } = job.data;

  try {
    // Update status to processing
    await pool.query(
      'UPDATE deliveries SET status = $1 WHERE id = $2',
      ['processing', delivery_id]
    );

    // Sign and send
    const signature = signPayload(payload, webhook_secret);
    const response = await axios.post(webhook_url, payload, {
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Signature': `sha256=${signature}`,
        'X-Delivery-ID': delivery_id
      },
      timeout: 10000
    });

    if (response.status >= 200 && response.status < 300) {
      // Success
      await pool.query(
        'UPDATE deliveries SET status = $1, attempts = $2, updated_at = NOW() WHERE id = $3',
        ['success', attempt + 1, delivery_id]
      );
      return { success: true };
    } else {
      throw new Error(`HTTP ${response.status}`);
    }
  } catch (error) {
    const errorMsg = error instanceof Error ? error.message : String(error);
    const nextRetryAt = calculateNextRetry(attempt);

    if (nextRetryAt === -1) {
      // Permanent failure - will move to DLQ in Module 04
      await pool.query(
        'UPDATE deliveries SET status = $1, attempts = $2, last_error = $3, updated_at = NOW() WHERE id = $4',
        ['failed', attempt + 1, errorMsg, delivery_id]
      );
    } else {
      // Retry scheduled
      await pool.query(
        'UPDATE deliveries SET status = $1, attempts = $2, last_error = $3, next_retry_at = $4, updated_at = NOW() WHERE id = $5',
        ['pending', attempt + 1, errorMsg, new Date(nextRetryAt), delivery_id]
      );

      // Re-enqueue with delay
      throw job.queue.add(job.data, {
        delay: BACKOFF_DELAYS[attempt],
        attempts: 1
      });
    }
  }
});

export default deliveryQueue;

Install axios: npm install axios

Start the worker as a separate process

Create src/worker.ts:

import './worker/deliveryWorker';

console.log('Webhook delivery worker started');

Update package.json scripts:

"scripts": {
  "dev": "ts-node src/index.ts",
  "worker": "ts-node src/worker.ts",
  ...
}

Now you can run the server and worker separately:

npm run dev        # Terminal 1
npm run worker     # Terminal 2

Test retry behavior

Create a test webhook endpoint locally:

// In a separate file or terminal, mock a failing webhook:
// This can be a simple Node server that returns 500 for testing
const express = require('express');
const app = express();
app.post('/webhook', (req, res) => {
  console.log('Received webhook:', req.body);
  res.status(500).send('Temporary failure'); // Will retry
});
app.listen(4000);

Register this webhook and emit an event. Watch the worker retry with exponential backoff.

Commit your work

git add -A
git commit -m "feat: implement reliable delivery with exponential backoff

- Bull + Redis job queue for webhook delivery
- Delivery worker with exponential backoff retries
- Idempotent delivery jobs with unique delivery_id
- Proper state transitions: pending → processing → success/retry"
git push origin main