Webhook Delivery Tutorial · Module 05 of 10

Testing Async Systems

Async systems are slippery. Test them hard: unit tests for retry logic, integration tests with fake webhook servers, and chaos tests that inject failures. By the end, you'll have confidence your system survives real-world chaos.

~5–7 hrsAdvancedTest-driven
← Back to Module 05 overview
What You'll Have at the End

Definition of Done

  • Unit tests for retry logic: backoff calculation, state transitions.
  • Integration tests: spin up Postgres + Redis in containers, run end-to-end.
  • Chaos tests: inject timeouts, 500 errors, network partitions.
  • Concurrency test: emit 1000 events to 10 webhooks; verify all 10k deliveries complete.
  • Test suite runs in under 2 minutes.
  • 75%+ code coverage for delivery worker.
The Steps

Build It

STEP 1

Install testing dependencies

npm install --save-dev jest ts-jest @types/jest testcontainers axios-mock-adapter

Create jest.config.js:

module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  testMatch: ['**/__tests__/**/*.test.ts'],
  collectCoverageFrom: ['src/**/*.ts'],
  coveragePathIgnorePatterns: ['/node_modules/', '/dist/'],
  coverageThreshold: {
    global: { lines: 75, functions: 75, branches: 70, statements: 75 }
  }
};

Update package.json:

"scripts": {
  "test": "jest",
  "test:coverage": "jest --coverage",
  ...
}
✓ Verify: npm test runs (even with no tests yet).
STEP 2

Write unit tests for retry logic

Create src/__tests__/retryLogic.test.ts:

describe('Retry Logic', () => {
  const BACKOFF_DELAYS = [1000, 4000, 16000, 64000, 256000];

  test('calculates exponential backoff correctly', () => {
    expect(BACKOFF_DELAYS[0]).toBe(1000);
    expect(BACKOFF_DELAYS[1]).toBe(4000);
    expect(BACKOFF_DELAYS[4]).toBe(256000);
  });

  test('stops retrying after 5 attempts', () => {
    let attempt = 5;
    const shouldRetry = attempt < BACKOFF_DELAYS.length;
    expect(shouldRetry).toBe(false);
  });

  test('generates increasing delays', () => {
    for (let i = 0; i < BACKOFF_DELAYS.length - 1; i++) {
      expect(BACKOFF_DELAYS[i + 1]).toBeGreaterThan(BACKOFF_DELAYS[i]);
    }
  });

  test('total retry time is under 6 minutes', () => {
    const totalTime = BACKOFF_DELAYS.reduce((a, b) => a + b, 0);
    expect(totalTime).toBeLessThan(6 * 60 * 1000);
  });
});
✓ Verify: npm test -- retryLogic.test.ts passes.
STEP 3

Write integration tests with test containers

Create src/__tests__/integration.test.ts:

import { GenericContainer, Network } from 'testcontainers';
import axios from 'axios';
import pool from '../db/pool';

describe('Webhook Delivery Integration', () => {
  let postgresContainer: any;
  let redisContainer: any;

  beforeAll(async () => {
    const network = await new Network().start();

    postgresContainer = await new GenericContainer('postgres:16')
      .withEnvironment({ POSTGRES_PASSWORD: 'test' })
      .withNetwork(network)
      .start();

    redisContainer = await new GenericContainer('redis:7')
      .withNetwork(network)
      .start();
  });

  afterAll(async () => {
    await postgresContainer.stop();
    await redisContainer.stop();
  });

  test('creates and retrieves webhook', async () => {
    // Test webhook creation
    const response = await axios.post('http://localhost:3000/webhooks', {
      url: 'https://example.com/hook',
      secret: 'secret',
      event_types: ['order.created']
    });

    expect(response.status).toBe(201);
    expect(response.data.id).toBeDefined();
  });

  test('emits event and enqueues deliveries', async () => {
    // First create a webhook
    const webhookRes = await axios.post('http://localhost:3000/webhooks', {
      url: 'https://example.com/hook',
      secret: 'secret',
      event_types: ['order.created']
    });

    // Then emit an event
    const eventRes = await axios.post('http://localhost:3000/events', {
      type: 'order.created',
      payload: { order_id: 123 }
    });

    expect(eventRes.status).toBe(201);

    // Wait briefly for queue processing
    await new Promise(r => setTimeout(r, 1000));

    // Verify deliveries were created
    const result = await pool.query(
      'SELECT COUNT(*) FROM deliveries WHERE event_id = $1',
      [eventRes.data.id]
    );
    expect(parseInt(result.rows[0].count)).toBeGreaterThan(0);
  });
});
✓ Verify: npm test -- integration.test.ts passes.
⚠ Gotcha: Test containers take time to start. Consider increasing Jest timeout: jest.setTimeout(30000).
STEP 4

Write chaos tests (failure injection)

Create src/__tests__/chaos.test.ts:

import axios from 'axios';
import MockAdapter from 'axios-mock-adapter';

describe('Chaos Tests', () => {
  const mock = new MockAdapter(axios);

  afterEach(() => {
    mock.reset();
  });

  test('retries on 500 error', async () => {
    mock.onPost(/example.com/).replyOnce(500).replyOnce(200, { ok: true });

    // Simulate sending webhook
    let attempts = 0;
    const send = async () => {
      attempts++;
      try {
        return await axios.post('https://example.com/hook', {});
      } catch (err) {
        if (attempts < 2) {
          await new Promise(r => setTimeout(r, 100));
          return send();
        }
        throw err;
      }
    };

    const result = await send();
    expect(result.status).toBe(200);
    expect(attempts).toBe(2);
  });

  test('times out and retries on slow webhook', async () => {
    mock.onPost(/example.com/).reply(() =>
      new Promise(resolve => setTimeout(resolve, 15000)) // Longer than timeout
    );

    let attempts = 0;
    const send = async () => {
      attempts++;
      try {
        return await axios.post('https://example.com/hook', {}, { timeout: 5000 });
      } catch (err) {
        if (attempts < 2) {
          await new Promise(r => setTimeout(r, 100));
          return send();
        }
        throw err;
      }
    };

    try {
      await send();
    } catch (err) {
      expect(attempts).toBeGreaterThanOrEqual(1);
    }
  });
});
✓ Verify: npm test -- chaos.test.ts passes.
STEP 5

Add concurrency test

Add to src/__tests__/integration.test.ts:

test('handles 1000 events to 10 webhooks concurrently', async () => {
  // Create 10 webhooks
  const webhooks = [];
  for (let i = 0; i < 10; i++) {
    const res = await axios.post('http://localhost:3000/webhooks', {
      url: `https://example.com/webhook${i}`,
      secret: 'secret',
      event_types: ['order.created']
    });
    webhooks.push(res.data);
  }

  // Emit 1000 events
  const events = [];
  for (let i = 0; i < 1000; i++) {
    const res = await axios.post('http://localhost:3000/events', {
      type: 'order.created',
      payload: { order_id: i }
    });
    events.push(res.data);
  }

  // Wait for worker to process
  await new Promise(r => setTimeout(r, 5000));

  // Verify all 10k deliveries exist
  const result = await pool.query(
    'SELECT COUNT(*) FROM deliveries'
  );
  expect(parseInt(result.rows[0].count)).toBeGreaterThanOrEqual(10000);
}, 60000); // 60 second timeout
✓ Verify: Test runs without crashing. All deliveries are created.
STEP 6

Commit and run coverage

git add -A
git commit -m "test: comprehensive test suite for async delivery

- Unit tests for retry logic and backoff
- Integration tests with test containers
- Chaos tests: failure injection, timeouts
- Concurrency test: 1000 events × 10 webhooks
- 75%+ code coverage threshold"
npm test -- --coverage
git push origin main
✓ Verify: npm test runs all tests in under 2 minutes. Coverage is 75%+.
Next Steps

Ready for Module 06?

You now have confidence in your delivery system. Next, you'll optimize for performance: batching, load testing, and throughput. Head to Module 06.