Idempotency keys for webhooks: the ten lines that stop double charges

Every webhook provider worth using promises at-least-once delivery. Read that phrase slowly: it means “sometimes more than once.” A provider retries when your endpoint times out, when its own worker crashes after sending but before recording the 200, and occasionally during a regional failover for reasons nobody will ever explain to you.

Last spring our billing service received the same invoice.paid event from a payment provider, Corvel Payments, twice within 400 milliseconds. Both requests passed signature verification. Both credited the customer’s account. Support found it three days later when the customer asked, politely, why they had twice the credits they paid for.

The fix is small. Getting it right under concurrency is the interesting part.

Key on the event ID, not the payload

Most providers send a stable event ID with every delivery, and the retries reuse it. That ID is your idempotency key. Do not hash the payload instead: some providers add a delivery timestamp or attempt counter to the body, so two deliveries of the same event can hash differently.

We store processed IDs in a table with a primary key, which gives us a unique index for free:

CREATE TABLE processed_events (
  event_id     text PRIMARY KEY,
  event_type   text NOT NULL,
  received_at  timestamptz NOT NULL DEFAULT now()
);

-- keep 30 days; providers stop retrying long before that
DELETE FROM processed_events WHERE received_at < now() - interval '30 days';

Record and apply in one transaction

The rule: the insert into processed_events and the side effects of the event must commit together, or not at all. If you record the ID first and crash before applying the event, you have silently dropped it. If you apply first and record later, a duplicate can slip in between.

// Pool and PoolClient are the types from the pg package

type WebhookEvent = { id: string; type: string; data: unknown };

export async function handleWebhook(
  pool: Pool,
  event: WebhookEvent,
  apply: (db: PoolClient, event: WebhookEvent) => Promise<void>,
): Promise<'processed' | 'duplicate'> {
  const db = await pool.connect();
  try {
    await db.query('BEGIN');
    const res = await db.query(
      `INSERT INTO processed_events (event_id, event_type)
       VALUES ($1, $2)
       ON CONFLICT (event_id) DO NOTHING`,
      [event.id, event.type],
    );
    if (res.rowCount === 0) {
      await db.query('ROLLBACK');
      return 'duplicate';
    }
    await apply(db, event);
    await db.query('COMMIT');
    return 'processed';
  } catch (err) {
    await db.query('ROLLBACK');
    throw err;
  } finally {
    db.release();
  }
}

Both outcomes return HTTP 200 to the provider. A duplicate is not an error from their point of view, and answering with a 4xx just earns you more retries.

What happens when two deliveries race

This is the part that makes the pattern safe, and it comes from Postgres rather than from our code. When two transactions insert the same primary key, the second insert does not fail immediately. It waits on the first transaction’s uncommitted row.

• If the first transaction commits, the second insert resolves the conflict, DO NOTHING kicks in, and rowCount is zero. We report a duplicate.
• If the first transaction rolls back, because apply threw, the second insert succeeds and processes the event properly.

No advisory locks, no SELECT before the INSERT, no Redis. The unique index is the lock, and it releases at exactly the right moment.

Things we got wrong the first time

Our first version checked for the ID with a SELECT, then inserted it. Under load, both racing requests saw no row, both inserted, and one failed with a unique violation after the side effects had already run in a separate connection. Checking and claiming must be one statement.

We also returned a 409 for duplicates. The provider treated that as a failure and retried for six hours, filling our logs with the same event ID. Answer 200, log the duplicate at debug level, and move on.

Finally, we forgot that our own retry job could replay events from the dead-letter queue. It goes through the same handler now, so it gets the same protection. Since the change we have counted 212 duplicate deliveries in five months, and zero double credits.