Webhooks: The System Design Interview Answer That Actually Runs in Production

You click "Pay" on a hosting invoice. Three seconds later, your server is activated, your dashboard updates, and a receipt lands in your inbox.

Three different systems just learned about one event. Nobody was polling. How?

Webhooks. And here is the uncomfortable truth: webhooks look like the easiest thing in distributed systems (it is just an HTTP POST!), yet most implementations quietly lose events, double-charge customers, and expose endpoints that process anything an attacker sends.

In this post, we will design a complete webhook system, both sides of it:

1. The Provider: the system that emits events (think Stripe, GitHub, or a hosting platform announcing invoice.paid)
2. The Consumer: the system that receives them without losing money

We will start naive, break it, and fix it. Code is in TypeScript/Node.js.

---

1. The 30,000-foot view

First, why webhooks at all?

Webhooks win on latency and efficiency. Polling wins on reliability. Remember this table, because the final design uses both. That is the twist ending.

Here is the full system we are about to build:

Looks like a lot? Every box exists because something terrible happens without it. Let us earn each box one failure at a time.

---

2. One rule governs everything

Before any code, burn this into memory:

Webhook delivery over the public internet is at-least-once and unordered.

Exactly-once delivery between two systems that can each crash independently does not exist. It is not hard. It is impossible.

So we split the responsibility:

• Provider promises: every event WILL be delivered. Maybe twice. Maybe out of order. But never zero times.
• Consumer promises: processing the same event twice has the same effect as processing it once (idempotency).

At-least-once delivery + idempotent consumer = effectively-once outcome. That is the whole game. Everything below is just implementation.

---

3. Provider side: the naive version (spot the bug)

// ❌ Looks fine. Ships fine. Fails on a Tuesday.
await db.invoice.update({ where: { id }, data: { status: "paid" } });
await queue.publish("invoice.paid", { invoiceId: id });

What happens if the process crashes between line 1 and line 2?

The invoice is paid in the database. The event announcing it never exists. The customer paid; the server never activates; support ticket at 2 a.m.

This is the classic dual-write problem: two systems (DB + queue), two writes, no shared transaction. One succeeds, one fails, and your data tells two different stories.

Fix: the Transactional Outbox

Do not write to two systems. Write to one (your database) atomically, and let a relay move events out later.

// ✅ State change and event share ONE transaction
await db.$transaction(async (tx) => {
  await tx.invoice.update({ where: { id }, data: { status: "paid" } });
  await tx.outbox.create({
    data: {
      eventId: crypto.randomUUID(),   // the consumer's dedupe key later
      type: "invoice.paid",
      payload: { invoiceId: id, amount, currency },
    },
  });
});

Quiz: what if the relay crashes AFTER publishing to the queue but BEFORE marking the row published?

The event gets published twice. And that is... fine! We promised at-least-once. The consumer will dedupe. What we eliminated is the unforgivable case: zero times.

---

4. Delivering the event (where the internet fights back)

A delivery worker picks up the event, finds every subscribed endpoint, and POSTs to each. Simple. Except the internet is a hostile wasteland where consumers time out, return 500s, and disappear for six hours during their own incidents.

Each delivery attempt looks like this:

import crypto from "node:crypto";

function sign(secret: string, eventId: string, timestamp: string, body: string) {
  // Sign id.timestamp.body, so a captured request cannot be replayed later
  const signedContent = `${eventId}.${timestamp}.${body}`;
  return crypto.createHmac("sha256", secret).update(signedContent).digest("hex");
}

async function attemptDelivery(sub: Subscription, event: OutboxEvent) {
  const body = JSON.stringify({ id: event.eventId, type: event.type, data: event.payload });
  const timestamp = Math.floor(Date.now() / 1000).toString();

  const res = await fetch(sub.url, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "webhook-id": event.eventId,
      "webhook-timestamp": timestamp,
      "webhook-signature": `v1=${sign(sub.secret, event.eventId, timestamp, body)}`,
    },
    body,
    signal: AbortSignal.timeout(5_000),  // slow consumer = failed consumer
  });

  if (res.status < 200 || res.status >= 300) throw new DeliveryFailed(res.status);
}

Three deliberate choices hiding in there:

1. Only 2xx is success. A 3xx is a failure; following redirects with signed POSTs is how tokens leak.
2. 5-second timeout. Your workers are a shared resource. One slow endpoint must not starve everyone.
3. The timestamp is inside the signature. We will see why in the security section.

The retry lifecycle

When delivery fails (it will), the event enters a retry state machine:

Notice the schedule spans hours to days, not seconds. Real consumer outages last as long as their incident does. Jitter prevents every failed delivery from retrying at the same instant (the thundering herd that turns their recovery into their second outage).

And the dead letter is not a grave. It feeds a customer-visible delivery log with a redeliver button. That one dashboard converts 2 a.m. incident calls into self-service clicks. If you look at what separates beloved webhook providers (Stripe, GitHub) from hated ones, it is mostly this dashboard.

Two more provider must-haves, briefly:

• Per-endpoint circuit breaker. One customer's dead endpoint should be paused and its owner notified, not allowed to clog workers for everyone.
• SSRF protection. This one is spicy. Customers register arbitrary URLs, and your infrastructure obediently POSTs to them. Without validation, someone registers http://169.254.169.254/latest/meta-data/ (cloud metadata) or an internal admin service, and congratulations: your webhook sender is now a proxy into your own private network. Resolve destination IPs and block private/loopback/metadata ranges, or route all webhook egress through a guard proxy like Smokescreen. Most tutorials never mention this. Attackers love that.

---

5. Consumer side: verify, enqueue, ack. That is the whole endpoint.

Here is the receiving side, and the design rule is brutal: the HTTP handler does zero business logic.

import express from "express";

const app = express();

// RAW body. JSON.parse + re-stringify changes bytes and breaks signatures.
app.post("/webhooks/billing", express.raw({ type: "application/json" }), async (req, res) => {
  const body = req.body as Buffer;
  const id = req.header("webhook-id");
  const timestamp = req.header("webhook-timestamp");
  const signature = req.header("webhook-signature");

  if (!id || !timestamp || !signature || !verify(body, id, timestamp, signature)) {
    return res.status(401).end();   // fail closed
  }

  await inboundQueue.add("process", { id, body: body.toString("utf8") });
  res.status(200).end();            // fast-ack, we are done
});

Why fast-ack? Imagine your handler provisions a server inline and takes 8 seconds. The provider times out at 5, marks the delivery failed, and retries. Now you are provisioning the same server twice AND being punished for succeeding slowly. Acknowledge fast; work async.

Verification: two details separate correct from vulnerable

const TOLERANCE_SECONDS = 300; // 5 minutes

function verify(body: Buffer, id: string, timestamp: string, header: string): boolean {
  // Detail 1: replay window. A captured request is worthless after 5 minutes.
  const age = Math.abs(Date.now() / 1000 - Number(timestamp));
  if (Number.isNaN(age) || age > TOLERANCE_SECONDS) return false;

  const expected = crypto
    .createHmac("sha256", process.env.WEBHOOK_SECRET!)
    .update(`${id}.${timestamp}.${body.toString("utf8")}`)
    .digest();

  const received = Buffer.from(header.replace(/^v1=/, ""), "hex");

  // Detail 2: constant-time comparison. Plain === leaks timing information
  // that lets attackers brute-force signatures byte by byte.
  return received.length === expected.length && crypto.timingSafeEqual(expected, received);
}

Why is the timestamp inside the signed content? Because if it were just a header, an attacker replaying a captured request could freshen the timestamp. Signed, it is tamper-proof: the replay window actually closes.

(The v1= prefix is doing quiet work too: versioned signatures let the provider rotate keys with zero downtime by sending multiple signatures during rotation.)

Idempotency: the consumer's half of the bargain

Duplicates are not an edge case. They are the delivery guarantee working as designed. Let the database be the referee:

async function processEvent(job: { id: string; body: string }) {
  const event = JSON.parse(job.body);

  try {
    await db.processedEvent.create({ data: { eventId: job.id } }); // UNIQUE(eventId)
  } catch (e) {
    if (isUniqueViolation(e)) return;  // seen it, done
    throw e;
  }

  switch (event.type) {
    case "invoice.paid":
      await activateService(event.data.invoiceId); // itself safe to re-run
      break;
  }
}

One more trap: ordering. Networks reorder, retries reorder harder. invoice.paid can arrive before invoice.created. The robust pattern (Stripe's own advice for consuming their webhooks): treat the event as a ping that something changed, then fetch current state from the provider's API and act on that. State from the source of truth beats state reconstructed from event payloads.

---

6. The twist ending: webhooks + polling

Remember the comparison table from the start? Here is where it pays off.

You WILL miss events. A bad deploy 500s for an hour. A subscription gets misconfigured. A route gets fat-fingered. Webhooks are the fast path, not the source of truth.

So the final piece is a boring nightly job:

// The unglamorous job that makes the glamorous system trustworthy
async function reconcile() {
  const since = await getLastReconciledCursor();
  for await (const event of billingApi.listEvents({ since })) {
    await inboundQueue.add("process", { id: event.id, body: JSON.stringify(event) });
    // pipeline is idempotent, so replaying already-processed events costs nothing
  }
}

Webhooks give you speed. Polling gives you certainty. Production systems use both, and because the whole pipeline is idempotent, the backfill is free to run.

Most tutorials skip this. Most teams add it after their first "why is this customer's server not activated" incident. Add it on day one. It is twenty lines.

---

7. Cheat sheet

None of these pieces is hard. The hard part is remembering that the happy path was never the design problem. The design problem is Tuesday's deploy window, and this system shrugs it off.

---

I write about backend systems and agentic AI at aweshislam.com. If you have shipped webhooks at scale and disagree with something here, I would genuinely like to hear it.