Webhook Management

Effective webhook management is crucial for building reliable, event-driven applications. This guide covers advanced management techniques, processing strategies, and best practices.

Webhook Architecture

Recommended Architecture

TODO: Add diagram showing webhook architecture: Webhook Endpoint → Queue → Workers → Database Webhook Architecture Diagram

Components:

Webhook Endpoint - Receives webhooks and responds quickly
Message Queue - Queues webhooks for processing
Workers - Process webhooks asynchronously
Database - Store webhook events and processing status
Monitoring - Track webhook processing and health

Webhook Processing Patterns

Pattern 1: Fire and Forget

Process webhook immediately in the endpoint. Pros:

Simple implementation
No additional infrastructure

Cons:

Slow responses may cause retries
No retry mechanism if processing fails
Blocks webhook endpoint

When to Use:

Simple, fast processing
Low volume
Non-critical events

Pattern 2: Queue-Based Processing

Queue webhooks for asynchronous processing. Pros:

Fast webhook response
Built-in retry mechanism
Scalable processing
Better error handling

Cons:

Requires queue infrastructure
More complex setup

When to Use:

Complex processing
High volume
Critical events
Production systems

TODO: Add diagram comparing fire-and-forget vs queue-based processing Webhook Processing Patterns Diagram

Pattern 3: Event Sourcing

Store all webhooks as events and process asynchronously. Pros:

Complete event history
Reprocessing capability
Audit trail
Event replay

Cons:

More complex architecture
Storage requirements

When to Use:

Need event history
Audit requirements
Complex event processing

Implementation Examples

Queue-Based Implementation

const Queue = require('bull');
const webhookQueue = new Queue('webhooks', {
  redis: { host: 'localhost', port: 6379 }
});

// Webhook endpoint - quick response
app.post('/webhooks/pooler', async (req, res) => {
  // Verify signature
  if (!verifyWebhookSignature(req)) {
    return res.status(401).send('Invalid signature');
  }
  
  // Add to queue
  await webhookQueue.add('process', req.body, {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 2000
    }
  });
  
  // Respond immediately
  res.status(200).send('OK');
});

// Worker - process webhooks
webhookQueue.process('process', async (job) => {
  const webhook = job.data;
  
  // Store webhook
  await storeWebhook(webhook);
  
  // Process based on event type
  switch (webhook.event) {
    case 'payment.completed':
      await handlePaymentCompleted(webhook.data);
      break;
    case 'virtual_account.payment_received':
      await handleVirtualAccountPayment(webhook.data);
      break;
    // ... other events
  }
});

from celery import Celery

app = Celery('webhooks', broker='redis://localhost:6379')

# Webhook endpoint - quick response
@app.route('/webhooks/pooler', methods=['POST'])
def handle_webhook():
    if not verify_webhook_signature(request):
        return 'Invalid signature', 401
    
    # Add to queue
    process_webhook.delay(request.json)
    
    # Respond immediately
    return 'OK', 200

# Worker - process webhooks
@app.task
def process_webhook(webhook):
    # Store webhook
    store_webhook(webhook)
    
    # Process based on event type
    event_type = webhook['event']
    if event_type == 'payment.completed':
        handle_payment_completed(webhook['data'])
    elif event_type == 'virtual_account.payment_received':
        handle_virtual_account_payment(webhook['data'])
    # ... other events

Idempotency Handling

Why Idempotency Matters

Webhooks may be delivered multiple times. Your processing must be idempotent to prevent duplicate actions. TODO: Add diagram showing idempotency check flow Webhook Idempotency Flow Diagram

Implementation

async function processWebhook(webhook) {
  const eventId = webhook.data.id || webhook.signature;
  
  // Check if already processed
  const processed = await checkIfProcessed(eventId);
  if (processed) {
    console.log(`Event ${eventId} already processed, skipping`);
    return;
  }
  
  // Mark as processing
  await markAsProcessing(eventId);
  
  try {
    // Process webhook
    await handleWebhookEvent(webhook);
    
    // Mark as completed
    await markAsCompleted(eventId);
  } catch (error) {
    // Mark as failed
    await markAsFailed(eventId, error);
    throw error;
  }
}

def process_webhook(webhook):
    event_id = webhook['data'].get('id') or webhook.get('signature')
    
    # Check if already processed
    if is_processed(event_id):
        logger.info(f'Event {event_id} already processed, skipping')
        return
    
    # Mark as processing
    mark_as_processing(event_id)
    
    try:
        # Process webhook
        handle_webhook_event(webhook)
        
        # Mark as completed
        mark_as_completed(event_id)
    except Exception as error:
        # Mark as failed
        mark_as_failed(event_id, error)
        raise

Event Routing

Route Events to Handlers

Route different event types to appropriate handlers:

const eventHandlers = {
  'payment.completed': handlePaymentCompleted,
  'payment.failed': handlePaymentFailed,
  'virtual_account.payment_received': handleVirtualAccountPayment,
  'recipient.verified': handleRecipientVerified,
  // ... other handlers
};

async function handleWebhookEvent(webhook) {
  const handler = eventHandlers[webhook.event];
  
  if (!handler) {
    console.warn(`No handler for event: ${webhook.event}`);
    return;
  }
  
  await handler(webhook.data);
}

TODO: Add diagram showing event routing to different handlers Webhook Event Routing Diagram

Error Handling

Error Handling Strategy

Catch Errors

Wrap webhook processing in try-catch blocks.

Log Errors

Log errors with full context for debugging.

Retry Logic

Implement retry logic for transient errors.

Dead Letter Queue

Move permanently failed webhooks to dead letter queue.

Alerting

Alert on high error rates or critical failures.

TODO: Add diagram showing error handling flow with retries and dead letter queue Webhook Error Handling Flow Diagram

Error Types

Transient Errors - Retry with backoff:

Network timeouts
Temporary service unavailability
Rate limiting

Permanent Errors - Don’t retry:

Invalid webhook data
Business logic errors
Authentication failures

Webhook Storage

Store Webhook Events

Store all webhook events for:

Audit trail
Debugging
Reprocessing
Analytics

Database Schema Example

CREATE TABLE webhook_events (
  id UUID PRIMARY KEY,
  event_type VARCHAR(100) NOT NULL,
  event_data JSONB NOT NULL,
  signature VARCHAR(255),
  received_at TIMESTAMP NOT NULL,
  processed_at TIMESTAMP,
  processing_status VARCHAR(50),
  error_message TEXT,
  retry_count INTEGER DEFAULT 0,
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_webhook_events_event_type ON webhook_events(event_type);
CREATE INDEX idx_webhook_events_status ON webhook_events(processing_status);
CREATE INDEX idx_webhook_events_received_at ON webhook_events(received_at);

Monitoring and Observability

Key Metrics

Monitor these metrics:

Webhook Volume - Events received per time period
Processing Time - Average processing time
Success Rate - Percentage of successfully processed webhooks
Error Rate - Percentage of failed webhooks
Retry Rate - Percentage of webhooks requiring retries
Latency - Time from receipt to processing completion

TODO: Add diagram showing webhook monitoring dashboard Webhook Monitoring Dashboard Diagram

Alerting

Set up alerts for:

High error rates (> 5%)
Slow processing times (> 10 seconds)
Missing critical events
Queue backup

Testing Strategies

Unit Testing

Test individual event handlers:

describe('Payment Completed Handler', () => {
  it('should update order status', async () => {
    const webhook = {
      event: 'payment.completed',
      data: {
        id: 'payment_123',
        reference: 'ORDER-001',
        amount: 1000
      }
    };
    
    await handlePaymentCompleted(webhook.data);
    
    const order = await getOrder('ORDER-001');
    expect(order.status).toBe('paid');
  });
});

Integration Testing

Test webhook endpoint end-to-end:

Send test webhooks
Verify processing
Check database updates
Verify side effects

Load Testing

Test webhook endpoint under load:

High volume of concurrent webhooks
Verify queue handling
Check processing capacity
Monitor resource usage

Best Practices Summary

Respond Quickly - Acknowledge webhooks within 5 seconds to prevent retries.

Process Asynchronously - Use queues for complex processing to keep responses fast.

Handle Idempotency - Always check if events were already processed.

Verify Signatures - Always verify webhook signatures for security.

Log Everything - Log all webhook events for debugging and audit.

Monitor Closely - Set up monitoring and alerts for webhook health.

Test Thoroughly - Test webhook handling in all scenarios.

Next Steps

Review Webhooks Overview for basics
Learn about Signature Security for secure implementation
Check the List Events API for available events

Getting started

Business

Virtual Accounts

Recipients

Webhooks

Payments

Webhook Architecture

Recommended Architecture

Webhook Processing Patterns

Pattern 1: Fire and Forget

Pattern 2: Queue-Based Processing

Pattern 3: Event Sourcing

Implementation Examples

Queue-Based Implementation

Idempotency Handling

Why Idempotency Matters

Implementation

Event Routing

Route Events to Handlers

Error Handling

Error Handling Strategy

Error Types

Webhook Storage

Store Webhook Events

Database Schema Example

Monitoring and Observability

Key Metrics

Alerting

Testing Strategies

Unit Testing

Integration Testing

Load Testing

Best Practices Summary

Next Steps

​Webhook Architecture

​Recommended Architecture

​Webhook Processing Patterns

​Pattern 1: Fire and Forget

​Pattern 2: Queue-Based Processing

​Pattern 3: Event Sourcing

​Implementation Examples

​Queue-Based Implementation

​Idempotency Handling

​Why Idempotency Matters

​Implementation

​Event Routing

​Route Events to Handlers

​Error Handling

​Error Handling Strategy

​Error Types

​Webhook Storage

​Store Webhook Events

​Database Schema Example

​Monitoring and Observability

​Key Metrics

​Alerting

​Testing Strategies

​Unit Testing

​Integration Testing

​Load Testing

​Best Practices Summary

​Next Steps

Webhook Architecture

Recommended Architecture

Webhook Processing Patterns

Pattern 1: Fire and Forget

Pattern 2: Queue-Based Processing

Pattern 3: Event Sourcing

Implementation Examples

Queue-Based Implementation

Idempotency Handling

Why Idempotency Matters

Implementation

Event Routing

Route Events to Handlers

Error Handling

Error Handling Strategy

Error Types

Webhook Storage

Store Webhook Events

Database Schema Example

Monitoring and Observability

Key Metrics

Alerting

Testing Strategies

Unit Testing

Integration Testing

Load Testing

Best Practices Summary

Next Steps