Webhook Retry Policy

TruthVouch automatically retries failed webhook deliveries with exponential backoff.

Retry Schedule

Events are retried up to 5 times with exponential backoff:

Attempt	Delay	Total Time
1	Immediate	0 seconds
2	1 minute	1 minute
3	5 minutes	6 minutes
4	30 minutes	36 minutes
5	2 hours	2 hours 36 minutes
Final	12 hours	14 hours 36 minutes

Example timeline:

14:00 - Attempt 1 fails (timeout)
14:01 - Attempt 2 fails (connection refused)
14:06 - Attempt 3 fails (500 error)
14:36 - Attempt 4 fails (timeout)
16:36 - Attempt 5 fails (400 error)
04:36 (next day) - Dead letter queue

Failure Conditions

Webhook delivery fails if:

Connection timeout — No response within 30 seconds
Network error — DNS failure, connection refused
HTTP 5xx — Server error (500, 502, 503, etc.)
HTTP 429 — Rate limiting (retried with backoff)
HTTP 4xx — Client error (400, 401, 403, 404, etc.)

Success Criteria

Delivery succeeds if:

HTTP 2xx — Any 200-299 status code
Response within 30 seconds
Valid response body (optional)

Partial Success

If your endpoint returns HTTP 200 but processing fails internally:

TruthVouch considers it successful (won’t retry)
Implement idempotency to handle retries

Exponential Backoff Algorithm

def calculate_next_retry(attempt: int) -> timedelta:
    """Calculate backoff for attempt N (1-indexed)."""
    base_delays = [0, 60, 300, 1800, 7200, 43200]  # seconds
    jitter = random.uniform(0, 10)  # Add randomness
    delay = base_delays[min(attempt, len(base_delays) - 1)] + jitter
    return timedelta(seconds=delay)

Dead Letter Queue

What Happens to Failed Events

After 5 retries, events are moved to the dead letter queue (DLQ):

Retention: 30 days Access: Dashboard or API Retry: Manual replay

View DLQ Events

Dashboard:

Settings → Webhooks → Failed Events
See event ID, type, error, timestamp
Click to view full payload

API:

curl https://api.truthvouch.io/v1/webhooks/dlq \
  -H "Authorization: Bearer token"

# Response
{
  "events": [
    {
      "event_id": "evt-abc123",
      "event_type": "alert.detected",
      "timestamp": "2024-01-15T14:00:00Z",
      "error": "connection_timeout",
      "webhook_url": "https://yourserver.com/webhooks"
    }
  ],
  "total": 5,
  "cursor": "dlq-cursor-xyz"
}

Replay Failed Events

Manual replay via dashboard:

Settings → Webhooks → Failed Events
Select event(s)
Click “Replay”
Events requeued immediately

Manual replay via API:

curl -X POST https://api.truthvouch.io/v1/webhooks/dlq/replay \
  -H "Authorization: Bearer token" \
  -H "Content-Type: application/json" \
  -d '{
    "event_ids": ["evt-abc123", "evt-def456"]
  }'

# Response
{
  "replayed": 2,
  "status": "success"
}

Webhook Configuration

Set Retry Behavior

Dashboard:

Settings → Webhooks
Create or edit webhook
Retry policy options:
- ✓ Automatic retry (default)
- ✗ Disable retries (fail once)

API:

curl -X POST https://api.truthvouch.io/v1/webhooks \
  -H "Authorization: Bearer token" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://yourserver.com/webhooks",
    "retry_policy": {
      "max_attempts": 5,
      "backoff_multiplier": 1.5,
      "exponential": true
    }
  }'

Handling Retries

Implement Idempotency

Since retries are automatic, handle duplicate deliveries:

Python:

from flask import Flask, request
import hashlib

app = Flask(__name__)

# Track processed events
processed_ids = set()

@app.route('/webhooks', methods=['POST'])
def handle_webhook():
    event = request.json
    event_id = event['event_id']

    # Check if already processed
    if event_id in processed_ids:
        return {'status': 'already_processed'}, 200

    # Process event
    try:
        process_event(event)
        processed_ids.add(event_id)
        return {'status': 'success'}, 200
    except Exception as e:
        # Return 5xx so TruthVouch retries
        return {'error': str(e)}, 500

Node.js:

const express = require('express');
const app = express();

const processedIds = new Set();

app.post('/webhooks', (req, res) => {
  const eventId = req.body.event_id;

  // Already processed?
  if (processedIds.has(eventId)) {
    return res.status(200).json({ status: 'already_processed' });
  }

  // Process event
  try {
    processEvent(req.body);
    processedIds.add(eventId);
    res.status(200).json({ status: 'success' });
  } catch (error) {
    // Return 5xx for retry
    res.status(500).json({ error: error.message });
  }
});

.NET:

using Microsoft.AspNetCore.Mvc;
using System.Collections.Generic;

[ApiController]
[Route("webhooks")]
public class WebhookController : ControllerBase
{
    private static HashSet<string> ProcessedIds = new HashSet<string>();

    [HttpPost]
    public IActionResult HandleWebhook([FromBody] WebhookEvent @event)
    {
        var eventId = @event.EventId;

        // Check if already processed
        if (ProcessedIds.Contains(eventId))
        {
            return Ok(new { status = "already_processed" });
        }

        try
        {
            ProcessEvent(@event);
            ProcessedIds.Add(eventId);
            return Ok(new { status = "success" });
        }
        catch (Exception ex)
        {
            // Return 500 for retry
            return StatusCode(500, new { error = ex.Message });
        }
    }
}

Store Event ID Globally

For distributed systems, use a database:

CREATE TABLE processed_webhooks (
  event_id VARCHAR(50) PRIMARY KEY,
  received_at TIMESTAMP,
  processed_at TIMESTAMP
);

Respond Quickly

Return 200 within 5 seconds, process asynchronously:

from celery import shared_task

@app.route('/webhooks', methods=['POST'])
def handle_webhook():
    event = request.json

    # Queue for async processing
    process_event_async.delay(event)

    # Return immediately
    return {'status': 'queued'}, 200

@shared_task
def process_event_async(event):
    # Process in background
    process_event(event)

Monitoring Retries

View Retry Status

Dashboard:

Settings → Webhooks
Select webhook
View “Delivery Status” tab:
- Successful deliveries
- Failed deliveries
- Pending retries
- DLQ events

API:

curl https://api.truthvouch.io/v1/webhooks/{webhook_id}/status \
  -H "Authorization: Bearer token"

# Response
{
  "webhook_id": "wh-abc123",
  "url": "https://yourserver.com/webhooks",
  "active": true,
  "stats": {
    "total_events": 1000,
    "successful": 980,
    "failed": 20,
    "pending_retry": 3
  },
  "last_success": "2024-01-15T15:00:00Z",
  "last_failure": "2024-01-15T14:30:00Z"
}

Set Up Alerts

Alert on too many failures:

Settings → Webhooks → Alerts
Create alert: “Failures > 10 in 1 hour”
Receive email/Slack notification

Best Practices

1. Implement Idempotency

Always handle retried events gracefully:

# Good: Check event_id before processing
if event_id in processed_events:
    return 200

# Bad: Process every delivery
process_event(event)
return 200

2. Return Proper Status Codes

2xx → Success (don't retry)
4xx → Don't retry (likely bad request)
5xx → Retry (likely transient error)

3. Respond Within 30 Seconds

Good: Return 200 immediately, queue async
Bad: Take 60 seconds to process

4. Log Webhook Activity

logger.info(f"Webhook: {event_id} received")
logger.error(f"Webhook: {event_id} processing failed: {error}")

5. Monitor DLQ

Check dashboard weekly for failed events requiring manual attention.

Troubleshooting

Webhook Not Being Delivered

Check status:

curl https://api.truthvouch.io/v1/webhooks/{webhook_id}/status

Common causes:

Webhook URL is unreachable
Server not responding within 30s
Server returns 4xx (permanent failure)
TruthVouch IP blocked by firewall

Too Many Retries

Disable retries:

Settings → Webhooks
Edit webhook
Toggle off “Automatic Retry”

Or: Return 200 immediately (even if processing fails):

@app.route('/webhooks')
def webhook():
    # Return success immediately
    return 200

    # Process asynchronously
    # (retries won't help anyway)

Idempotency Not Working

Ensure event_id is unique and persistent:

Use provided event_id field
Store in database before processing
Check before processing (not after)

See Webhook Events, Signatures, and Testing for more details.