Monitoring & Alerts

Monitor your CDN health and get notified when things go wrong. Includes automated health checks, email/Slack alerts, and built-in observability dashboards.

What Gets Monitored

Terraform automatically configures:

API health - /v1/status endpoint checked every 60s from 6 global regions (requires Pro plan or higher)
Health check recovery - Notification when API comes back online

Additional alerts available (configure manually in dashboard):

Workers usage - CPU spikes and weekly summaries
Billing thresholds - Alert at custom percentage of monthly cost
SSL certificates - Expiry warnings
Custom domains - Validation failures

Health checks require a paid plan (Pro $20/month minimum). On Free plan, disable health checks in .env:

bash

ENABLE_HEALTH_CHECKS='false'

Architecture

mermaid

graph LR
    A[Health Check] -->|Every 60s| B[v1\/status]
    B -->|OK| C[No Alert]
    B -->|3 Failures| D[Alert]
    B -->|Recovers| D[Alert]

    D -->|Sends to| E[Email]
    D -->|Sends to| F[Slack/Discord]
    D -->|Sends to| G[PagerDuty]

The health check monitors your status endpoint from multiple regions. When 3 consecutive failures occur (180 seconds total), notifications are triggered. Requires Pro plan or higher.

Setup

Step 1: Update API Token Permissions

Your API token needs additional permissions for monitoring. Update your existing token or create a new one:

Go to https://dash.cloudflare.com/profile/api-tokens
Create Token → Create Custom Token
Add permissions:
- Account → Notifications → Edit (for alert policies)
- Account → Account Settings → Read (for account info)
- Zone → Health Checks → Edit (for API monitoring)
Set zone resources to "All zones" or your specific zone
Copy the token and update your .env file

Step 2: Create Notification Destinations

Email Destinations

bash

# Create email destination
curl -X POST "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/alerting/v3/destinations/email" \
     -H "Authorization: Bearer YOUR_API_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"name": "Admin", "email": "admin@example.com"}' | jq '.result.id'

# Output: "abc123def456ghi789"

Check your inbox and click the verification link. Then add the ID to .env:

bash

NOTIFICATION_EMAILS='abc123def456ghi789'

For multiple emails, comma-separate the IDs:

bash

NOTIFICATION_EMAILS='abc123,def456,ghi789'

Slack Webhooks

Create webhook at https://api.slack.com/apps:
- Create New App → From scratch
- Incoming Webhooks → Activate
- Add New Webhook to Workspace
- Copy webhook URL
Register with Cloudflare:

bash

curl -X POST "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/alerting/v3/destinations/webhooks" \
     -H "Authorization: Bearer YOUR_API_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "name": "Slack Alerts",
       "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
     }' | jq '.result.id'

# Output: "webhook_xyz789"

Add to .env:

bash

NOTIFICATION_WEBHOOK_IDS='webhook_xyz789'

Discord Webhooks

Same as Slack, but append /slack to the Discord webhook URL:

bash

# Discord URL format
https://discord.com/api/webhooks/XXX/YYY/slack

Step 3: Get Destination IDs

If you created destinations in the dashboard instead of via API, fetch their IDs:

bash

# List email IDs
curl -X GET "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/alerting/v3/destinations/email" \
     -H "Authorization: Bearer YOUR_API_TOKEN" | jq '.result[] | {id, email}'

# List webhook IDs
curl -X GET "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/alerting/v3/destinations/webhooks" \
     -H "Authorization: Bearer YOUR_API_TOKEN" | jq '.result[] | {id, name}'

Copy the IDs to your .env file.

Step 4: Configure Environment

At least one notification destination (email, webhook, or PagerDuty) is required for alerts to work.

bash

# Enable monitoring (defaults to true)
ENABLE_HEALTH_CHECKS='true'   # Set to 'false' on Free plan
ENABLE_NOTIFICATIONS='true'

# Destination IDs from Step 2/3 (at least one required)
NOTIFICATION_EMAILS='abc123def456ghi789'
NOTIFICATION_WEBHOOK_IDS='webhook_xyz789'
# PAGERDUTY_INTEGRATION_IDS='pagerduty_xyz789'  # Optional

Step 5: Deploy

bash

make env-to-tfvars
cd terraform && terraform apply

Terraform creates:

Health check for /v1/status endpoint (if ENABLE_HEALTH_CHECKS='true' and on paid plan)
2 notification policies: health check failure and recovery (if destinations configured)

For additional alerts (Workers usage, billing thresholds, SSL expiration), configure manually in the Cloudflare dashboard.

Health Check Details

The health check monitors your API status endpoint:

URL: https://your-domain.com/api/v1/status (or /v1/status without gateway)
Interval: Every 60 seconds
Timeout: 10 seconds
Retries: 2 attempts before marking as failed
Threshold: 3 consecutive failures (180s total) triggers alert
Regions: WNAM, ENAM, WEU, EEU, SEAS, OC (6 global regions)
Expected: 200 status code + body contains "status":

When the endpoint recovers after 2 consecutive successes, a recovery notification is sent.

Status Endpoint

The monitored endpoint returns system health:

bash

curl https://your-domain.com/api/v1/status | jq

json

{
  "status": "ok",
  "timestamp": "2025-01-26T12:00:00.000Z",
  "app": {
    "name": "Pixflare API",
    "version": "1.0.0",
    "environment": "production"
  },
  "dependencies": {
    "database": { "ok": true, "latency_ms": 5 },
    "storage": { "ok": true },
    "cache": { "ok": true, "latency_ms": 2 },
    "analytics": { "ok": true },
    "queues": { "ok": true }
  }
}

Status values:

ok - All systems operational
degraded - Optional services down (analytics, queues, etc)
error - Core services down (database, storage, cache)

Alert Types

Automated via Terraform

Health Check Failure

Trigger: API down for 180 seconds (3 failed checks)
Recipients: Email, webhook, and/or PagerDuty (based on your configured destinations)
Action: Check /v1/status manually, review recent deployments, check dependency health

Health Check Recovery

Trigger: API recovers after 2 consecutive successful checks
Recipients: Same as above
Action: Confirms downtime is resolved

Manual Setup Required

The following alerts require manual configuration in the Cloudflare dashboard due to Terraform provider limitations (missing filter support, incompatible alert types, or recipient restrictions).

Configure at: Cloudflare Notifications Dashboard

Workers Usage & Performance

CPU spike alerts (25% above baseline)
Weekly usage summaries
Error rate thresholds

Billing Thresholds

Alert at custom percentage of monthly estimate (e.g., 80%, 100%)
Per-product monitoring (Workers, R2, D1, Queues, etc.)

SSL Certificate Expiration

30-day and 14-day expiry warnings
Validation failure alerts

You can set ENABLE_WORKERS_USAGE_NOTIFICATIONS, ENABLE_BILLING_ALERTS, and ENABLE_SSL_ALERTS in your .env to document your intent, but these won't be auto-configured by Terraform.

Observability

Workers Logs

Real-time logs available at:

https://dash.cloudflare.com/?to=/:account/workers/observability/logs

Features:

Live log streaming with tail mode
Filter by time range, status code, search terms
Export to CSV
Request/response tracking with unique IDs

All logs include structured context with request IDs for tracing.

Analytics Dashboard

Built-in analytics at https://your-domain.com/app/stats:

Total requests and bandwidth
Per-image, per-album, per-variant stats
Geographic distribution
Traffic sources and referrers
Time-series visualizations

See Analytics for configuration details.

Audit Logs

All user actions are logged in the database:

bash

GET /v1/audit-logs?limit=50

Includes image uploads, deletions, settings changes, API key generation, etc.

External Monitoring

For additional observability, integrate with:

Logpush - Send logs to S3, Datadog, Splunk, etc:

bash

# See: https://developers.cloudflare.com/workers/observability/logpush/

Sentry - Error tracking with stack traces:

bash

pnpm --filter @pixflare/api add @sentry/cloudflare

UptimeRobot - External uptime checks from outside Cloudflare:

Monitor /v1/status every 5 minutes from multiple continents.

Disabling Monitoring

To disable monitoring features:

bash

# In .env
ENABLE_HEALTH_CHECKS='false'      # Disable health checks (required on Free plan)
ENABLE_NOTIFICATIONS='false'      # Disable notification policies

Then run make env-to-tfvars && terraform apply.

Note: ENABLE_NOTIFICATIONS='false' prevents notification policy creation even if destinations are configured. ENABLE_HEALTH_CHECKS='false' disables the health check itself. They can be controlled independently.

Troubleshooting

"health checks disabled for zone" (1002)

Health checks require a paid Cloudflare plan (Pro $20/month or higher). On Free plan, disable them:

bash

# In .env
ENABLE_HEALTH_CHECKS='false'

Then regenerate and apply:

bash

make env-to-tfvars
cd terraform && terraform apply

"Authentication error (10000)" for health checks

Your API token is missing health check permissions. Add:

Zone → Health Checks → Edit

"POST method not allowed for api_token"

Your API token doesn't have notification permissions. Add:

Account → Notifications → Edit
Account → Account Settings → Read

"Invalid recipient (17002)"

You're using email addresses instead of destination IDs. The NOTIFICATION_EMAILS value should be the destination ID (like abc123def456), not the actual email address.

Get the correct destination IDs:

bash

curl -X GET "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/alerting/v3/destinations/email" \
     -H "Authorization: Bearer YOUR_API_TOKEN" | jq '.result[] | {id, email}'

Copy the id value (not the email) to your .env file.

Dashboard doesn't show destination IDs

Cloudflare UI doesn't display IDs. Use the API to fetch them:

bash

curl -X GET "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/alerting/v3/destinations/email" \
     -H "Authorization: Bearer YOUR_API_TOKEN" | jq '.result[] | {id, email}'

Not receiving emails

Verify email is verified (check inbox for confirmation link)
Check NOTIFICATION_EMAILS contains the destination ID, not the email address
Confirm terraform apply ran successfully
Check spam folder

Health check shows unhealthy but API works

Firewall rules or Access policies may be blocking health checks. Whitelist the user agent:

hcl

# In terraform/modules/security/main.tf
resource "cloudflare_filter" "allow_health_checks" {
  zone_id     = var.zone_id
  expression  = "(http.user_agent contains \"Cloudflare-Traffic-Manager\")"
}

resource "cloudflare_firewall_rule" "allow_health_checks" {
  zone_id   = var.zone_id
  filter_id = cloudflare_filter.allow_health_checks.id
  action    = "allow"
  priority  = 1
}

"Invalid index" or "empty tuple" error

This means notification policies are enabled but no destinations are configured. You need at least one of:

NOTIFICATION_EMAILS
NOTIFICATION_WEBHOOK_IDS
PAGERDUTY_INTEGRATION_IDS

Follow Steps 2-3 above to create and configure notification destinations.

No notifications created

Check:

ENABLE_NOTIFICATIONS='true' in .env
At least one destination ID is configured (email, webhook, or PagerDuty)
make env-to-tfvars was run after updating .env
Terraform module is included in terraform/main.tf

Cost

Notification features (free on all plans):

Notification Policies: Free (unlimited)
Email Notifications: Free (unlimited)
Webhook Notifications: Free (unlimited)

Health Checks (requires paid plan):

Not available on Free plan
Pro plan ($20/month): 5 health checks included
Business/Enterprise: Unlimited health checks

No additional charges beyond plan costs. Set ENABLE_HEALTH_CHECKS='false' on Free plan to use notifications without health checks.

Monitoring & Alerts ​

What Gets Monitored ​

Architecture ​

Setup ​

Step 1: Update API Token Permissions ​

Step 2: Create Notification Destinations ​

Email Destinations ​

Slack Webhooks ​

Discord Webhooks ​

Step 3: Get Destination IDs ​

Step 4: Configure Environment ​

Step 5: Deploy ​

Health Check Details ​

Status Endpoint ​

Alert Types ​

Automated via Terraform ​

Manual Setup Required ​

Observability ​

Workers Logs ​

Analytics Dashboard ​

Audit Logs ​

External Monitoring ​

Disabling Monitoring ​

Troubleshooting ​

"health checks disabled for zone" (1002) ​

"Authentication error (10000)" for health checks ​

"POST method not allowed for api_token" ​

"Invalid recipient (17002)" ​

Dashboard doesn't show destination IDs ​

Not receiving emails ​

Health check shows unhealthy but API works ​

"Invalid index" or "empty tuple" error ​

No notifications created ​

Cost ​

Monitoring & Alerts

What Gets Monitored

Architecture

Setup

Step 1: Update API Token Permissions

Step 2: Create Notification Destinations

Email Destinations

Slack Webhooks

Discord Webhooks

Step 3: Get Destination IDs

Step 4: Configure Environment

Step 5: Deploy

Health Check Details

Status Endpoint

Alert Types

Automated via Terraform

Manual Setup Required

Observability

Workers Logs

Analytics Dashboard

Audit Logs

External Monitoring

Disabling Monitoring

Troubleshooting

"health checks disabled for zone" (1002)

"Authentication error (10000)" for health checks

"POST method not allowed for api_token"

"Invalid recipient (17002)"

Dashboard doesn't show destination IDs

Not receiving emails

Health check shows unhealthy but API works

"Invalid index" or "empty tuple" error

No notifications created

Cost