Varnish Integration

Integrate redirect lookups with Varnish Cache using VCL (Varnish Configuration Language) and periodic sync from the Edge Query API.

Overview

This guide shows you how to integrate redirect lookups into Varnish Cache, the high-performance HTTP accelerator. You'll learn two integration patterns:

Periodic Sync + VCL (Strongly Recommended): Download redirects and generate VCL snippets for native, zero-overhead lookups
Real-Time Backend Proxy (Advanced): Use a separate backend to proxy redirect lookups to the API

Estimated time: 30 minutes

Prerequisites:

Varnish 7.4+
curl command-line tool
API key for the Edge Query API
Optional: vmod_curl for real-time lookups (requires compilation)

Important: VCL (Varnish Configuration Language) does not have a built-in HTTP client for external API calls. The sync + VCL pattern is strongly recommended for production use.

Request Flow

flowchart TD
    A[Client Request] --> B[Varnish vcl_recv]
    B --> C{Check Redirect}
    C -->|Match in VCL| D[vcl_synth - 301 Redirect]
    C -->|No Match| E[Pass to Backend]
    D --> F[Client]
    E --> G[Origin Server]

Pattern 1: Periodic Sync + VCL (Strongly Recommended)

This pattern downloads redirects from the API and generates a VCL include file containing redirect logic. Varnish performs lookups using native VCL conditionals with zero external dependencies and zero latency overhead.

How It Works

A sync script downloads redirects from the Edge Query API
The script generates a VCL snippet with if/elseif conditionals
The main VCL includes this snippet in vcl_recv
A cron job regenerates the VCL and reloads Varnish without downtime

Step 1: Create the Sync Script

Create /usr/local/bin/sync-varnish-redirects.sh:

#!/bin/bash
set -euo pipefail

API_URL="https://api.3xx.app"
PROJECT_ID="YOUR_PROJECT_ID"
API_KEY="YOUR_API_KEY"
VCL_FILE="/etc/varnish/redirects.vcl"
TEMP_FILE="/tmp/redirects.vcl.$$"

# Download redirects as CSV
REDIRECTS=$(curl -sf \
  -H "X-API-Key: ${API_KEY}" \
  "${API_URL}/v1/export?projectId=${PROJECT_ID}&amp;format=csv")

# Generate VCL snippet
cat &gt; "${TEMP_FILE}" &lt;&lt;'VCL_HEADER'
# Auto-generated redirect rules
# Do not edit manually - regenerated by sync script

sub vcl_recv_redirects {
VCL_HEADER

# Parse CSV and generate VCL if/elseif blocks
echo "$REDIRECTS" | tail -n +2 | while IFS=',' read -r source destination redirect_type; do
    # Remove quotes from CSV fields
    source=$(echo "$source" | sed 's/^"//;s/"$//')
    destination=$(echo "$destination" | sed 's/^"//;s/"$//')
    redirect_type=$(echo "$redirect_type" | sed 's/^"//;s/"$//')

    # Determine status code (301 or 302)
    if [ "$redirect_type" = "temporary" ]; then
        status_code="302"
    else
        status_code="301"
    fi

    # Generate VCL conditional
    cat &gt;&gt; "${TEMP_FILE}" &lt;&lt;VCL_RULE
    if (req.url == "${source}") {
        return (synth(${status_code}, "${destination}"));
    }
VCL_RULE
done

# Close subroutine
cat &gt;&gt; "${TEMP_FILE}" &lt;&lt;'VCL_FOOTER'
}
VCL_FOOTER

# Atomic replacement
mv "${TEMP_FILE}" "${VCL_FILE}"

# Load new VCL and make it active (hot reload, no restart)
VCL_NAME="redirects_$(date +%s)"
varnishadm vcl.load "${VCL_NAME}" /etc/varnish/default.vcl
varnishadm vcl.use "${VCL_NAME}"

# Clean up old VCL versions (keep last 3)
varnishadm vcl.list | grep '^available' | head -n -3 | awk '{print $2}' | while read vcl; do
    varnishadm vcl.discard "$vcl" 2&gt;/dev/null || true
done

echo "Varnish redirects synced and reloaded: VCL ${VCL_NAME}"

Make the script executable:

chmod +x /usr/local/bin/sync-varnish-redirects.sh

Inline verification: Run the sync script manually:

/usr/local/bin/sync-varnish-redirects.sh

You should see output like "Varnish redirects synced and reloaded: VCL redirects_1738368000".

Check the generated VCL:

cat /etc/varnish/redirects.vcl

You should see:

# Auto-generated redirect rules
# Do not edit manually - regenerated by sync script

sub vcl_recv_redirects {
    if (req.url == "/old-path") {
        return (synth(301, "https://example.com/new-path"));
    }
    if (req.url == "/another-path") {
        return (synth(302, "https://example.com/temporary"));
    }
}

Step 2: Configure Varnish VCL

Edit /etc/varnish/default.vcl:

vcl 4.1;

# Backend configuration
backend default {
    .host = "192.168.1.10";
    .port = "8080";
    .connect_timeout = 5s;
    .first_byte_timeout = 10s;
    .between_bytes_timeout = 2s;
}

# Include auto-generated redirects
include "/etc/varnish/redirects.vcl";

sub vcl_recv {
    # Call redirect subroutine
    call vcl_recv_redirects;

    # Continue with normal request processing
    return (hash);
}

sub vcl_synth {
    # Handle redirects generated by vcl_recv_redirects
    if (resp.status == 301 || resp.status == 302) {
        # resp.reason contains the destination URL
        set resp.http.Location = resp.reason;
        set resp.reason = "Moved";
        return (deliver);
    }

    # Handle other synthetic responses (errors, etc.)
    return (deliver);
}

sub vcl_backend_response {
    # Normal caching logic
    return (deliver);
}

sub vcl_deliver {
    # Clean up internal headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;
    return (deliver);
}

Create empty redirect file (prevents Varnish from failing to start):

cat &gt; /etc/varnish/redirects.vcl &lt;&lt;'EOF'
# Auto-generated redirect rules
# Initially empty - will be populated by sync script

sub vcl_recv_redirects {
    # No redirects configured yet
}
EOF

Inline verification: Test the VCL compilation:

varnishd -C -f /etc/varnish/default.vcl

This command compiles the VCL and outputs the C code. If there are syntax errors, you'll see them here.

Load the VCL:

varnishadm vcl.load test_config /etc/varnish/default.vcl
varnishadm vcl.use test_config

Test a redirect:

curl -I http://localhost/old-path

You should see:

HTTP/1.1 301 Moved
Location: https://example.com/new-path

Step 3: Schedule Periodic Sync

Add a cron job to sync redirects every 5 minutes. Run crontab -e:

*/5 * * * * /usr/local/bin/sync-varnish-redirects.sh &gt;&gt; /var/log/varnish-redirect-sync.log 2&gt;&amp;1

Advanced: Prefix Matching with VCL

For prefix-based redirects, modify the sync script to use VCL string matching:

# In sync script, for prefix matches:
cat &gt;&gt; "${TEMP_FILE}" &lt;&lt;VCL_RULE
    if (req.url ~ "^${source}") {
        return (synth(${status_code}, "${destination}"));
    }
VCL_RULE

Note: VCL uses PCRE (Perl Compatible Regular Expressions). The ~ operator performs regex matching.

Advanced: Query String Handling

To match redirects regardless of query strings:

sub vcl_recv_redirects {
    # Strip query string for redirect matching
    if (req.url.path == "/old-path") {
        return (synth(301, "https://example.com/new-path"));
    }
}

Use req.url.path instead of req.url to ignore query strings.

Pattern 2: Real-Time Backend Proxy (Advanced)

Important Caveat: VCL does not have a built-in HTTP client for making external API calls during request processing. Real-time lookups require either:

vmod_curl (requires compilation and installation)
A separate redirect-checking backend (proxy approach)

We'll show the proxy approach as it requires no VMODs.

How It Works

Varnish routes redirect checks to a separate backend
The backend makes API calls to the Edge Query API
The backend returns redirect information via HTTP headers
Varnish processes the response and issues redirects

Step 1: Create Redirect Proxy Backend

Create a simple redirect proxy using Node.js (or any language):

// redirect-proxy.js
const http = require('http');
const https = require('https');

const API_URL = 'https://api.3xx.app';
const PROJECT_ID = process.env.REDIRECT_PROJECT_ID || 'YOUR_PROJECT_ID';
const API_KEY = process.env.REDIRECT_API_KEY || 'YOUR_API_KEY';

const server = http.createServer(async (req, res) =&gt; {
    const path = req.url;

    // Query Edge API
    const apiUrl = `${API_URL}/v1/lookup?projectId=${PROJECT_ID}&amp;path=${encodeURIComponent(path)}`;

    https.get(apiUrl, {
        headers: {
            'X-API-Key': API_KEY,
            'Accept': 'application/json'
        }
    }, (apiRes) =&gt; {
        let data = '';

        apiRes.on('data', chunk =&gt; data += chunk);
        apiRes.on('end', () =&gt; {
            try {
                const result = JSON.parse(data);

                if (result.redirect &amp;&amp; result.redirect.destination) {
                    // Redirect found - return info via headers
                    res.writeHead(200, {
                        'X-Redirect-Found': 'true',
                        'X-Redirect-Destination': result.redirect.destination,
                        'X-Redirect-Status': result.redirect.type === 'temporary' ? '302' : '301'
                    });
                    res.end('redirect');
                } else {
                    // No redirect found
                    res.writeHead(200, {
                        'X-Redirect-Found': 'false'
                    });
                    res.end('no-redirect');
                }
            } catch (err) {
                res.writeHead(500);
                res.end('error');
            }
        });
    }).on('error', (err) =&gt; {
        res.writeHead(500);
        res.end('error');
    });
});

server.listen(8888, () =&gt; {
    console.log('Redirect proxy listening on port 8888');
});

Run the proxy:

node redirect-proxy.js

Step 2: Configure Varnish to Use Proxy Backend

Edit /etc/varnish/default.vcl:

vcl 4.1;

# Origin backend
backend origin {
    .host = "192.168.1.10";
    .port = "8080";
}

# Redirect proxy backend
backend redirect_proxy {
    .host = "127.0.0.1";
    .port = "8888";
    .connect_timeout = 1s;
    .first_byte_timeout = 2s;
}

sub vcl_recv {
    # Save original URL and backend
    set req.http.X-Original-URL = req.url;
    set req.http.X-Original-Backend = "origin";

    # Route to redirect proxy
    set req.backend_hint = redirect_proxy;
    return (pass);
}

sub vcl_backend_response {
    # Check if redirect was found
    if (beresp.http.X-Redirect-Found == "true") {
        # Store redirect info
        set bereq.http.X-Redirect-Destination = beresp.http.X-Redirect-Destination;
        set bereq.http.X-Redirect-Status = beresp.http.X-Redirect-Status;

        # Trigger synthetic redirect response
        return (synth(std.integer(beresp.http.X-Redirect-Status, 301)));
    }

    # No redirect - fetch from origin
    if (bereq.http.X-Original-Backend == "origin") {
        set bereq.backend_hint = origin;
        return (retry);
    }

    return (deliver);
}

sub vcl_synth {
    if (resp.status == 301 || resp.status == 302) {
        set resp.http.Location = req.http.X-Redirect-Destination;
        set resp.reason = "Moved";
        return (deliver);
    }

    return (deliver);
}

Important Limitations:

This adds latency (network roundtrip to proxy + API)
More complex error handling required
Proxy becomes a single point of failure
Not recommended for production - use sync pattern instead

Inline verification: Test the VCL:

varnishd -C -f /etc/varnish/default.vcl

This pattern is not recommended for production. It's shown here for completeness.

Fallback Behavior

Sync + VCL Pattern

Missing VCL include file: Varnish will fail to start if the include file doesn't exist. Always create an empty default file:

cat &gt; /etc/varnish/redirects.vcl &lt;&lt;'EOF'
sub vcl_recv_redirects {
    # No redirects configured
}
EOF

VCL compilation errors: If the generated VCL has syntax errors, the varnishadm vcl.load command will fail, and the previous VCL version will remain active. Check sync script logs.

Empty redirect list: If the API returns no redirects, an empty subroutine is generated (safe - no redirects processed).

Backend Proxy Pattern

Proxy unreachable: Varnish will use backend health checks. If the proxy fails, requests can be routed directly to origin based on your vcl_backend_error configuration.

Timeout: Configure appropriate timeouts in backend definition to prevent hanging requests.

Invalid response: Check for X-Redirect-Found header. Missing or invalid headers should fall through to origin.

Testing

Test Exact Match Redirect

curl -I http://localhost/old-path

Expected:

HTTP/1.1 301 Moved
Location: https://example.com/new-path

Test Non-Existent Path

curl -I http://localhost/does-not-exist

Expected:

HTTP/1.1 200 OK
(response from backend)

Verify VCL Reload

Update a redirect in your dashboard
Wait for sync (or run /usr/local/bin/sync-varnish-redirects.sh manually)
Check VCL file: grep "updated-path" /etc/varnish/redirects.vcl
Test redirect: curl -I http://localhost/updated-path

Debug Redirect Flow with varnishlog

Varnish provides powerful logging for debugging:

# Watch all requests
varnishlog

# Filter for specific URL
varnishlog -q "ReqURL eq '/old-path'"

# Show only redirects (synth responses)
varnishlog -g request -q "RespStatus eq 301 or RespStatus eq 302"

# Show VCL decision points
varnishlog -i VCL_call,VCL_return

Check Active VCL

# List all loaded VCLs
varnishadm vcl.list

# Show active VCL
varnishadm vcl.list | grep active

Test VCL Syntax

Before deploying changes:

varnishd -C -f /etc/varnish/default.vcl

This compiles VCL to C code. Syntax errors will be shown.

Troubleshooting

VCL Compilation Fails

Problem: varnishd -C -f /etc/varnish/default.vcl shows errors

Common causes:

Syntax errors in generated VCL: Check /etc/varnish/redirects.vcl for malformed conditionals
Missing include file: Ensure redirects.vcl exists before starting Varnish
C compiler errors: VCL compiles to C. Check for incompatible VCL syntax

Solution:

# Check VCL syntax
varnishd -C -f /etc/varnish/default.vcl 2&gt;&amp;1 | less

# Verify include file exists
ls -l /etc/varnish/redirects.vcl

# Test with minimal VCL
cat &gt; /tmp/test.vcl &lt;&lt;EOF
vcl 4.1;
backend default { .host = "127.0.0.1"; .port = "8080"; }
EOF
varnishd -C -f /tmp/test.vcl

Redirects Not Working

Problem: Requests pass through to backend instead of redirecting

Solution:

Check VCL is loaded: varnishadm vcl.list | grep active

Verify redirect subroutine is called: Add debug logging:

sub vcl_recv {
    std.log("Calling redirect subroutine");
    call vcl_recv_redirects;
}

Check varnishlog: varnishlog -q "ReqURL eq '/old-path'"
Verify VCL syntax: cat /etc/varnish/redirects.vcl

VCL Reload Fails

Problem: varnishadm vcl.load fails

Solution:

Check new VCL syntax: varnishd -C -f /etc/varnish/default.vcl
View detailed error: varnishadm vcl.load test_config /etc/varnish/default.vcl
Previous VCL remains active: Failed loads don't affect running config
Check sync script logs: tail -f /var/log/varnish-redirect-sync.log

Empty Redirect File on Startup

Problem: Varnish fails to start with "include file not found"

Solution: Create a default empty file in your Varnish systemd service:

# Create override
systemctl edit varnish

# Add:
[Service]
ExecStartPre=/bin/bash -c 'test -f /etc/varnish/redirects.vcl || echo "sub vcl_recv_redirects {}" &gt; /etc/varnish/redirects.vcl'

Backend Proxy Pattern Issues

Problem: Proxy backend not responding

Solution:

Check proxy is running: curl -v http://127.0.0.1:8888/test-path
Check backend health: varnishadm backend.list

Add backend health checks:

backend redirect_proxy {
    .host = "127.0.0.1";
    .port = "8888";
    .probe = {
        .url = "/health";
        .interval = 5s;
        .timeout = 1s;
        .window = 5;
        .threshold = 3;
    }
}

Monitor with varnishlog: varnishlog -g raw -i Backend_health

High Latency

Problem: Requests are slow

Likely cause: You're using the backend proxy pattern (real-time lookups)

Solution:

Switch to sync + VCL pattern for zero-latency lookups
If you must use real-time:
- Add caching in the proxy
- Reduce API timeout
- Use multiple proxy instances with load balancing

Memory Usage with Large Redirect Sets

Problem: Varnish using too much memory

Context: VCL code is compiled to C and loaded into memory

Solution:

For <10,000 redirects: VCL conditionals are fine
For 10,000+ redirects: Consider:
- Using vmod_selector for hash-based lookups
- Splitting redirects across multiple VCL files
- Using external lookup service (backend proxy pattern)
Monitor VCL memory: varnishadm vcl.list shows loaded VCLs

For more troubleshooting tips, see the Troubleshooting Guide.

Performance Considerations

Sync + VCL Pattern:

Lookup time: O(n) for if/elseif chains (VCL evaluates sequentially)
For large redirect sets (>1000), consider VMOD alternatives
Memory usage: ~100 bytes per redirect in compiled VCL
Network overhead: Zero (lookups are pure VCL)
Sync delay: 5 minutes (configurable)
VCL reload: Hot reload, zero downtime

Backend Proxy Pattern:

Lookup time: Network RTT + API response time (~50-100ms)
Memory usage: Minimal
Network overhead: One API call per unique path
Freshness: Real-time
Single point of failure (proxy process)

Recommendation: Use sync + VCL pattern for production. It's simple, fast, and reliable. The backend proxy pattern is shown for educational purposes but not recommended due to complexity and latency.

VCL Best Practices

Use Subroutines for Organization

sub vcl_recv {
    call vcl_recv_redirects;
    call vcl_recv_normalize;
    call vcl_recv_auth;
    return (hash);
}

Handle Edge Cases

sub vcl_recv_redirects {
    # Normalize URL before checking redirects
    # Remove trailing slash
    if (req.url ~ "/$" &amp;&amp; req.url != "/") {
        set req.url = regsub(req.url, "/$", "");
    }

    # Now check redirects
    if (req.url == "/old-path") {
        return (synth(301, "https://example.com/new-path"));
    }
}

Optimize for Common Paths

Put frequently-accessed redirects at the top of the if/elseif chain:

sub vcl_recv_redirects {
    # Most common redirect first
    if (req.url == "/homepage") {
        return (synth(301, "https://example.com/new-homepage"));
    }

    # Less common redirects...
}

Clean Up Headers

sub vcl_deliver {
    # Don't expose internal headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;
    unset resp.http.X-Redirect-Found;
    unset resp.http.X-Redirect-Destination;
    return (deliver);
}

Next Steps

Configure nginx integration for comparison
Set up HAProxy integration for load balancer integration
Explore redirect types to understand temporary vs permanent redirects
Learn about VCL basics in the official Varnish documentation

Varnish

On this page