Integrated Caching

Varnish as the First Layer

• Varnish runs in front of the content retriever (which fetches content from the Hosting Layer).

• It checks if the requested CID is already cached before forwarding the request.

Cache Storage in Varnish

Varnish stores:

1. Static files (HTML, CSS, JS, images, videos)

2. Pre-fetched content for fast loading

3. Frequently accessed website CIDs

Step-by-Step Caching Process:

A. User Requests Content (example.com)

1. The request reaches Varnish (gateway1.com).

2. Varnish checks its cache memory for the CID.

B. Cache Hit (Content Found in Varnish)

1. If the content exists in cache, it is immediately served from RAM.

2. Response time: <1ms (instant page load).

C. Cache Miss (Content Not in Varnish)

1. Varnish forwards the request to the Webhash Gateway Backend.

2. The Gateway Backend queries the Hosting Layer for the CID.

3. If content is found, it is stored in Varnish and then served to the user.

Varnish Cache Configuration for Webhash

The Varnish Configuration Language (VCL) is used to control caching behavior.

vcl 4.1;

backend hosting_layer {
    .host = "localhost";
    .port = "5050"; # Webhash Gateway Backend Port
    .first_byte_timeout = 60s; # Wait for hosting layer to respond
    .connect_timeout = 5s;
    .between_bytes_timeout = 30s;
}

sub vcl_recv {
    # Extract CID from the request URL
    if (req.url ~ "^/ipfs/") {
        set req.backend_hint = hosting_layer;
    }

    # Set caching policy
    set req.http.Cache-Control = "max-age=3600"; # Cache for 1 hour
}

sub vcl_backend_response {
    # Cache only successful responses
    if (beresp.status == 200) {
        set beresp.ttl = 1h; # Keep in cache for 1 hour
    } else {
        set beresp.ttl = 0s; # Do not cache errors
    }
}

sub vcl_deliver {
    # Add debug header to indicate cache status
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }
}

Explanation of Varnish Cache Rules

1. Caches content for 1 hour to improve performance.

2. Requests are forwarded to Hosting Layer only if necessary.

3. Responses are tagged with X-Cache (HIT/MISS) for debugging.

Cache Purging for New Website Updates

When a new CID is published, gateways can invalidate the cache for a specific website.

varnishadm "ban req.url ~ /ipfs/QmOldCID"