Cloudflare Integration

We've used Cloudflare workers as an example in our setup, but the general idea is the same no matter which CDN you're using. You'll run a lightweight client that talks to our "auth service" to handle the checks. The specifics of that client might change depending on your CDN, but the core process of verifying access stays consistent.

This document will walk you through how to build and set up that filter, and how to connect it to the paywalls.net system. We'll show you the Cloudflare example and explain how to adapt it for other CDNs.

Cloudflare Publisher Integration Walkthrough

• Register as a publisher with paywalls.net: Sign up for an account at paywalls.net and receive your API key and publisher ID.
• Set up a Cloudflare Worker: In your Cloudflare account, create a new Worker. This will be the filter that checks content access.
• Deploy the Code: Copy and adapt the provided code example into your Cloudflare Worker.
• Configure API Keys and IDs: In your Cloudflare Worker settings, add your API Key and Publisher ID as Secrets or Environment Variables. Then, in your code, you can access them using env.PAYWALLS_API_KEY and env.PAYWALLS_PUBLISHER_ID. This keeps your API credentials out of your code repository and makes them safer.
  • Note : Cloudflare’s wrangler utility does not support secrets management in a localhost environment. When using a local environment for development or testing you should use a .dev.vars configuration or equivalent.
• Configure and Test: Deploy the Worker and configure access to your content. The Worker will intercept requests, check with the paywalls.net "auth service," and either allow or deny access based on the bot’s authorization.

Sample Code Description

The provided code is a Cloudflare Worker designed to filter bot-like requests using the paywalls.net handler. Here's a breakdown:

• Initialization: The init function is used to initialize the paywalls.net handler for Cloudflare. This handler is configured to identify bots, log access attempts, and handle responses.
• Main Decision: The fetch function is invoked by Cloudflare Workers. It uses both paywalls.net user-agent classification and Cloudflare's bot management metadata to determine if the request is from a bot. If not, the request proceeds as normal. If it is from a bot, the Worker performs the following:
  • Token Extraction: The bot's OAuth2 access token is extracted from the "Authorization" request header.
  • Agent Check: Metadata about the agent and the authorization token are checked via an API call to paywalls.net. This verifies whether the bot is authorized to access your content.
  • Access Logging: The content access request is logged with paywalls.net. This is used for reporting and billing of bots.
  • Response Handling: If access is denied, the handler returns an appropriate response (e.g., 401 Unauthorized or 402 Payment Required) along with HTML and headers provided by paywalls.net. If access is allowed, the request is passed through to the CDN for regular processing.

Sample Code

/**
* Example publisher-hosted client code for a Cloudflare Worker that
* filters bot-like requests by using paywalls.net authorization services.
*/
import { init } from '@paywalls-net/filter';

// Initialize the paywalls.net handler for Cloudflare
const handleRequest = await init('cloudflare');

export default {
   async fetch(request, env, ctx) {
       let pw_response = await handleRequest(request, env, ctx);
       if (pw_response) {
           // If the handler returns a response, return it
           return pw_response;
       }
       return fetch(request); // Proceed to origin/CDN
   }
};

Sample Cloudflare Worker Configuration

To deploy the above code, you need to set up your Cloudflare Worker with the necessary environment variables. Here is a sample wrangler.jsonc configuration file:

{
    "name": "paywalls-bot-filter",
    "main": "index.js",
    "compatibility_date": "2024-03-29",
    "workers_dev": true,
    "env": {
        "localhost": {
            "vars": {
                "ENV": "localhost",
                "PAYWALLS_CLOUD_API_HOST": "https://cloud-api.paywalls.net",
                "PAYWALLS_CLOUD_API_KEY": "<your-api-key>",
                "PAYWALLS_PUBLISHER_ID": "<your-publisher-id>",
            }
        },
        "prod": {
            "vars": {
                "ENV": "prod",
                "PAYWALLS_CLOUD_API_HOST": "https://cloud-api.paywalls.net",
                "PAYWALLS_CLOUD_API_KEY": "<your-api-key>",
                "PAYWALLS_PUBLISHER_ID": "<your-publisher-id>",
            },
            "routes": [
                "/protected/*",
            ]
        }
    }
}

testing your Cloudflare Worker locally

To test your Cloudflare Worker locally, you can use the wrangler dev command. This command will start a local development server that simulates the Cloudflare environment. You can then make requests to your Worker as if it were deployed on Cloudflare.

# Run locally on port 8080 and reverse proxy "example.com"
npx wrangler dev --env localhost --port 8080 --local-upstream example.com