How It Works

AI systems now interact directly with publisher content — reading, summarizing, and generating answers for users. For publishers, this makes AI traffic a new operational category that needs to be understood, classified, and managed.

paywalls.net provides the infrastructure to do that — from initial visibility through to traffic management and revenue optimization.

Step 1 — See It

Before you can manage AI traffic, you need to see it.

paywalls.net offers two low-friction ways to get started:

• robots.txt Check: Validate your current configuration and get AI-specific recommendations. No integration required.
• Log Analysis: Upload a sample of your web server logs and get a breakdown of AI crawler and automation traffic — who, how much, and what they're requesting.

Both tools are available immediately, with no account or integration needed.

Step 2 — Classify It

Once you have visibility, the next step is real-time classification.

paywalls.net integrates at the CDN or server level to classify traffic as it arrives — identifying AI crawlers, agents, automation, and human visitors using IP analysis, behavior patterns, HTTP headers, and crawler signature databases.

Classification categories include:

• Search crawlers (Googlebot, Bingbot) — indexing for search results
• AI training crawlers (GPTBot, CCBot, Anthropic) — harvesting content for model training
• AI agents — autonomous systems retrieving content on behalf of users
• General automation — scrapers, monitors, and other non-human traffic
• Human visitors — real users browsing your site

This classification drives everything that follows — access policies, ad-auction routing, and licensing signals.

Step 3 — Set Access Policies

With classification in place, you define how each traffic category is handled.

• Allow indexing, limit summarization — separate search from AI training with granular rules
• Require attribution — set terms for AI systems that use your content in generated responses
• Set rate limits and conditions — control access frequency and enforce rules at the edge

Policies are expressed as structured configuration and enforced at the CDN level.

Step 4 — Protect Ad Revenue with VAI

Automation that enters advertising auctions without converting dilutes CPMs and erodes buyer confidence.

Validated Actor Inventory (VAI) classifies traffic before the ad auction so publishers can route impressions based on confidence:

• High-confidence human traffic → premium demand paths
• Uncertain traffic → lower-risk demand (preserving revenue without blocking)
• Automation → excluded from premium inventory

VAI works server-side and integrates with existing ad stacks. Learn more about VAI.

Step 5 — Prepare for Content Licensing

As AI systems increasingly rely on licensed content, publishers need infrastructure to participate.

paywalls.net provides:

• Domain verification and identity signals
• Machine-readable access policies
• Usage data to support licensing negotiations

Content marketplace participation is currently available via invite-only beta.

Integration

paywalls.net supports integration at the edge with major CDN providers:

• Cloudflare
• AWS CloudFront
• Fastly
• Akamai

Or start without any integration using log analysis and robots.txt tools.

Get Started

1. Analyze your logs or check your robots.txt — no account needed.
2. Create an account to set up real-time analytics and access policies.
3. Integrate at the CDN level for classification, VAI, and enforcement.