Batch Domain Categorization API - Classify Whole Domain Lists at Once

Why Send Domains in Batches?

Hand us up to 100 domains in a single call and they are categorized in parallel against an index of more than 100 million domains mapped to 59 content categories.

Parallel Batches

Up to 100 domains per call, fanned out across workers so wall-clock time barely moves as the list grows.

Fire and Forget

Queue a job, walk away, and let a webhook tell your endpoint the moment the results are ready.

Straight From CSV

Push a spreadsheet export at the endpoint and pick whichever output format your pipeline already consumes.

100M+

Indexed Domains

100

Per Batch

<50ms

Each Domain

59

Send Domains However You Have Them

Four ingest paths, none of which require you to preprocess anything first

1

Plain JSON Arrays

Post an array of domain strings in the request body – the quickest route when your application is already holding the list in memory. A single request takes up to 10,000 domains. Every entry is normalized on arrival, with protocols, paths and query parameters stripped away until only the registrable domain remains, and repeated entries are collapsed server-side so the same lookup is never run twice.

2

Spreadsheet and CSV Exports

Upload whatever CSV your log tooling, BI export or database dump produced. We read the header row looking for likely column names – "domain," "url," "hostname," "site" – and pick the right one on our own; if the file has no header at all, pass the column index instead. Anything up to 50 MB is answered inline, while larger uploads roll over to a background job with a progress counter and a webhook once it finishes.

3

One Domain per Line

The simplest payload we accept: raw text, a single domain on each row, ready to pipe out of a shell. It drops into Unix workflows without ceremony, so pulling hostnames out of DNS logs and pushing them through curl to the batch endpoint is a one-liner. Rows beginning with # count as comments and empty rows are skipped, which makes this the natural choice for cron jobs and scripted refreshes.

4

Full URLs, Unmodified

When your source rows are complete web addresses rather than bare hostnames, send them exactly as they are. Each one is resolved down to its registrable domain, quietly discarding subdomains, ports, paths, query strings and fragments on the way. Proxy logs, exported browser history and crawler output all work untouched, and every result carries the original URL back with it so you can join it to your own records.

Background Jobs and Callback Notifications

Inline Responses

Anything up to 10,000 domains can be answered on the connection you opened, which stays alive until the work is finished. Expect somewhere between 5 and 30 seconds depending on how long the list runs. A progress header goes out at intervals so impatient HTTP clients do not time out mid-job. Reach for this mode in dashboards, interactive tools and anywhere the caller has to block until it has answers.

Queued Jobs

Past 10,000 domains, or whenever you ask for it explicitly, the endpoint replies straight away with a job ID and a status URL. From there you can poll for a completion percentage, or hand over a webhook URL and simply wait for the POST to arrive. That notification carries a download link for the finished results, summary counts, and a list of anything that went wrong along the way.

Setting Up Callbacks

Callbacks fire at four points in a job's life: accepted, 50% complete, processing finished, and results available for download. Every POST carries an HMAC-SHA256 signature header derived from your API secret, so your receiver can prove the message really came from us. A delivery that fails is retried up to five times with exponential backoff. A single job can register several webhook URLs, which is useful when you want redundancy or need to wake up more than one downstream system.

Limits, Quotas and How Big to Make a Batch

Published ceilings and clear headers, so capacity planning stays arithmetic rather than guesswork

Ceilings per Request

A synchronous call tops out at 10,000 domains; a queued job will swallow up to 1 million. Those ceilings exist to keep processing times predictable and to stop any one account crowding out everybody else. Go over and the response is a 413, along with a hint on how to split the payload up.

Request Pacing

Batch traffic is metered on its own counter, separate from single-domain lookups. A standard plan permits 60 batch requests a minute alongside 10 concurrent async jobs; enterprise agreements can raise concurrency as far as 100 simultaneous jobs. Remaining quota and the reset timestamp travel in headers on every response.

How Quota Is Counted

Your monthly allowance is pooled across single and batch lookups alike. Billing is per unique domain, so a hostname that turns up in five separate batches during one cycle still costs a single unit. Usage dashboards and configurable threshold alerts show where you stand well before you hit the ceiling.

Picking a Batch Size

The fact that 10,000 domains fit in one synchronous call does not mean they should go that way. Throughput actually peaks somewhere between 1,000 and 5,000 per batch, because smaller payloads come back faster and let you keep several requests in flight from the client. A handful of medium batches running side by side will beat one maximum-size batch nearly every time.

What Comes Back, and in What Shape

Choose the serialization your downstream tooling already speaks

JSON Objects

By default you receive a JSON object keyed by domain. Every value holds the primary category, the classification detail, each matching category with its own score, a category level indicator, and a timestamp showing when the record was last refreshed. The nesting mirrors our full 59-category taxonomy, IAB tier mappings included.

CSV for Spreadsheets and Warehouses

Ask for CSV when the results are headed straight into a spreadsheet, a relational table or a warehouse loader. Rows carry the domain, primary category, classification detail, any secondary categories and the category level, with header names chosen to match the conventions most BI tools expect. CSV output from async jobs arrives as a gzipped download, which trims transfer size by as much as 85 percent on large result sets.

NDJSON for Streaming

Newline-delimited JSON emits one complete object per line, so your consumer can start working long before the final byte lands. It pairs naturally with streaming parsers and with jq at the command line. Because each line stands on its own, a response cut short is still perfectly parseable up to the break, which is exactly what you want when results feed directly into another processing step.

Enriched Records

Turn on enrichment and every record gains context beyond the label itself: how old the domain is, who registered it, a summary of its DNS records, SSL certificate status, a history of category changes, and other domains sitting on the same infrastructure. This is the mode for content classification workflows where a category alone will not settle the question and an analyst needs the surrounding picture to decide.

Where Batch Classification Earns Its Keep

Three patterns that turn up again and again across customer deployments

Security Operations Triage

SOC teams push indicator lists, classification feeds and extracted suspicious domains through the batch endpoint to sort signal from noise. Anything landing in unsafe content or Command-and-Control is escalated on its own, while benign labels drop out of the analyst review queue entirely. The same approach scales to commercial research, where profiling web footprints in bulk is how analysts find companies with owners nearing retirement.

Brand Safety in Ad Tech

Ad platforms run their publisher inventory through the batch API so a campaign never lands beside content the advertiser would object to. A whole exchange inventory can be categorized overnight, tagging anything in Adult, Violence or Hate Speech so demand-side platforms have their exclusion lists in place before the first bid goes out.

Log Review and Compliance Audits

Compliance staff dump DNS query or proxy access logs, reduce them to unique domains and classify the whole set in one pass to surface policy violations. The monthly report that comes out of it shows how traffic splits across categories, highlights every hit on a restricted label, and makes long-run shifts in browsing behaviour visible across the organization.

Making Batch Pipelines Resilient

In production, generate your own request ID for every submission so that a retry after a network hiccup re-uses the job already created rather than spawning a duplicate. The same discipline carries over to any high-volume workflow, including an acquisition pipeline for aging business owners chewing through thousands of company domains. Break long lists into chunks of 5,000 before sending, which lets several batches process side by side, and persist the job ID that comes back so a restarted worker can still find its results.

Recurring work, whether that is a nightly log sweep or a weekly pass over a classification feed, should run in incremental mode, which skips anything already seen in an earlier batch. Each account keeps its own domain cache with a TTL you control, so resubmitted domains come straight back from cache and cost no quota at all. The practical effect is that a daily job only ever pays for the domains that are genuinely new.

Calling the Batch Endpoint

A handful of lines in whichever language you already work in

Working Through a Domain List

Hand the endpoint a list of hostnames and read every category back out of one response.

import requests

API_URL = "https://webfilteringservice.com/api/v1/classify"
API_KEY = "YOUR_API_KEY"

# The list we want categorized
domains = [
    "google.com",
    "facebook.com",
    "gambling-site.xyz",
    "github.com",
    "adult-content.xxx"
]

# Walk the list, keeping each answer
results = {}
for domain in domains:
    response = requests.post(API_URL, json={
        "api_key": API_KEY,
        "query": domain
    })
    data = response.json()
    results[domain] = {
        "category": data.get("primary_category", "Unknown"),
        "risk": data.get("risk_level", "low")
    }

# Dump what we found
for domain, info in results.items():
    print(f"{domain}: {info['category']} (risk: {info['risk']})")

const API_URL = 'https://webfilteringservice.com/api/v1/classify';
const API_KEY = 'YOUR_API_KEY';

const domains = [
    'google.com',
    'facebook.com',
    'gambling-site.xyz',
    'github.com',
    'adult-content.xxx'
];

async function bulkClassify(domainList) {
    const results = await Promise.all(
        domainList.map(async (domain) => {
            const res = await fetch(API_URL, {
                method: 'POST',
                headers: { 'Content-Type': 'application/json' },
                body: JSON.stringify({ api_key: API_KEY, query: domain })
            });
            const data = await res.json();
            return { domain, category: data.primary_category, risk: data.risk_level };
        })
    );
    return results;
}

bulkClassify(domains).then(results => {
    results.forEach(r =>
        console.log(`${r.domain}: ${r.category} (risk: ${r.risk})`)
    );
});

# Loop over a handful of domains
for domain in google.com facebook.com gambling-site.xyz github.com; do
  curl -s -X POST "https://webfilteringservice.com/api/v1/classify" \
    -H "Content-Type: application/json" \
    -d "{\"api_key\": \"YOUR_API_KEY\", \"query\": \"$domain\"}" \
    | jq "{domain: .domain, category: .primary_category, risk: .risk_level}"
done

# A single response looks like this:
# {
#   "domain": "gambling-site.xyz",
#   "category": "Gambling",
#   "risk": "high"
# }

<?php
$apiUrl = 'https://webfilteringservice.com/api/v1/classify';
$apiKey = 'YOUR_API_KEY';

$domains = ['google.com', 'facebook.com', 'gambling-site.xyz', 'github.com'];

foreach ($domains as $domain) {
    $ch = curl_init($apiUrl);
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
        'api_key' => $apiKey,
        'query'   => $domain
    ]));
    curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $result = json_decode(curl_exec($ch), true);
    curl_close($ch);

    echo "$domain: {$result['primary_category']} (risk: {$result['risk_level']})\n";
}
?>

Got a Domain List to Get Through?

From a few thousand rows up to a few million, the batch endpoint works against an index of 100 million categorized domains.

Try a Lookup View Plans Grab a Sample CSV

Batch Domain Categorization at Scale