Operator
Firecrawl
Firecrawl runs 2 bots across 2 agent types. Each one is a separate user-agent so you can allow or block them independently.
Firecrawl is operated by Firecrawl. Generic programmatic data extraction with intent that varies case-by-case. Worth checking the operator before blocking, some are legitimate research, some are abusive.
FirecrawlAgent is operated by Firecrawl. Aggregates structured web data and resells it to AI products and dataset marketplaces. Crawl pattern is broad and persistent, often touching long-tail URLs other crawlers miss.
Live data · via Cloudflare Radar
See how Firecrawl's bots show up in real internet traffic
Cloudflare Radar publishes a continuously-updating view of AI bot activity across the websites Cloudflare protects. You can see which user-agents drive the most requests, how that share shifts week to week, and which industries each bot crawls hardest. The data refreshes on Cloudflare's side, so the link below is always current.
Open Cloudflare Radar AI InsightsWhat you'll find there
- ·User-agent share for every tracked AI bot, including Firecrawl's
- ·Robots.txt and crawl-purpose breakdowns per operator
- ·Industry and content-type targets for each crawler
- ·Time-series trends going back several months