JayBizzle/Crawler-Detect

crawlerdetect.io

About

CrawlerDetect is a PHP library for detecting bots, crawlers and spiders via the User-Agent and HTTP_FROM headers. It currently recognises thousands of user agents and is updated regularly.

Installation

composer require jaybizzle/crawler-detect

Usage

use Jaybizzle\CrawlerDetect\CrawlerDetect;

$CrawlerDetect = new CrawlerDetect;

// Check the user agent of the current visitor
if ($CrawlerDetect->isCrawler()) {
    // true if a crawler user agent was detected
}

// Pass a user agent as a string
if ($CrawlerDetect->isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')) {
    // true if a crawler user agent was detected
}

// Output the part of the user agent that matched (if any)
echo $CrawlerDetect->getMatches();

Passing headers from a request object

With no arguments, CrawlerDetect reads $_SERVER. If your headers come from somewhere else — a PSR-7 request, Symfony's HeaderBag, Swoole, or a Lambda event — pass them in directly. Both real header names (User-Agent) and PHP's SAPI names (HTTP_USER_AGENT) are understood, and values may be strings or arrays of strings.

// PSR-7 (Slim, Mezzio, Laminas, League)
$CrawlerDetect = new CrawlerDetect($request->getHeaders());

// Symfony HttpFoundation
$CrawlerDetect = new CrawlerDetect($request->headers->all());

// Swoole
$CrawlerDetect = new CrawlerDetect($request->header);

if ($CrawlerDetect->isCrawler()) {
    // ...
}

Prefer this over isCrawler($request->getHeaderLine('User-Agent')). Some crawlers — Googlebot in particular — send a genuine browser User-Agent and identify themselves in another header such as From or Sec-CH-UA. Passing the full set lets CrawlerDetect check all of them; passing a single string can only ever check one.

Contributing

If you find a bot, spider or crawler that CrawlerDetect fails to detect, please open a pull request that:

• adds the regex pattern to the $data array in src/Fixtures/Crawlers.php
• adds the failing user agent string to tests/data/user_agent/crawlers.txt

The raw/Crawlers.json and raw/Crawlers.txt files are regenerated automatically by export.php after merge — no need to touch them.

If you're not able to submit a PR, open an issue with the user agent string and we'll take it from there.

Ports & Integrations

CrawlerDetect has been ported to a number of other languages and frameworks. If you maintain a port not listed here, please open a PR.

Platform	Project
Laravel	Laravel-Crawler-Detect
Symfony 2 / 3 / 4	CrawlerDetectBundle
Yii2	yii2-crawler-detect
Node.js / ES6	es6-crawler-detect
Python	crawlerdetect
JVM (Java, Scala, Kotlin)	CrawlerDetect
.NET / .NET Core	NetCrawlerDetect
Ruby	crawler_detect
Go	crawlerdetect

Credits

Parts of this library are based on the excellent MobileDetect.

License

Released under the MIT License.