NightshiftDirectScanner
The crawler behind the free website scan. It only visits a site when someone asks for a scan of it.
How it behaves
- It identifies itself with this user agent:
Mozilla/5.0 (compatible; NightshiftDirectScanner/1.0; +https://nightshiftdirect.com/scanner/) - It reads robots.txt before anything else, and skips anything disallowed for NightshiftDirectScanner or for all crawlers.
- It reads at most 25 pages per scan, no more than 4 at a time, and makes at most 40 requests to your site in all, redirects included.
- Every scan stops at 45 seconds, whatever it has read by then.
- Results are reused for 6 hours, so the same site is not scanned over and over.
- It does not submit forms, log in, or follow links to other sites.
How to block it
Add this to your robots.txt:
User-agent: NightshiftDirectScanner Disallow: /
The next scan of your site will stop at robots.txt.