robots.txt Checkerfree · runs locally
Check whether a URL path is allowed or disallowed for a given user-agent before you crawl it. Implements the Google robots.txt matching rules: longest-match wins, Allow beats Disallow on ties, wildcards and end anchors supported. Also lists crawl-delay and sitemaps.
Ad
About this tool
Check whether a URL path is allowed or disallowed for a given user-agent before you crawl it. Implements the Google robots.txt matching rules: longest-match wins, Allow beats Disallow on ties, wildcards and end anchors supported. Also lists crawl-delay and sitemaps. Everything happens in your browser with plain JavaScript. No account, no upload, no limits.
Other tools
- HTML to Markdown Converter — Paste scraped HTML, get clean Markdown or plain text.
- CSS Selector Tester — Test selectors against pasted HTML and see every match instantly.
- Regex Data Extractor — Pull emails, prices, URLs, or any pattern out of text. Export CSV.
- JSON to CSV Converter — Flatten scraped JSON arrays into spreadsheet-ready CSV.
- User-Agent Generator — Fresh, realistic browser User-Agent strings for rotation.
- Proxy Cost Calculator — Estimate monthly proxy spend by pages, page size, and provider rate.