Loading…
Loading…
Create a robots.txt file: block paths, block AI training crawlers like GPTBot and ClaudeBot, add sitemaps, and test URLs against the rules.
Also covers: Robots.txt tester · Block AI crawlers
Full URLs, one per line.
User-agent: *Disallow: Test a path against these rules
✓ Allowed — no rule matches in the User-agent: * group
Allow everything, block everything, or list the paths to keep crawlers out of. Wildcards (*) and end anchors ($) follow Google's syntax.
One click blocks the crawlers that collect training data, like GPTBot, ClaudeBot, Google-Extended, and CCBot, without affecting Google or Bing search.
Try paths against the rules for any crawler: the tester applies the longest-match rule that Google uses. Then save the file as robots.txt at your site's root.
Not reliably. It stops crawling, but a blocked page can still be indexed without its content if other sites link to it. To keep a page out of search results, allow crawling and use a noindex meta tag instead.
No. Google-Extended is a separate control token for using your content in Gemini models. Googlebot, which crawls for Search, isn't affected.
No. It's a public file, and it's a request that reputable crawlers honor, not access control. Never list private URLs in it hoping to hide them; protect them with authentication.
No. The code is generated in your browser, and nothing you type is sent to a server or stored.
Encoding
Percent-encode text for URLs and query strings, or decode %20-style escapes back to text. Handles UTF-8, + as space, and double encoding.
Inspect
Break a URL into protocol, host, port, path, query, and fragment, and list every query parameter decoded. Export as JSON.
Analytics
Add utm_source, utm_medium, utm_campaign, and other UTM parameters to a URL for tracking campaigns in Google Analytics and other tools.
Privacy
Strip utm_*, fbclid, gclid, and other tracking parameters from links, and unwrap Google and Facebook redirect links. Clean one link or a whole message.
SEO
Turn titles into clean URL slugs: lowercase, hyphenated, accents transliterated (é → e), with optional length limits and stop-word removal.
Naming
Generate brandable domain name ideas from your keywords: prefixes, suffixes, pairings, and blends across .com, .io, .ai, .dev, and more.