User Agent

Show my user agent

Available to DeepCrawl only

Custom Request Headers

Custom User-agent

#01 Robots Agent Case is Ignored

#02 Robots Rule Case is Ignored

#03 Meta Noindex in Body

#04 Robots Comma Separated Rules with Spaces

#05 Robots Comma Separated Rules without Spaces

#06 Robots Comma Separated Rules with Whitespace

#07 Robots Space Separated Rules

#08 Agent Specific Rule Match

#09 Agent Specific Rule Not Matched

#10 Robots Multiple Specific Agent Ruled Summed

#11 Robots Specific Agent Rule & All Crawlers Rule are Summed

#12 Robots All Rule Has No Effect

#13 Robots None Rule Equivalent Noindex & Nofollow

#14 Robots Meta and X-Robots Combined

#15 Robots Meta and X-Robots Most Restrictive Rule Wins

#16 Robots Agent Specific X-Robots-Tag Match

#17 Robots Agent Specific X-Robots-Tag Not Match

#18 Robots Noarchive

#19 Robots Nositelinkssearchbox

#20 Robots Nosnippet

#21a Robots Indexifembedded with Noindex

#21b Robots Indexifembedded without Noindex

#22 Robots Max-snippet:0

#23 Robots Max-snippet:20

#24 Robots Max-snippet:-1

#25 Robots Max-snippet:invalid-value

#26 Robots Max-image-preview:none

#27 Robots Max-image-preview:standard

#28 Robots Max-image-preview:invalid-value

#29 Robots Max-image-preview:large

#30 Robots Notranslate

#31 Robots Noimageindex

#32 Robots Txt Noindexed