Robots.txt

What is Robots.txt?

A small file on your website that tells search engines and AI crawlers which parts they are and are not allowed to visit.

Also known as: robots

Robots.txtA small file says which parts of a site search engines may and may not visit.robots.txtallowednot

If a restriction is included by mistake, search enginesSearch engine A website used to search the internet, such as Google or Bing. or AIAI Computer programs that perform tasks normally requiring human intelligence, such as writing text or answering questions. will not read your site. This is a common cause of poor visibility.