S sadianisar New member Jun 25, 2016 #2 Robots.txt is a file in which we write commands for crawler to not visit and index the webpage which we don not want to. The file consists of two codes named ueser-agent and disallow. It is important to submit in the website for SEO.
Robots.txt is a file in which we write commands for crawler to not visit and index the webpage which we don not want to. The file consists of two codes named ueser-agent and disallow. It is important to submit in the website for SEO.
