Welcome to the Webmaster Forum.
Results 1 to 10 of 10
  1. #1
    Member
    Join Date
    Aug 2014
    Posts
    33

    What Is Robots.txt?

    Hello friends please suggest me,
    What Is Robots.txt?

  2. #2
    Member
    Join Date
    Sep 2016
    Posts
    88

    Re: What Is Robots.txt?

    robots.txt is very important file which is control the search engine bot from indexing and caching.
    Best Salesforce Training, Hadoop Bigdata Training, PMP Certification Training and many more.

  3. #3
    Member
    Join Date
    May 2016
    Location
    Florida, United State
    Posts
    97

    Re: What Is Robots.txt?

    The robots.txt file is a simple text file placed on your web server which tells webcrawlers like Googlebot if they should access a file or not.

  4. #4
    Member
    Join Date
    Jun 2015
    Posts
    53

    Re: What Is Robots.txt?

    In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. ... The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  5. #5
    Senior Member
    Join Date
    Oct 2016
    Posts
    160

    Re: What Is Robots.txt?

    A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want to be accessed by search engine crawlers. The file uses the Robots Exclusion Standard, which is a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers (such as mobile crawlers vs desktop crawlers).

  6. #6
    Member
    Join Date
    Nov 2016
    Posts
    60

    Re: What Is Robots.txt?

    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  7. #7
    Senior Member
    Join Date
    Jul 2014
    Location
    Forum
    Posts
    1,297

    Re: What Is Robots.txt?

    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS Hosting
    Cheap Dedicated Hosting | Free IPMI
    Cheap Linux VPS | Cheap Windows VPS | Unmetered Bandwidth

  8. #8
    Senior Member
    Join Date
    Jun 2014
    Posts
    396

    Re: What Is Robots.txt?

    Robots.txt file help to control the indexing of your site by search engines.

  9. #9
    Member
    Join Date
    Jul 2016
    Posts
    47

    Re: What Is Robots.txt?

    Robot.txt is document through this, you allow search engine to crawl your wenpages or not.

  10. #10
    Senior Member
    Join Date
    May 2015
    Posts
    127

    Re: What Is Robots.txt?

    As My Opinion Robot.txt is a text file. It is through this file it gives instruction to the search engine crawler about indexing and caching of a webpage file of website.

 

 

Similar Threads

  1. Which is better Robots.txt or Meta Robots Tag?
    By ShushIncalls in forum Search Engine Optimization (SEO)
    Replies: 0
    Last Post: 12-08-2016, 06:08 AM
  2. Which is better Robots.txt or Meta Robots Tag?
    By kalmakuran in forum Search Engine Optimization (SEO)
    Replies: 0
    Last Post: 04-18-2016, 03:08 AM
  3. Which is better Robots.txt or Meta Robots Tag?
    By riyakhan in forum Search Engine Optimization (SEO)
    Replies: 5
    Last Post: 01-22-2016, 04:29 AM
  4. Which one is better Robots.txt or Meta robots Tag
    By adriennerich in forum Search Engine Optimization (SEO)
    Replies: 9
    Last Post: 09-12-2014, 03:11 AM
  5. using robots.txt vs meta robots tag
    By easyarticles in forum Search Engine Optimization (SEO)
    Replies: 0
    Last Post: 08-02-2013, 07:49 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Back to top

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73