What is working process of spiders and bots?

**Rammadhur** · 11-11-2016, 02:34 AM

**NamRay** · 11-11-2016, 03:59 AM

Spider's search engine:
- First Spider retrieve a list of servers and popular sites. Spider will start searching for a particular site, it indexes from its site and follow the links (links) found within this Site. Under this method, the search engine Google will do the job quickly and spread out all the sections of the most widely used web.

- When Spider review sites (in HTML format), it noted: The words within the site & find where it's from.
The word appears in the Title tag, Meta Description .... it says it is an important part related to the user's search later. So Google it for each website will have many ways to index the index, listed the main keywords. But how does Google use will be trying to make the system go faster search so that users can search more efficiently, or both.

**jainteq** · 11-11-2016, 04:10 AM

A Search Engine Spider (also known as a crawler, Robot, SearchBot or simply a Bot) is a program that most search engines use to find what’s new on the Internet.

**william1194** · 11-11-2016, 06:46 AM

To know how spiders work, it's practical to think of them as automatic data searching robots. As I've said, spiders passage the every website to find as many new or updated web pages and links as possible. When you submit your web pages to a search engine at the "Submit a URL" page in webmaster tool, you will be added to the spider's list of web pages to visit on its next search mission out onto the internet. Your web pages could be found, even if you didn't submit them. Spiders can find you if your web page is linked from any other web page on a "known" web site.
When spider reaches at your webpage, it head looks for a robots.txt file that used to tell spiders which areas of your site to be index and which one not. The next step of spider is to gather outbound links from the page. Spiders track links from one page to another page. This is the original indication behind spiders.