Crawl and index
Consider the World Wide Web as a network that works like a different metro station in a big city.
Each station is a unique document (usually every page of a site is a document, although these documents may also be in PDF, JPG or other files). The search engines need a way to crawl around the city, and they should mark different stations in order to find the best paths available, that is, the same links.
Link structure on the web links all pages together.
The link to automated search engine robots (known as crawlers or spiders) allows the web to access billions of related documents.
When the search engines find these pages, they decode their codes and store the selected parts in a huge database. By doing so, they can call them when they need it (when searching for a query by the user). For access to such huge data in a matter of seconds, search engine companies have created various data centers in different parts of the world.
These centers have thousands of devices that can process a large amount of information at a high speed. When someone searches for a search engine, he or she will seek information quickly and without interruption. Even one or two seconds of delay can lead to users’ dissatisfaction. That’s why search engines try to provide the users with the required speed at a faster rate.
Comments
Post a Comment