The General Search Engine Architecture

Google searches this immense collection of web pages often in less than half a second.

For this assignment, I was allowed to improvise on a provided base code to develop a functioning web crawler. The web crawler needed to accept a starting URL and then develop a URL frontier queue of “out links” to be further explored. The crawler needed to track the number of URLs and stop adding them once the queue had reached 500 links. The crawler needed to also extract text and remove HTML tags and formatting. The assignment instructions offered using the BeautifulSoup module to achieve those goals, which I chose to do. Finally, the web crawler program needed to report metrics including the number of documents (web pages), the number of tokens extracted and processed, and the number of unique terms added to the term dictionary.

Bi et al \cite{Rec:Bi} provides ranked related entities to the user query along with the results of the main entity. In order to do this, this articles makes use of user's search history, click history and knowledge base. A matrix is created comprising of the user information which connects to the entities along with the ranking, click results. A tri-linear function\cite{Rec:Bi} is defined mapping these details and which will be used to rank the related entities

This algorithm was developed as part of the research project supported by National Science Foundation grant provided to Stanford University. The patent was filed by Stanford University in 1998 and is still owned by Stanford University, however Google has exclusive rights to use the patent. PageRank is trademark of Google Inc. and Google pays royalties to Stanford University for using the patent in Google search.

More importantly, she mainly covers why Google is the most efficient search engine and how it operates more accurately than other engines and Web browsers. Kraft shares the same positive outlook on Google as the preferred search engine as is evidenced in this paper.

Through the methodology proposed, we aspire to achieve a more efficient technology for generating keywords and finding more accurate data from the search engine. By saving physical memory and storing only what is important rather than all the data from a random website. Also, due to this we may achieve faster response time. So, here we can conclude that the proposed system may be more better than the previous systems

Due to the increased competition from Yahoo and Microsoft, it has posed a great threat to Google Company. Entry of such like companies in the market has been relatively easy and even offering similar services to the consumers is no longer a problem as unlike a while back, technology has greatly improved. With an improvement in technology, creating sites that help people access information from the internet has been quite cheap and all it calls for is the creativity of a company. Google, however, has been able to remain at the top of the chart. They have been able to come up with a user-friendly search engine for their clients, one that is easy to understand as well as easy to use. Also, Google has been able to cut on cost as they are able to create web pages using UNIX web servers which are relatively cheap. This makes it possible for them to minimize the cost of input and hence maximizing their profit margins. This makes it possible for them to hire qualified personnel, offer quality services, advertise their products, and even offer promotions, helping them overcome the competition in the market structure.

Google’s search engine allows users to input and submit data online. In return, the user would receive relevant search results. Behind the scenes upon the submission, web crawlers scan through billions of pages and link keywords from a user’s data to the publish data on the web. Their PageRank technology ranks these pages by the number and popularity of other sites that link to the page. This provides the user with accurate and popular results. Google search engines generated high revenues between advertising on its websites and selling its technology to other sites.

The Internet is vast. To the casual user, the Internet represents the collection of those websites accessible via search engines such as Google or Bing. Search engines function by utilization of a web crawler which locates and indexes linked pages that are then provided as search results when it meets a particular search’s criteria. But, those web crawlers are only able to identify static pages, leaving out the dynamic pages of the deep web. Imagine a commercial fishing trawler on the open ocean pulling in its catch. The trawler only gathers fish from just barely below the surface and misses the massive

6 Google quickly rose to prominence................................................................................................................................ 6 Other search engines have struggled to compete with Google .........................................................................................7 Google differs from its competitors ................................................................................................................................ 7 Other companies began to use Google for their search technology ..............................................................................7 Google’s prominent position is reflected in its market share .............................................................................................8 Google’s market share is largest in Europe ...................................................................................................................8 Google’s revenues come largely from advertising...........................................................................................................10 Google introduced AdWords ........................................................................................................................................ 10 Conclusions......................................................................................................................................................................... 11 Can Google

Search engines got their grip on the internet in the early 1990s, the most popular of them being browsers such as Google or Bing. These websites channel information

Crawler must avoid the overloading of Web sites or network links while doing its task. Unless it has unlimited computing resources and unlimited time, it must carefully decide what URLs to scan and in what order as it deals with huge volumes of data .Crawler must decide how frequently to revisit pages it has already seen, in order to keep its client informed of changes on the

* Google’s PageRank technology “measures the importance of different Wed pages by solving an equation with more than 500 million variables and 2 billion terms.”

Following the success of Netscape and its web browser, Internet became a resource and communication platform idolized by many IT students in the universities. What started off as a hobby-cum-research[1] work by Jerry Yang (now Chief of Yahoo!) and David Filo (Co-founder of Yahoo!) for their Ph.D. dissertations; has evolved and became an Internet sensation over time. What they did was to compile all their favourite web links to form an online directory for easy navigation in the World Wide Web. The duo’s work immediately garnered a lot of attention from many surfers in the Internet world and before they realized it, Yahoo! became one of the most highly visited websites of all time. The duo saw the

Professionally, Google is known as a company based in California that is labeled as an internet company which is multi-national. It provides online searching, as well as cloud computing, software, and advertising. The company actually didn 't start off as a company, but rather as a research project back in 1996. The project was being conducted by Sergey Brin and Larry Page who at the time were studying at Standford University as PhD students. At the time, in internet-land, the search engines that existed operated where they ranked the results by counting the number of times keywords results were on a page. The two students came with a better idea (called PageRank at the time), that looked at relationship between websites. It would rank websites by determining it 's relevance, which was based on the importance of pages, and the number of pages, and how it linked back to the main website. After the idea 's creation, the two founders made the project into a business, and changed the name to "Google", which is a neat miss-spelling of the word "googol" which had significance because it stands for the number one followed by one hundred zeros, and it related to their goal because they wanted to create a search engine that offered a large quantity of information.