Performance For Web Documents Mining Using Nlp And Latent Semantic Indexing With Singular Value Decomposition

Launched on 15 January 2001, Wikipedia is a free encyclopedia that uses the web platform for online users to access. Boasting with over 26 million pieces of writing in 285 languages, Wikipedia has transformed to be a giant in the field of search engines optimization technology. The open source concept that it rides have made it cheap to access and a better choice for many online users. This is especially among the users who find it cumbersome to follow prolonged registration processes to access information on the internet. Any search term queried on the Google™ home page search engine will definitely give a hit from the Wikipedia site, and if not present, a prompt will request the user to create a page for such a term. In this way,

The first versions of WWW ((what most people call “The Web”))) provide means for people around the world to exchange information between, to work together, to communicate, and to share documentation more efficiently. Tim Berners-Lee wrote the first browser (called WWW browser) and Web server in March 1991, allowing hypertext documents to be stored, fetched, and viewed. The Web can be seen as a tremendous document store where these documents (web pages) can be fetched by typing their address into a web browser. To do that, two im- portant techniques have been developed. First, a language called Hypertext Markup Languag (HTML) tells the computers how to display documents which contain texts, photos, sounds, visuals (video), and animation, interactive

RI.3.5 Use text features and search tools (e.g., key words, sidebars, hyperlinks) to locate information relevant to a given topic efficiently.

For this assignment, I was allowed to improvise on a provided base code to develop a functioning web crawler. The web crawler needed to accept a starting URL and then develop a URL frontier queue of “out links” to be further explored. The crawler needed to track the number of URLs and stop adding them once the queue had reached 500 links. The crawler needed to also extract text and remove HTML tags and formatting. The assignment instructions offered using the BeautifulSoup module to achieve those goals, which I chose to do. Finally, the web crawler program needed to report metrics including the number of documents (web pages), the number of tokens extracted and processed, and the number of unique terms added to the term dictionary.

This section discuss about the common traits or ideas observed in the three research topics. Although, each of the three articles discuss a unique idea, all of them are aimed at utilizing the web data to produce better results. Web data mining is a hot research topic in the current realm of big data. These papers discuss about the utilization of the valuable user generated data from the social media or the browser cookies to provide the best user experience in order to maintain the user interest in the company's product or to take effective decisions by an individual. All the three articles propose an idea to solution the problem stated, compared their results to the existing models and showed significant improvement.

With the advent of computer technology in 1990’s the need to search large databases was increasingly becoming vital. The search engines prior to PageRank had limitations, the then most widely used algorithm used text based indexes to provide search results on World Wide Web however had limitations of improper search results as the logic used by the search engines looked at the number of occurrences of the search word in webpage which sometimes resulted in improper search results. Another technique used during the time was based on variations of standard vector space model – i.e. search based on how recent the webpage was updated and/or how close the search terms are to the

This week we went a little bit more into the actual process of constructing an inverted index and how to implement the handling of misspellings, the accuracy of retrievals based on context, and the structures of resulting indexes for the purpose of searching them. For the discussion assignment, I needed to cover the differences between BSBI and SPIMI. For our programming assignment, we needed to make some alterations to a SPIMI python program to create an index and list of documents.

Through the methodology proposed, we aspire to achieve a more efficient technology for generating keywords and finding more accurate data from the search engine. By saving physical memory and storing only what is important rather than all the data from a random website. Also, due to this we may achieve faster response time. So, here we can conclude that the proposed system may be more better than the previous systems

Although NegExpander works very well for this document, identifying all noun phrase concepts, its overall precision in identifying concepts correctly is only 93 percent. We randomly selected ten documents and used NegExpander to find the contained concepts. The documents were then manually reviewed to identify noun phrase concepts. Some errors were due to incorrect part-of-speech tagging by Jtag, and some errors were made by NegExpander itself.

Final system suggests the predicted matches based on first hits stream data; this has been achieved by two step similarity functions, which built and developed similar incidents in web platform using Lucene and Clustering libraries. The K-means clustering and logistic regression methods has been trained to be used real-time to evaluate final safety scores categorized by labels, and provided label matching in particular safety concept group(s).

Text Retrieval and techniques are modified by many commercial and many open sources in the domain of information retrieval. Text retrieval refers to the process of searching for texts, information within collections, or metadata about documents. It is assigned to answer for relevant documents, not just simple matches to patterns. When considering indexing and searching applications, users may find and consider among many products available on the market. Mainly, the products can be grouped into two categories. The first of these categories are information retrieval libraries that can be easily developed and embedded into application. The second consists of ready to apply indexing and searching applications that are basically designed to work with particular types of data, and are therefore less flexible (Molková 2011). Some of the famous libraries cover text retrieval topic are introduced in next section.

Ontology contains a set of concepts and relationship between concepts, and can be applied into information retrieval to deal with user queries.

Another criterion of information retrieval searching flexibility which was mentioned in this article is the free-text search option. Full text search refers to a technique for searching a computer-stored document or database. In a full text search, the search engine examines all of the words in every stored document as it tries to match search words supplied by the user (Wikipedia). The good example of free text search in information retrieval is a web search engines. One of the search engines using full text search technique is AltaVista. Even the free text provide flexibility to do searching, the results are not good as controlled vocabulary searching.

The main aim of this project is to research on the integration of “Natural Language Processing “ and information systems engineering to enhance query retrieval in natural language processing.

Nesi (1998) classifies dictionaries into two modes of presentation: paper-based dictionaries and computer-based dictionaries (also known as electronic dictionaries). The main difference between electronic dictionaries and paper-based dictionaries is not their information content but their retrieval system (Nesi, 2000). In a study presented by Zainab Saleh AlBulushy on E-dictionaries versus paper-based ones, she presented the main differences between both types. She used ten features to compare between the two types. These differences are presented below: