Using Parallel Processing For Large Scale Database Analysis

guidelines are an essential part of making sure the database is well constructed and serves the

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

Over the past few years, the needs for special-purpose applications that could handle large amount of data have increased dramatically. However, these applications required complex concepts of computations such as parallelizing the tasks, distributing data, and taking care of failures. As a reaction to this problem, a new abstract layer that allows us to express the simple computations we were trying to perform but hides the complex details was designed, MapReduce. This paper is an influential paper in the field of large scale data processing. It simplifies the programming model for processing large data set. The paper describes a new programming model based on lisp’s map and reduces primitives for processing large data set. In addition, the paper also describes a framework to automatically parallelize the map tasks across various worker machines.

The key functions to be implemented are Map and Reduce. The MapReduce framework operates on key and value pairs. Each Map task processes an input split block generating intermediate data of key and value format. Then, they are sorted and partitioned by key, so later at Reduce phase, pairs of the same key will be aggregated to the same reducer for further processing. Partitions from different nodes with the same key are transferred from the shuffle phase to a single node and then merged and get ready to be fed to the reduce task. The output of Reduce tasks is same format, key and value, as

Hadoop1 provides a distributed filesystem and a framework for the analysis and transformation of very large data sets using the MapReduce [DG04] paradigm. While the interface to HDFS is patterned after the Unix filesystem, faithfulness to standards was sacrificed in favor of improved performance for the applications at hand.

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

When we went to the visit the library as a class, I apprehended valuable information that I may need for my research project and annotated bibliography. Attending the library allowed me to experience the database system and meet one of our librarians on campus. It made me feel really comfortable because of how nice the librarian was and how she was very helpful. While our class was there, I learned how to format my research question, learned how to explore the UC Merced database, and how to create the direction for my research question. All of this new acquired information will help me at the end when I am doing my final assignments and connecting my research question.

For this assignment I chose the topic "Study Drugs." I used the Gale General OneFile database, from Motlow's library page, to find my two sources. The sources were also only chosen if it was peer reviewed. Using a database through the library and making sure they were peer reviewed were simple measures I took to prove the sources reliability.

Research topic was derived from the understanding of query processing in MySQL and Hadoop, the database performance issues, performance tuning and the importance of database performance. Thus, it was decided to develop a comparative analysis to observe the effectiveness of the performance of MySQL (non cluster) and Hadoop in structured and unstructured dataset (Rosalia, 2015). Furthermore, the analysis included a comparison between those two platforms in two variance of data size.

Data. It is all around every person on this earth whether they realize it or not. Throughout each and everyone’s life they collect data and have their data collected by others. Height, weight, shopping habits, health history are all examples of data that is tracked. The question is what is done with this data? People, companies, even the government analyze the data they collect and analyze it with hope of discovering new information. How they do this is particularly interesting and opens the door for a larger discussion.

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

What is database management system? Database Management System is a database program. The DBMS manage incoming data, organize it, and provided ways for the data to be modified or extract by users or other programs. This cause, most database software comes with an Open Database Connectivity (ODBC) driver that allows the database to integrate with other databases. For example, common SQL statements such as SELECT and INSERT are translated from a program's proprietary syntax into a syntax other databases can understand. Some DBMS examples include PostgreSQL, MySQL, SQL Server, Microsoft Access, Oracle, FileMaker, RDBMS, dBASE, Clipper, and FoxPro. It is a software system that uses a standard method of retrieving, and running queries on data.

Map/Reduce is a structure for preparing parallelizable issues crosswise over gigantic datasets utilizing a substantial number of PCs (hubs), all things considered alluded to as a group i.e. if all hubs are on the same local network and use the same hardware) or a framework i.e. if the hubs are shared crosswise over geologically and authoritatively conveyed systems, and use a more heterogenous hardware. Handling can happen on data saved either in a file system - unstructured or in a database - organized. Map/Reduce can exploit domain of data, preparing it on or close to the storage resources with a specific end goal to reduce the length over which it must be transmitted. Jiaxing et al. proposes SUDO, an advancement system that reasons about data partition characteristics, working properties, and data shuffling. They contend that reasoning about data partition properties crosswise phases opens up chances to reduce extravagant data shuffling. For instance, in the event that we realize that data partitions from past computation phases as of now have alluring properties for the following phase, we have the method to stay away from superfluous data shuffling steps. The fundamental obstruction to reasoning about data partition properties crosswise over processing phases is the utilization of UDFs [4]. At the point when a UDF is viewed as a "black-box", which is typically the case, we must expect conservatively that all data partition characteristics are lost subsequent to applying

The use of big data not only creating huge traffic load on internet, also changing the traffic patterns. These traffic flows are resulting performance degrade in terms of application performance and affecting the service provider’s revenue in turn. Figure 2 2 depicts the present day traffic flows ( mouse flow and elephant flow ) in data center. With the dominance of huge mobile traffic especially due to video streaming the size and shape of data traffic are changing in data center.

MapReduce is a simple and powerful programming model which enables development of scalable parallel applications to process large amount of data which is scattered on a cluster of machines. The original implementations of Map Reduce framework had some limitations which have been faced by many research follow up work after its introduction. It is gaining a lot of attraction in both research and industrial community as it has the capacity of processing large data. Map reduce framework used in different applications and for different purposes.