Introduction Of Mapreduce Programming Model

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

Over the past few years, the needs for special-purpose applications that could handle large amount of data have increased dramatically. However, these applications required complex concepts of computations such as parallelizing the tasks, distributing data, and taking care of failures. As a reaction to this problem, a new abstract layer that allows us to express the simple computations we were trying to perform but hides the complex details was designed, MapReduce. This paper is an influential paper in the field of large scale data processing. It simplifies the programming model for processing large data set. The paper describes a new programming model based on lisp’s map and reduces primitives for processing large data set. In addition, the paper also describes a framework to automatically parallelize the map tasks across various worker machines.

The key functions to be implemented are Map and Reduce. The MapReduce framework operates on key and value pairs. Each Map task processes an input split block generating intermediate data of key and value format. Then, they are sorted and partitioned by key, so later at Reduce phase, pairs of the same key will be aggregated to the same reducer for further processing. Partitions from different nodes with the same key are transferred from the shuffle phase to a single node and then merged and get ready to be fed to the reduce task. The output of Reduce tasks is same format, key and value, as

0166: However, it is important to state that a MapReduce application can be favored to an MPI implementation only if the data to be processed are enormous.

As we can see from the comparison tables, all computing tools provide these facilities. Hadoop distributes data as well as computing via transferring it to various storage nodes. Also, it linearly scales by adding a number of nodes to computing clusters but shows a single point failure. Cloudera Impala also quits execution of the entire query if a single part of it stops. IBM Netezza and Apache Giraph whereas does not have single point failure. In terms of parallel computation IBM Netezza is fastest due to hardware built parallelism.

An important characteristic of Hadoop is the partitioning of data and computation across many (thousands) of hosts, and the execution of application computations in parallel close to their data. A Hadoop cluster scales computation capacity, storage capacity and I/O bandwidth by simply adding commodity servers. Hadoop clusters at Yahoo! span 40,000 servers, and store 40 petabytes of application data, with the largest cluster

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

International Journal of Cloud Computing: Peer-reviewed open access journal, it publishes research crossing all aspects of Cloud Computing. Basically centered around center components, including Cloud applications, Cloud systems and the advances that will prompt the Clouds without bounds, the journal will likewise show review and survey papers that present new bits of knowledge and establish the frameworks for encouraging exploratory and experimental work. The journal disseminates research that imparts progressed hypothetical establishing and functional application of Clouds and related systems, as empowered by mixes of web-based programming, advancement stacks and database availability and virtualized equipment for storing, handling, analysis and visualizing data. A scope will look at Clouds nearby such different standards as Peer to Peer (P2P) figuring, Cluster processing and Grid registering. Scope reaches out to issues of administration, governance, trust and

The paper “A Comparison to Approaches to Large-Scale Data Analysis” by Pavlo, compares and analyze the MapReduce framework with the parallel DBMSs, for large scale data analysis. It benchmarks the open source Hadoop, build over MapReduce, with two parallel SQL databases, Vertica and a second system form a major relational vendor (DBMS-X), to conclude that parallel databases clearly outperform Hadoop on the same hardware over 100 nodes. Averaged across 5 tasks on 100 nodes, Vertica was 2.3 faster than DBMS-X which in turn was 3.2 times faster than MapReduce. In general, the parallel SQL DBMSs were significantly faster and required less code to implement each task, but took longer to tune and load the data. Finally, the paper talk about

Other than disturbed file system, many higher-level programming frame work has developed. Most used commonly framework is MapReduce frame work for data intensive apps which is developed by google. MapReduce uses the ideas of the functional programming here the developer assigns map and reduce tasks to the process sets of distributed data. Implementation of MapReduce this ensures all the calculations of high scale data to be executed on the computer hubs which can withstand the hardware failures during computation. The main advantages of the map reduce programming framework are high degree of parallelism proportional with programming framework and its large variety of app domains. The map functions input pairs key1, value1 returning some other pairs that is key2, value2. The reduce output new function that is key3, value3

Big data is an extensive collection of structured and unstructured data. It is a modern day technology which is applied to store, manage and analyze data that are not possible to manage, store and analyze by using the commonly used software or tools. Since all of our daily tasks are overtaken by the modern technologies and all the businesses and organizations are using internet system to operate, the production of data has increased significantly in past

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

Provide an approach for research efforts towards developing highly scalable and autonomic data management systems associated with programming models for processing Big Data. Aspects of such systems should address challenges related to data analysis algorithms, real-time processing and visualisation, context awareness, data management and performance and scalability, correlation and causality and to some extent, distributed storage [1]. Provide an approach for framework for evaluating big data initiatives [2]. Provide an approach for summarize opportunities and challenges with big data. Recent technological advances and novel applications, such as sensors, cyber-physical systems, smart mobile devices ,cloud systems, data analytics, and social networks, are making possible to capture, process, and share huge amounts of data – referred to as big data - and to extract useful knowledge, such as patterns, from this data and predict trends and events. Big data is making possible tasks that before were impossible, like preventing disease spreading and crime, personalizing healthcare, quickly identifying business opportunities, managing emergencies, protecting the homeland, and so on [3]. Provide an approach for sources of structured and unstructured big data. Unstructured data is everywhere. In fact, most individuals and organizations conduct their lives around unstructured data [4]. Successful decision-making will increasingly be driven by analytics-generated

Abstract —Data Clustering is key point used in data processing algorithms for Data Mining. Clustering is a data mining technique used to place data elements into related groups without advance knowledge of the group definitions. Popular clustering techniques include k-means clustering. Clustering is imperative idea in data investigation and data mining applications. In last decade, K-means has been popular clustering algorithm because of its ease of use and simplicity. Now days, as data size is continuously increasing, some researchers started working over distributed environment such as MapReduce to get high performance for big data clustering.

MapReduce is a simple and powerful programming model which enables development of scalable parallel applications to process large amount of data which is scattered on a cluster of machines. The original implementations of Map Reduce framework had some limitations which have been faced by many research follow up work after its introduction. It is gaining a lot of attraction in both research and industrial community as it has the capacity of processing large data. Map reduce framework used in different applications and for different purposes.