Mapreduce, The Core Programming Language Of The Distributed File System

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

Over the past few years, the needs for special-purpose applications that could handle large amount of data have increased dramatically. However, these applications required complex concepts of computations such as parallelizing the tasks, distributing data, and taking care of failures. As a reaction to this problem, a new abstract layer that allows us to express the simple computations we were trying to perform but hides the complex details was designed, MapReduce. This paper is an influential paper in the field of large scale data processing. It simplifies the programming model for processing large data set. The paper describes a new programming model based on lisp’s map and reduces primitives for processing large data set. In addition, the paper also describes a framework to automatically parallelize the map tasks across various worker machines.

MapReduce Parallel programming model if we ever get a chance. In Hadoop, there are two nodes in the cluster when using the algorithm, Master node and Slave node. Master node runs Namenode, Datanode, Jobtracker and Task tracker processes. Slave node runs the Datanode and Task tracker processes. Namenode manages partitioning of input dataset into blocks and on which node it has to store. Lastly, there are two core components of Hadoop: HDFS layer and MapReduce layer. The MapReduce layer read from and write into HDFS storage and processes data in parallel.

The Hadoop employs MapReduce paradigm of computing which targets batch-job processing. It does not directly support the real time query execution i.e OLTP. Hadoop can be integrated with Apache Hive that supports HiveQL query language which supports query firing, but still not provide OLTP tasks (such as updates and deletion at row level) and has late response time (in minutes) due to absence of pipeline

Hadoop1 provides a distributed filesystem and a framework for the analysis and transformation of very large data sets using the MapReduce [DG04] paradigm. While the interface to HDFS is patterned after the Unix filesystem, faithfulness to standards was sacrificed in favor of improved performance for the applications at hand.

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

Research topic was derived from the understanding of query processing in MySQL and Hadoop, the database performance issues, performance tuning and the importance of database performance. Thus, it was decided to develop a comparative analysis to observe the effectiveness of the performance of MySQL (non cluster) and Hadoop in structured and unstructured dataset (Rosalia, 2015). Furthermore, the analysis included a comparison between those two platforms in two variance of data size.

For analysis, advanced statistical tools are used and the experimenter can draw necessary conclusions and inferences. In business sectors, they have a huge amount of scattered data in terms of profits, loss, demand, supply, sales and production. The industrial, insurance, agricultural, banking, information technology, food industry, telecommunication, retail, utilities, travel, pharmacy and many more have challenges to manage their data. As the coin always has two sides, there are both advantages and a few disadvantages of data analysis. The key advantages of data analysis are- The organizations can immediately come across errors, the service provided after optimizing the system using data analysis reduces the chances of failure, saves time and leads to advancement. It is also used to compare strategies between two companies so as to reduce the prices and gaining attention of target customers, ultimately leading to maximization of profit and minimization of cost(as done in Game Theory). However, big data analysis sometimes becomes more tedious and disadvantageous because it uses software Hadoop which requires special provisions in the computers. For now use of Hadoop for real-time analysis is not available. The manner in which the data is collected and the decision making view can vary from one person to another. Here, the quality of data gets affected and leaves the data insufficient or inefficient. In order to tackle this problem, the researcher must be professional, well experienced and should have deep knowledge about the characteristic under study. Also, we need to update data from time to time so as to avoid the changes in trend caused by the past data especially, for the rapidly growing

Hadoop exceeds expectations at putting away information of discretionary, or even unstructured arrangements, since it gives us a chance to choose how to decipher the information at investigation time, permitting to change the way we order the information whenever: once we have upgraded the calculations, essentially run the examination once more. Hadoop additionally supplements existing database frameworks of any sort. It offers a boundless pool into which one can sink information and still haul out what is required when the time is correct. It is streamlined for substantial document stockpiling and clump situated, gushing access. This makes investigation simple and quick, however clients additionally need access to the last information, not in cluster mode however utilizing arbitrary access—this is similar to a full table sweep versus utilizing files in a database framework

Abstract: these days, data-intensive issues are thus prevailing that varied organizations in numerous IT industries face them in their business operation. It 's usually crucial for enterprises to own the potential of analyzing massive volumes of knowledge in a good and timely manner. Mapreduce and its ASCII text file implementation of Hadoop dramatically simplified the event of parallel knowledge computing applications for normal users for knowledge intensive parallerly, and therefore the combination of Hadoop and cloud computing created large-scale parallel knowledge computing rather more accessible and reliable to all or any potential users than ever before. Though Hadoop has become the foremost in style knowledge management framework for parallel data-intensive computing within the clouds, the Hadoop computer hardware isn 't an ideal match for the cloud environments. In this paper, we discuss the issues of Hadoop task assignment scheme and present an improved scheme for heterogeneous computing environments using Apache Ambari with Parallel Fast Fourier Transform. We conducted in depth simulation to judge the performance of the projected theme compared with the Hadoop theme in two forms of heterogeneous computing environments that are typical on the general public cloud platforms. The simulation

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

With the rise of Hadoop in the workplace, along with it comes a rise of vendors offering various platforms to store, access, and analyze vast amounts of data. These platforms vary in their functionality, cost, and ease of use, among other factors. Three of the more popular vendors are Amazon Web Services, Map-R, and Cloudera. While each of these is based on Apache Hadoop’s open source offerings, it is the applications and reach that differentiates them. Although price will always be a factor as well, this comparison seeks only to explore the vendors themselves, not the price of admission.

Abstract—Parallel databases are the high performance databases in RDBMS world that can used for setting up data intensive enterprise data warehouse but they lack scalability whereas, MapReduce paradigm highly supports scalability, nevertheless cannot perform as good as parallel databases. Deriving an architectural hybrid model of best of both worlds that can support high performance and scalability at the same time.

Business analysts, Data scientists, Data analysts, Statisticians, want to analyze the data for collecting the important characteristics of the data set. In 2004 Google introduced MapReduce. It simplified the data processing on a large cluster. But many organizations have only few developers who can write good MapReduce code, which is written in java. (MapReduce can be written in other languages).

“MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster” [1]. Before the introduction of MapReduce, biggest challenge in front of Google was processing large input data and distributed computations. There was no solution for parallelizing the computations, distributing the data, and handling failures. In order to solve these problems, Google introduced the concept of MapReduce for parallel computation in a distributed system for synthesizing large datasets.