Disadvantages Of Map Reduce

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

Over the past few years, the needs for special-purpose applications that could handle large amount of data have increased dramatically. However, these applications required complex concepts of computations such as parallelizing the tasks, distributing data, and taking care of failures. As a reaction to this problem, a new abstract layer that allows us to express the simple computations we were trying to perform but hides the complex details was designed, MapReduce. This paper is an influential paper in the field of large scale data processing. It simplifies the programming model for processing large data set. The paper describes a new programming model based on lisp’s map and reduces primitives for processing large data set. In addition, the paper also describes a framework to automatically parallelize the map tasks across various worker machines.

MapReduce Parallel programming model if we ever get a chance. In Hadoop, there are two nodes in the cluster when using the algorithm, Master node and Slave node. Master node runs Namenode, Datanode, Jobtracker and Task tracker processes. Slave node runs the Datanode and Task tracker processes. Namenode manages partitioning of input dataset into blocks and on which node it has to store. Lastly, there are two core components of Hadoop: HDFS layer and MapReduce layer. The MapReduce layer read from and write into HDFS storage and processes data in parallel.

0166: However, it is important to state that a MapReduce application can be favored to an MPI implementation only if the data to be processed are enormous.

The Hadoop employs MapReduce paradigm of computing which targets batch-job processing. It does not directly support the real time query execution i.e OLTP. Hadoop can be integrated with Apache Hive that supports HiveQL query language which supports query firing, but still not provide OLTP tasks (such as updates and deletion at row level) and has late response time (in minutes) due to absence of pipeline

Hadoop1 provides a distributed filesystem and a framework for the analysis and transformation of very large data sets using the MapReduce [DG04] paradigm. While the interface to HDFS is patterned after the Unix filesystem, faithfulness to standards was sacrificed in favor of improved performance for the applications at hand.

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

Cost reduction: Big data technologies such as Hadoop and cloud based analytics bring significant cost advantages when it comes to storing large amounts of data – plus they can identify and implement more efficient ways of doing business.

In a style like traditional programming, calculations can happen in parallel on numerous machines in the Map stage. A similar thing likewise applies to the Reduce stage, with the goal that MapReduce applications can be greatly parallelized on a Cluster of machines.

The MapReduce programming structure utilizes two undertakings normal in functional programming: Map and Reduce. MapReduce is another parallel preparing structure and Hadoop is its open-source usage on Clusters.

Hadoop is one of the most popular technologies for handling Big Data as it is entirely open source. One of the reasons why Hadoop is used is because it is flexible enough to be able to work with multiple data sources. The multiple data sources can be combined in order to enlarge scaling processing and it can run processor intensive machine learning jobs through reading data from a database says Rodrigues in his article on Big Data. He states that Hadoop has many different applications but one that it excels in is being able to handle large volumes of data that are constantly changing.This is extremely good as it receives location based data from traffic devices and weather satellites. They also work with social media data and web-based data as

• Involved in creating Hive Tables, loading with data and writing Hive queries which will invoke and run MapReduce jobs in the backend.

Abstract - Hadoop Distributed File System, a Java based file system provides reliable and scalable storage for data. It is the key component to understand how a Hadoop cluster can be scaled over hundreds or thousands of nodes. The large amounts of data in Hadoop cluster is broken down to smaller blocks and distributed across small inexpensive servers using HDFS. Now, MapReduce functions are executed on these smaller blocks of data thus providing the scalability needed for big data processing. In this paper I will discuss in detail on Hadoop, the architecture of HDFS, how it functions and the advantages.

Cluster computing frameworks like MapReduce has been widely successful in solving numerous Big data problems. However, they tend to use one well none map and reduce pattern to solve these problems. There are many other class of problems that cannot fit into this closed box which may be handed using other set of programming model. This is where Apache Spark comes in to help solve these

It is an algorithm that is used for handling the huge amount of data in distributed manner [6]. Many Mapreduce algorithms are designed by various