The Basic Idea Of Hadoop

When a file is written in HDFS, it is divided into fixed size blocks. The client first contacts the NameNode, which get the list of DataNode where actual data can be stored. The data blocks are distributed across the Hadoop cluster. Figure \ref{fig.clusternode} shows the architecture of the Hadoop cluster node used for both computation and storage. The MapReduce engine (running inside a Java virtual machine) executes the user application. When the application reads or writes data, requests are passed through the Hadoop \textit{org.apache.hadoop.fs.FileSystem} class, which provides a standard interface for distributed file systems, including the default HDFS. An HDFS client is then responsible for retrieving data from the distributed file system by contacting a DataNode with the desired block. In the common case, the DataNode is running on the same node, so no external network traffic is necessary. The DataNode, also running inside a Java virtual machine, accesses the data stored on local disk using normal file I/O

In addition to the Function that let the user map from an Entry to something else, he or she can also map from a Key-Value-pair to something else. This is convenient, sure, but we can also add more specific mapping operations now that we are working with pairs of values.

Each map task produces an intermediate data set that is used by the reduce task to combine the map task results. This paper also proposes various extensions to the framework to allow the users to customize the data partitioning as well as combiner. Also, the framework provides a mechanism for the user programs to track relevant metrics and publish them. This paper describes the various error scenarios that happen in a large cluster of commodity hardware machines. Fault tolerance to handle various map tasks

Set associative mapping is made up of direct and full associated mapping. The branch address is mapped to multiple entries of the table while inside those set of entries, search is made fully associative.

It takes care of all cluster maintenance tasks and job scheduling operations and allows the programmer to focus on programming the logic of the application. Submitting a MapReduce job to the master node, results in splitting the input file to several chunks which are block sized that are processed by Map and Reduce tasks at parallel. Due to block replication of HDFS, tasks are scheduled to run on nodes where the required chunks of data already exist, minimizing unnecessary transfer of these data.

0166: However, it is important to state that a MapReduce application can be favored to an MPI implementation only if the data to be processed are enormous.

The Hadoop employs MapReduce paradigm of computing which targets batch-job processing. It does not directly support the real time query execution i.e OLTP. Hadoop can be integrated with Apache Hive that supports HiveQL query language which supports query firing, but still not provide OLTP tasks (such as updates and deletion at row level) and has late response time (in minutes) due to absence of pipeline

Hadoop1 provides a distributed filesystem and a framework for the analysis and transformation of very large data sets using the MapReduce [DG04] paradigm. While the interface to HDFS is patterned after the Unix filesystem, faithfulness to standards was sacrificed in favor of improved performance for the applications at hand.

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

Processing converts this raw input into a form that has meaning. Output transfers the processed information

Faster, better decision making: With the speed of Hadoop and in-memory analytics, combined with the capability to analyze new sources of data, businesses are able to analyze data immediately and make decisions based on what they’ve learned.

HDFS Design: The HDFS file system is designed for storing files which are very large means files that are hundreds of megabytes, gigabytes and terabytes in size, with streaming data access patterns, running on commodity hardware clusters. HDFS has a idea of write once read many times pattern. A dataset is typically generated or copied from the source and various analyses are performed on that dataset. And hadoop does not need expensive hardware. It is designed to run on commodity hardware.

Abstract - Hadoop Distributed File System, a Java based file system provides reliable and scalable storage for data. It is the key component to understand how a Hadoop cluster can be scaled over hundreds or thousands of nodes. The large amounts of data in Hadoop cluster is broken down to smaller blocks and distributed across small inexpensive servers using HDFS. Now, MapReduce functions are executed on these smaller blocks of data thus providing the scalability needed for big data processing. In this paper I will discuss in detail on Hadoop, the architecture of HDFS, how it functions and the advantages.

In the above figure at first the data or the files is divided into small blocks of records and are replicated at three different places. After the job tracker receives the job to be performed it will locate where the information is and then allots a task to the task tracker in the slave node then the mapping will be performed first and it produces the key, value pairs which is given as input to the reduce. In between the map and reduce shuffling/sorting will be done where similar data will be gathered together and sorted. Now the structure information ie key, value pairs are given as input to reduce and then it will generate set of key, value pairs as output for the given key value pairs. Here deciding what will be the key and what will be value is developer’s responsibility. A simple map-reduce example which explains the method more elaborately is as follows:

MapReduce is a simple and powerful programming model which enables development of scalable parallel applications to process large amount of data which is scattered on a cluster of machines. The original implementations of Map Reduce framework had some limitations which have been faced by many research follow up work after its introduction. It is gaining a lot of attraction in both research and industrial community as it has the capacity of processing large data. Map reduce framework used in different applications and for different purposes.