Hadoop Is Based On Master / Slave Architecture

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

HDFS is Hadoop’s distributed file system that provides high throughput access to data, high-availability and fault tolerance. Data are saved as large blocks making it suitable for applications

Hadoop1 provides a distributed filesystem and a framework for the analysis and transformation of very large data sets using the MapReduce [DG04] paradigm. While the interface to HDFS is patterned after the Unix filesystem, faithfulness to standards was sacrificed in favor of improved performance for the applications at hand.

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

Hadoop is one of the open source frameworks, is used as extension to big data analytics framework which are used by a large group of vendors. This type of framework makes work easy for the companies how they?re going to store and can use the data within the digital products as well as physical products (James, M. et al. 2011). We can analyze data using Hadoop, which is emerging as solution to

Cloud computing is a relatively new business model in the computing world. According to the official NIST definition, "cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction." The NIST definition lists five essential characteristics of cloud computing: on-demand self-service, broad network access, resource pooling, rapid elasticity or expansion, and measured service. It also lists three "service models" (software, platform and infrastructure), and four "deployment models" (private, community, public and hybrid) that together categorize ways to deliver cloud services. The definition is intended to serve as a means for broad comparisons of cloud services and deployment strategies, and to provide a baseline for discussion from what is cloud computing to how to best use cloud computing.[1]

Hadoop is an open source framework that could be very resourceful in data processing of the complex data systems, and has been reverently used in the recent past for query processing in the complex databases that contains millions of records. The major advantage of Hadoop is that it clusters the entire records to few blocks and the query is run on each cluster and the compiled information is displayed in effective terms.

Cloud computing includes three service models, it is important to differentiate between them because the service provided is not the same and they do not address the same users. The three service models are:

Cloud computing enables businesses to deploy or subscribe to services as it needs them. A cloud features self-service, broad network access, centralized shared resources, and measurement of service usage. There are three widely accepted types of cloud service models. They are Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). Each serves a different purpose and business may choose to use just one, two, or even all three of the cloud types simultaneously as the needed.

Hadoop is a great data storage choice and Hadoop Distributed File System (HDFS) or Hive is often used to store transactional data in its raw state. The map-reduce processing supported by these Hadoop frameworks can deliver great performance, but it does not support the same specialized query optimization that mature relational database technologies do. Improving query performance, at this time, requires acquiring query accelerators or writing code. Every company who chose to use Hadoop needs to optimize their architecture in a way compatible to Hadoop.

HDFS Design: The HDFS file system is designed for storing files which are very large means files that are hundreds of megabytes, gigabytes and terabytes in size, with streaming data access patterns, running on commodity hardware clusters. HDFS has a idea of write once read many times pattern. A dataset is typically generated or copied from the source and various analyses are performed on that dataset. And hadoop does not need expensive hardware. It is designed to run on commodity hardware.

The Hadoop disseminated record framework (HDFS) is a dispersed, adaptable, and compact document framework written in Java for the Hadoop system. Some consider HDFS to rather be an information store because of its absence of POSIX consistence and powerlessness to be mounted, however it provides shell orders and Java API strategies that are like other record frameworks. A Hadoop bunch has ostensibly a solitary namenode in addition to a group of datanodes, in spite of the fact that excess alternatives are accessible for the namenode because of its criticality. Each datanode serves up pieces of information over the system utilizing a square convention particular to HDFS. The record framework utilizes TCP/IP attachments for correspondence. Customers utilize remote methodology call (RPC) to convey between each other.

Hadoop Distributed File System (HDFS): It is designed to run on hardware which are less expensive and the data is stored in it distributed. It is highly fault tolerance and provides high throughput access to the applications that require big data.

Abstract - Hadoop Distributed File System, a Java based file system provides reliable and scalable storage for data. It is the key component to understand how a Hadoop cluster can be scaled over hundreds or thousands of nodes. The large amounts of data in Hadoop cluster is broken down to smaller blocks and distributed across small inexpensive servers using HDFS. Now, MapReduce functions are executed on these smaller blocks of data thus providing the scalability needed for big data processing. In this paper I will discuss in detail on Hadoop, the architecture of HDFS, how it functions and the advantages.

The above figure shows how the data in hadoop is stored in racks and each rack consists of many distributed blocks of files, where each block is of 64Mb and can be written three times and at least one block is written to a different server rack for redundancy. In the above figure there are three different blocks ie, block1, block2, block3. Each block has been replicated at three different places and at least one replica should be placed in the different server rack for redundancy. In the figure above each block is having one of its replica placed in different rack.block1, block2, block3 are replicated both in rack1 and rack2 for data redundancy [5]. If the nodes in the rack1 have been damaged or