Case Study : The Google File System

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

When write operation are done on existing file, the data is appended to end of the file. This method serializes the data write in GFS.

In a client and server caches, an each server cache is managed in one distributed system tier by a number of server members, usually cache server processes. In a separate tier, clients maintain their own caches that automatically call upon the server side to set off cache to update the server with client updates. In addition, the clients can member to server events using both data key specifications and queries. The client/server topology can be extended to any number of tiers.

This paper proposes backup task mechanism to improve the straggler tasks which are the final set of MapReduce tasks that take unusually longer time to complete. The simplified programming model proposed in this paper opened up the parallel computation field to general purpose programmers. This paper served as the foundation for the open source distributing computing software – Hadoop as well as tackles various common error scenarios that are encountered in a compute cluster and provides fault tolerance solution on a framework

GFS: Google File System is a distributed file system which is developed by Google in order to provide efficient, reliable access to data. . It is designed and implemented inorder to meet the requirements provided by Google’s data processing. The file system consists of hundreds of storage machines to provide inexpensive parts and it is accessed by different client machines. Here the search engine is providing huge amounts data that should be stored. GFS has 1,000 nodes with 300TB disk storage.

HDFS is Hadoop’s distributed file system that provides high throughput access to data, high-availability and fault tolerance. Data are saved as large blocks making it suitable for applications

0166: In spite of this efficiency in MPI, handling Big Data applications is a challenge job for MPI where fault-tolerance by checkpointing can be impractical at very large-scale data because of its excessive disk access and limited scalability.

[8] J. R. Douceur, A. Adya, W. J. Bolosky, D. Simon, and M. Theimer.Reclaiming space from duplicate files in a serverless distributedfile system. In ICDCS, pages 617–624, 2002.

An important characteristic of Hadoop is the partitioning of data and computation across many (thousands) of hosts, and the execution of application computations in parallel close to their data. A Hadoop cluster scales computation capacity, storage capacity and I/O bandwidth by simply adding commodity servers. Hadoop clusters at Yahoo! span 40,000 servers, and store 40 petabytes of application data, with the largest cluster

The aim of work is to provide service guarantees when multiple synchronous requests are present with high disk throughput. To address this problem we consider BFQ and modified versions of BFQ. It is found that MBFQV1 gives a better performance when compared with the BFQ. MBFQV2 is the suggested new disk scheduler which preserve both guarantees and a high throughput. In MBFQV2 we observed that the throughput, speed of transfer were better compared to the other schedulers for the normal size applications.

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

In order to overcome these limitations, a new database model known as Not Only SQL (NoSQL) database emerged with a set of new features. The main objective of NoSQL is not to discard SQL, but to be used as an alternative database data model for new features [1] [2] [3]. NoSQL database increases the performance of relational databases by a set of new characteristics and advantages. In contrast to relational databases, NoSQL databases introduced an additional feature that provides flexible and horizontal scalability and taking advantage of new clusters. The rise of NoSQL provides cost-effective management of data in modern web applications. With its new features, NoSQL can be used with applications that have a large transaction, and require low-latency access to huge datasets, service availability while

Distributed database is the study of how the communication can be build by creating the data on the local computer and distributed that information on different computer connected to the same physical loacation. With the involation of the Distributed database in the rapid growth of the Internet can

With the increasing focus on information and big data the storage capability of datacenters is in a state of flux and confusion. Consequently, many storage problems can be solved with a software-centric infrastructure and move storage from proprietary hardware devices to software-led services that become storage pools in the software-defined datacenter (SDC). HaaS is the answer with its software-defined storage (SDS).

Abstract - Hadoop Distributed File System, a Java based file system provides reliable and scalable storage for data. It is the key component to understand how a Hadoop cluster can be scaled over hundreds or thousands of nodes. The large amounts of data in Hadoop cluster is broken down to smaller blocks and distributed across small inexpensive servers using HDFS. Now, MapReduce functions are executed on these smaller blocks of data thus providing the scalability needed for big data processing. In this paper I will discuss in detail on Hadoop, the architecture of HDFS, how it functions and the advantages.