Apache Spark Resilient Distributed Datasets

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

This paper proposes backup task mechanism to improve the straggler tasks which are the final set of MapReduce tasks that take unusually longer time to complete. The simplified programming model proposed in this paper opened up the parallel computation field to general purpose programmers. This paper served as the foundation for the open source distributing computing software – Hadoop as well as tackles various common error scenarios that are encountered in a compute cluster and provides fault tolerance solution on a framework

MapReduce Parallel programming model if we ever get a chance. In Hadoop, there are two nodes in the cluster when using the algorithm, Master node and Slave node. Master node runs Namenode, Datanode, Jobtracker and Task tracker processes. Slave node runs the Datanode and Task tracker processes. Namenode manages partitioning of input dataset into blocks and on which node it has to store. Lastly, there are two core components of Hadoop: HDFS layer and MapReduce layer. The MapReduce layer read from and write into HDFS storage and processes data in parallel.

An important characteristic of Hadoop is the partitioning of data and computation across many (thousands) of hosts, and the execution of application computations in parallel close to their data. A Hadoop cluster scales computation capacity, storage capacity and I/O bandwidth by simply adding commodity servers. Hadoop clusters at Yahoo! span 40,000 servers, and store 40 petabytes of application data, with the largest cluster

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

Dynamo is therefore described as a store that meets the needs of these classes of services with a simple key/value interface, in this paper. Each service using dynamo runs its own dynamo instances.

In a style like traditional programming, calculations can happen in parallel on numerous machines in the Map stage. A similar thing likewise applies to the Reduce stage, with the goal that MapReduce applications can be greatly parallelized on a Cluster of machines.

The rise of Big Data and its attendant complexities has spawned a whole ecosystem to support the ever growing requirements of a 24x7 world. One of the key technologies coming out of the initial stages of Big Data has been Hadoop. Conceived in response to the rapidly growing needs of Yahoo!’s search engine, Hadoop provides a mechanism to store and collect vast amounts of data across a highly distributed environment using commodity hardware.

Spark is a cluster framework with an open source software. It was 1st invented by Berkely in AMP Lab. It was initially invented by Berkeley's AMP Lab and later donated to Apache Foundation software. Apache Spark follows the concept of RDD called resilient distributed dataset. This is just a readable dataset. Later it is added to Apache foundation software Spark is built on resilient distributed datasets (RDD) as a read-only multiset of data items. Spark core, Spark SQL, Spark MLib, Spark Streaming--- are the modules in the spark. Spark is a kind of API which has inbuilt memory data, a faster machine to run that. It enables

Mahout has classifiers which help in high quality implementation. It uses sequential processing instead of parallel processing which results in slow retrieval of data. It provides various algorithms in a systematic way. It also has an information retrieval library which named Lucene. Spark on the other hand uses MLlib which helps in really fast retrieval of data. It is primarily used for sophisticated analytics. It also supports predictions about data which lead to business growth exponentially. It can run along with other Hadoop tools like Pig and Hive. It is an iterative algorithm which helps in fast running and retrieval of data on Hadoop cluster. As a result its algorithms are much faster when compared to Mahout equivalent

FlowComb also uses MapReduce framework to influence the design of the system. MapReduce provides a divide and conquer data processing model, where large workloads are split into smaller tasks, each processed by a single server in a cluster (the map phase). The results of each task are sent over the cluster network (the shuffle phase) and merged to obtain the final result (the reduce phase). The network footprint of a MapReduce job consists pre dominantly of traffic sent during the shuffle

Abstract - Hadoop Distributed File System, a Java based file system provides reliable and scalable storage for data. It is the key component to understand how a Hadoop cluster can be scaled over hundreds or thousands of nodes. The large amounts of data in Hadoop cluster is broken down to smaller blocks and distributed across small inexpensive servers using HDFS. Now, MapReduce functions are executed on these smaller blocks of data thus providing the scalability needed for big data processing. In this paper I will discuss in detail on Hadoop, the architecture of HDFS, how it functions and the advantages.

The Apache Hadoop system is an open source software, it is a framework that allows for the distributed processing of large amount of data sets across cluster of

We will look at what is integrating hadoop and how it functions. We will also look at how it works with Map reduce and how to output data to Cassandra.

Spark is cluster framework with an open source software. It was 1st invented by Berkely in