What Does Spark Can Give The Better Performance Than Hadoop Distributed File System?

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

Over the past few years, the needs for special-purpose applications that could handle large amount of data have increased dramatically. However, these applications required complex concepts of computations such as parallelizing the tasks, distributing data, and taking care of failures. As a reaction to this problem, a new abstract layer that allows us to express the simple computations we were trying to perform but hides the complex details was designed, MapReduce. This paper is an influential paper in the field of large scale data processing. It simplifies the programming model for processing large data set. The paper describes a new programming model based on lisp’s map and reduces primitives for processing large data set. In addition, the paper also describes a framework to automatically parallelize the map tasks across various worker machines.

Improved performance: A distributed DBMS fragments the database to keep data closer to where it is needed most. This helps to avoid unnecessary data transfer.

MapReduce Parallel programming model if we ever get a chance. In Hadoop, there are two nodes in the cluster when using the algorithm, Master node and Slave node. Master node runs Namenode, Datanode, Jobtracker and Task tracker processes. Slave node runs the Datanode and Task tracker processes. Namenode manages partitioning of input dataset into blocks and on which node it has to store. Lastly, there are two core components of Hadoop: HDFS layer and MapReduce layer. The MapReduce layer read from and write into HDFS storage and processes data in parallel.

As we can see from the comparison tables, all computing tools provide these facilities. Hadoop distributes data as well as computing via transferring it to various storage nodes. Also, it linearly scales by adding a number of nodes to computing clusters but shows a single point failure. Cloudera Impala also quits execution of the entire query if a single part of it stops. IBM Netezza and Apache Giraph whereas does not have single point failure. In terms of parallel computation IBM Netezza is fastest due to hardware built parallelism.

An important characteristic of Hadoop is the partitioning of data and computation across many (thousands) of hosts, and the execution of application computations in parallel close to their data. A Hadoop cluster scales computation capacity, storage capacity and I/O bandwidth by simply adding commodity servers. Hadoop clusters at Yahoo! span 40,000 servers, and store 40 petabytes of application data, with the largest cluster

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

Since, real time processing acts as a game changer in big data the research developed would be to have an insight into real time analytics and streaming data to analyze the flow and to evaluate it using certain tools and techniques.

Research topic was derived from the understanding of query processing in MySQL and Hadoop, the database performance issues, performance tuning and the importance of database performance. Thus, it was decided to develop a comparative analysis to observe the effectiveness of the performance of MySQL (non cluster) and Hadoop in structured and unstructured dataset (Rosalia, 2015). Furthermore, the analysis included a comparison between those two platforms in two variance of data size.

This paper gives an overview about Grid Computing and the various key issues related to that.Today we are in the Internet world and everyone prefers to get fast access to the Internet. The growth of the internet with various resources and high speed networks changes the society enormously. We can improve the speed of the computer using Grid. This paper presents clearly the vision of the grid computing and the architecture of grid computing.The advantages and the disadvantages of the grid computing has also been discussed.

It gave more power for processing which was a just dream in the past. Now, Petabytes or more Data can be processed in less time with two different modes of processing i.e. Batch and Real-Time Big Data Analytics. Batch Analytics provide insights after several hours/days when a query is posed. Whereas, Real-Time Analytics provides ability to analyze data instantly and concentrate more on data insights than storage. In batch processing we store data and then analyze. But, in Real-Time processing we can process data as it is generated and then store only the results we are concerned

• Involved in creating Hive Tables, loading with data and writing Hive queries which will invoke and run MapReduce jobs in the backend.

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

“MapReduce Programming model is an associated implementation for processing and generating large datasets.” Prior to the development of MapReduce, Google was facing issues for processing large amounts of raw data, such as crawled documents, PageRank, Web request logs, etc. Even though computations were conceptually straightforward, the input data was very large. Also computations had to be distributed across many machines to reduce the computation time. So there was a need to find a better solution.

It is simple and easy to use – the MapReduce model is simple but expressive. With MapReduce,a programmer defines job with only Map and Reduce functions, without