The Importance Of Big Data

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

Hadoop1 provides a distributed filesystem and a framework for the analysis and transformation of very large data sets using the MapReduce [DG04] paradigm. While the interface to HDFS is patterned after the Unix filesystem, faithfulness to standards was sacrificed in favor of improved performance for the applications at hand.

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

The rise of Big Data and its attendant complexities has spawned a whole ecosystem to support the ever growing requirements of a 24x7 world. One of the key technologies coming out of the initial stages of Big Data has been Hadoop. Conceived in response to the rapidly growing needs of Yahoo!’s search engine, Hadoop provides a mechanism to store and collect vast amounts of data across a highly distributed environment using commodity hardware.

Hadoop is one of the most popular technologies for handling Big Data as it is entirely open source. One of the reasons why Hadoop is used is because it is flexible enough to be able to work with multiple data sources. The multiple data sources can be combined in order to enlarge scaling processing and it can run processor intensive machine learning jobs through reading data from a database says Rodrigues in his article on Big Data. He states that Hadoop has many different applications but one that it excels in is being able to handle large volumes of data that are constantly changing.This is extremely good as it receives location based data from traffic devices and weather satellites. They also work with social media data and web-based data as

Abstract: these days, data-intensive issues are thus prevailing that varied organizations in numerous IT industries face them in their business operation. It 's usually crucial for enterprises to own the potential of analyzing massive volumes of knowledge in a good and timely manner. Mapreduce and its ASCII text file implementation of Hadoop dramatically simplified the event of parallel knowledge computing applications for normal users for knowledge intensive parallerly, and therefore the combination of Hadoop and cloud computing created large-scale parallel knowledge computing rather more accessible and reliable to all or any potential users than ever before. Though Hadoop has become the foremost in style knowledge management framework for parallel data-intensive computing within the clouds, the Hadoop computer hardware isn 't an ideal match for the cloud environments. In this paper, we discuss the issues of Hadoop task assignment scheme and present an improved scheme for heterogeneous computing environments using Apache Ambari with Parallel Fast Fourier Transform. We conducted in depth simulation to judge the performance of the projected theme compared with the Hadoop theme in two forms of heterogeneous computing environments that are typical on the general public cloud platforms. The simulation

Hadoop is an Apache open source software (java framework). It runs on cluster of commodity machines and provides both distributed storage and distributed processing of huge data sets. It is capable of processing data sizes ranging from Gigabytes to Petabytes.

Big data is an extensive collection of structured and unstructured data. It is a modern day technology which is applied to store, manage and analyze data that are not possible to manage, store and analyze by using the commonly used software or tools. Since all of our daily tasks are overtaken by the modern technologies and all the businesses and organizations are using internet system to operate, the production of data has increased significantly in past

MapReduce refers to the application modules written by a programmer that run in two phases: first mapping the data (extract) then reducing it (transform). Hadoop scales out to large clusters of servers and storage using the Hadoop Distributed File System (HDFS) to manage huge data sets and spread them across the servers. Hadoop also has YARN (Yet another Resource Navigator) manages the clusters, and MapReduce performs data processing which helps in faster processing. (‘CITO Research’, 2014).

In modern times, the amount of data being stored is terrifically large. Companies must deal with such abundance of data on a daily basis in both storing and analyzing as fast as they can. One such company that not only store data is Google, they also analyze data from each user using their product. The platform used by google for this database management called BigQuery, which runs in the cloud and provides real time information. In this survey, the inner working of BigQuery is glossed over to show how this platform manages to do the job it is supposed to accomplish.

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

“Hadoop” it is not a language or technology, it is a frame developed by Yahoo and maintained by apache for big data problem. Data in the web, internet will be of different formats. Data will be of image, text, media files. Data can be of many structured, unstructured and semi structured formats. In the current world than static data, the streaming data is more and capturing it is a challenge for today’s world.

Jeffrey Dean and Sanjay Ghemawat came up with the concept of MapReduce at Google in 2004. The main idea behind MapReduce is to map your data set into a collection of (key, value) pairs and then reducing over all the pairs with same key. A (key, value) pair is given as an input to the map function. Map function then produces output in the form of a set of intermediate (key, value) pair. The MapReduce library looks for all the values of one key and groups them together. These are then passed to reduce function. Reduce function then merges these values together to form a possibly smaller set of values.

MapReduce: The MapReduce language establishes a base for Hadoop Eco System. It processes Hadoop Distributed File System (HDFS) on large clusters which are made of thousands of commodity hardware in a reliable and fault-tolerant manner. The operations of MapReduce are performed in Map and Reduce functions. The Map function works on a set of input values and transforms them into a set of key/value pairs. The reducer receives all the data for an individual "key" from all the mappers and applies Reduce function to achieve the final result.

MapReduce programming paradigm is similar to the concept of map and reduce primitives which are present in Lisp and many other functional languages. The input to a MapReduce program is a (key, value) pair. The output is a set of (key, value) pairs. There are two main functions, a map function and a reduce function. Map function takes an input (key, value) pair and produces a set of intermediate (key, value) pairs. The MapReduce library looks for all the values of one key and groups them together. These are then passed to reduce function. The reduce function gets an intermediate key and all the values associated with it and merges these values together to form a possibly smaller set of values.