Analyzing The Data Processing On A Large Cluster

Research topic was derived from the understanding of query processing in MySQL and Hadoop, the database performance issues, performance tuning and the importance of database performance. Thus, it was decided to develop a comparative analysis to observe the effectiveness of the performance of MySQL (non cluster) and Hadoop in structured and unstructured dataset (Rosalia, 2015). Furthermore, the analysis included a comparison between those two platforms in two variance of data size.

• Experience in partitioning the Big Data according the business requirements using Hive Indexing, partitioning and Bucketing.

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

MapReduce Parallel programming model if we ever get a chance. In Hadoop, there are two nodes in the cluster when using the algorithm, Master node and Slave node. Master node runs Namenode, Datanode, Jobtracker and Task tracker processes. Slave node runs the Datanode and Task tracker processes. Namenode manages partitioning of input dataset into blocks and on which node it has to store. Lastly, there are two core components of Hadoop: HDFS layer and MapReduce layer. The MapReduce layer read from and write into HDFS storage and processes data in parallel.

The Hadoop employs MapReduce paradigm of computing which targets batch-job processing. It does not directly support the real time query execution i.e OLTP. Hadoop can be integrated with Apache Hive that supports HiveQL query language which supports query firing, but still not provide OLTP tasks (such as updates and deletion at row level) and has late response time (in minutes) due to absence of pipeline

An important characteristic of Hadoop is the partitioning of data and computation across many (thousands) of hosts, and the execution of application computations in parallel close to their data. A Hadoop cluster scales computation capacity, storage capacity and I/O bandwidth by simply adding commodity servers. Hadoop clusters at Yahoo! span 40,000 servers, and store 40 petabytes of application data, with the largest cluster

The availability of RDDs facilitates the implementation of both iterative algorithms that visit their dataset multiple times in a loop, and exploratory data analysis, which is the querying of data repeatedly. The latency of applications builds with spark compared to Hadoop, a MapReduce platform may be reduced by several orders of magnitude. [3]

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

Abstract-The amount of data being generated nowadays has increased tremendously. With these large volumes of data, there is a need for efficient storage and processing of data. The data generated from a variety of sources like Social networking sites, Emails, audio and video files, text files and other various files is unstructured. The traditional database cannot handle this complex and high volumes of data efficiently. The Hadoop framework provides an effective solution for storing large volumes of data and processing the data for analysis. This paper deals about the Traditional Databases, Big data, Data analysis and how effectively Hadoop stores these large volumes of data and processes data for analysis. Furthermore, this paper also focuses on the various technologies used in Hadoop for the purpose of storing the data and for processing the data.

I just found a honey bee hive in a hole in a tree in my back yord. The honey bee hive shelter up to 80,00 honey bees and there young.A honey bee hive is made of bees wax and inside the honey bee hive there are cells that store food for about a year and can be used as a nursery.Also a group of cells form a honey bee hive.

Worker honey bees make hives to store honey and feed themselves throughout winter when they cannot go outdoors to forage for food. Honey bee hives are made of six-sided tubes, which are the shapes for optimal honey production because they require less wax and can hold more honey. Some hives develop broods which become dark in color over time because of cocoon tracks and travel stains. Other honey bee hives remain light in color.

Fault Tolerance – MapReduce is highly fault tolerant, continues working in spite of failures per analysis job at Google.

Hadoop is one of the most popular technologies for handling Big Data as it is entirely open source. One of the reasons why Hadoop is used is because it is flexible enough to be able to work with multiple data sources. The multiple data sources can be combined in order to enlarge scaling processing and it can run processor intensive machine learning jobs through reading data from a database says Rodrigues in his article on Big Data. He states that Hadoop has many different applications but one that it excels in is being able to handle large volumes of data that are constantly changing.This is extremely good as it receives location based data from traffic devices and weather satellites. They also work with social media data and web-based data as

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

Data has always been analyzed within companies and used to help benefit the future of businesses. However, the evolution of how the data stored, combined, analyzed and used to predict the pattern and tendencies of consumers has evolved as technology has seen numerous advancements throughout the past century. In the 1900s databases began as “computer hard disks” and in 1965, after many other discoveries including voice recognition, “the US Government plans the world’s first data center to store 742 million tax returns and 175 million sets of fingerprints on magnetic tape.” The evolution of data and how it evolved into forming large databases continues in 1991 when the internet began to pop up and “digital storage became more cost effective than paper. And with the constant increase of the data supplied digitally, Hadoop was created in 2005 and from that point forward there was “14.7 Exabytes of new information are produced this year" and this number is rapidly increasing with a lot of mobile devices the people in our society have today (Marr). The evolution of the internet and then the expansion of the number of mobile devices society has access to today led data to evolve and companies now need large central Database management systems in order to run an efficient and a successful business.