Analyzing Hadoop Framework Is A Solution For Big Data Problem

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

An important characteristic of Hadoop is the partitioning of data and computation across many (thousands) of hosts, and the execution of application computations in parallel close to their data. A Hadoop cluster scales computation capacity, storage capacity and I/O bandwidth by simply adding commodity servers. Hadoop clusters at Yahoo! span 40,000 servers, and store 40 petabytes of application data, with the largest cluster

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

Since the 1970’s databases and report generators have been used to aid business decisions. In the 1990’s technology in this area improved. Now technology such as Hadoop has gone another step with the ability to store and process the data within the same system which sparked new buzz about “big data”. Big Data is roughly the collection of large amounts of data – sourced internally or externally - applied as a tool – stored, managed, and analyzed - for an organization to set or meet certain goals.

During the period of my curriculum practical training I have learned the HADOOP technology, initial three weeks I was taught the concepts that I should be aware of in order to understand the whole concepts of HADOOP technology. In this regard I was taught collection framework in java at first.

For analysis, advanced statistical tools are used and the experimenter can draw necessary conclusions and inferences. In business sectors, they have a huge amount of scattered data in terms of profits, loss, demand, supply, sales and production. The industrial, insurance, agricultural, banking, information technology, food industry, telecommunication, retail, utilities, travel, pharmacy and many more have challenges to manage their data. As the coin always has two sides, there are both advantages and a few disadvantages of data analysis. The key advantages of data analysis are- The organizations can immediately come across errors, the service provided after optimizing the system using data analysis reduces the chances of failure, saves time and leads to advancement. It is also used to compare strategies between two companies so as to reduce the prices and gaining attention of target customers, ultimately leading to maximization of profit and minimization of cost(as done in Game Theory). However, big data analysis sometimes becomes more tedious and disadvantageous because it uses software Hadoop which requires special provisions in the computers. For now use of Hadoop for real-time analysis is not available. The manner in which the data is collected and the decision making view can vary from one person to another. Here, the quality of data gets affected and leaves the data insufficient or inefficient. In order to tackle this problem, the researcher must be professional, well experienced and should have deep knowledge about the characteristic under study. Also, we need to update data from time to time so as to avoid the changes in trend caused by the past data especially, for the rapidly growing

Hadoop is one of the open source frameworks, is used as extension to big data analytics framework which are used by a large group of vendors. This type of framework makes work easy for the companies how they?re going to store and can use the data within the digital products as well as physical products (James, M. et al. 2011). We can analyze data using Hadoop, which is emerging as solution to

The main purpose of this report is to provide a critical review of the processes and own experiences of Hadoop within the context of the assignment which was given to us. The review concentrates on the discussion and evaluation of the overall steps followed during the progress of the project and the reasons for which we have chosen these particular steps. It also draws attention at the main points that were accomplished, both with respect to individual, and with respect to the group 's perspectives. Finally, it concentrates on the project 's progress in terms of changes for a future implementation.

Big data is buzzword in every field of business as well as research. Organizations have found its application across various sectors from Sports to Security, from Healthcare to e-Commerce.

The rise of Big Data and its attendant complexities has spawned a whole ecosystem to support the ever growing requirements of a 24x7 world. One of the key technologies coming out of the initial stages of Big Data has been Hadoop. Conceived in response to the rapidly growing needs of Yahoo!’s search engine, Hadoop provides a mechanism to store and collect vast amounts of data across a highly distributed environment using commodity hardware.

As a result of the appearance of big data in our world, conventional data warehousing and data analysis methods no longer have the process power needed. What is Big Data you may ask and why is it such a big deal. NIST defines big data as anywhere “[…] data volume, acquisition velocity, or data representation limits the ability to perform effective analysis using traditional relational approaches […]” (Mell & Cooper, n.d.).

Big Data is creating great opportunities for businesses, companies and many large scale and small scale industries. Hadoop is an open-source cloud computing and big data framework, is increasingly used in the IT world. The rapid growth of Hadoop and Cloud Computing clearly indicates its importance as a Big Data enabling technology. Due to the loopholes of security mechanism, the security issues introduced through adaptation of this technology are also increasing. Hadoop services do not

Hadoop is an open source framework that could be very resourceful in data processing of the complex data systems, and has been reverently used in the recent past for query processing in the complex databases that contains millions of records. The major advantage of Hadoop is that it clusters the entire records to few blocks and the query is run on each cluster and the compiled information is displayed in effective terms.

Big data is an extensive collection of structured and unstructured data. It is a modern day technology which is applied to store, manage and analyze data that are not possible to manage, store and analyze by using the commonly used software or tools. Since all of our daily tasks are overtaken by the modern technologies and all the businesses and organizations are using internet system to operate, the production of data has increased significantly in past

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.