Importance Of Data Analysis On The Enterprise Big Data Project Essay

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

An important characteristic of Hadoop is the partitioning of data and computation across many (thousands) of hosts, and the execution of application computations in parallel close to their data. A Hadoop cluster scales computation capacity, storage capacity and I/O bandwidth by simply adding commodity servers. Hadoop clusters at Yahoo! span 40,000 servers, and store 40 petabytes of application data, with the largest cluster

This is one of the greatest challenge that is acting as a key barrier in widespread adoption and pervasive use of the tools fostering for development and progress in this field of study. The real time of the big data analytics would require some of the unique features and special computing powers and potentials (Gantz, 2012). Tools have to made specially advanced so as to incorporate process terms in real time (Chen, 2012). Every business oriented organization should be transformed into information centric (Kaisler, 2013) to focus upon the real time data analysis in terms of both input and output.

The paper “A Comparison to Approaches to Large-Scale Data Analysis” by Pavlo, compares and analyze the MapReduce framework with the parallel DBMSs, for large scale data analysis. It benchmarks the open source Hadoop, build over MapReduce, with two parallel SQL databases, Vertica and a second system form a major relational vendor (DBMS-X), to conclude that parallel databases clearly outperform Hadoop on the same hardware over 100 nodes. Averaged across 5 tasks on 100 nodes, Vertica was 2.3 faster than DBMS-X which in turn was 3.2 times faster than MapReduce. In general, the parallel SQL DBMSs were significantly faster and required less code to implement each task, but took longer to tune and load the data. Finally, the paper talk about

Big data is the present most-liked theme of today 's technology. These research goes through all description of techniques and technologies of extracting of the data, storing of data, distribution of data, analyzing of data, managing of data with high velocity and from the structured data and helps in the handling of the extreme data. Big data has the presentation the capacity to improve predictions, saving money and enhancing the decision making process in the fields of the traffic control, weather forecasting, disaster prevention, fraud control, business transaction, education system, health and the national security.

The main purpose of this report is to provide a critical review of the processes and own experiences of Hadoop within the context of the assignment which was given to us. The review concentrates on the discussion and evaluation of the overall steps followed during the progress of the project and the reasons for which we have chosen these particular steps. It also draws attention at the main points that were accomplished, both with respect to individual, and with respect to the group 's perspectives. Finally, it concentrates on the project 's progress in terms of changes for a future implementation.

Big Data is an outgrowth of the proliferation of databases and massive data sets. The insights needed to more intelligently manage an organization can be found in the myriad of data sets that comprise a Big Data platform. The greatest challenge of Big Data is contextual intelligence supported by integration to legacy, 3rd party and homegrown application systems located throughout an enterprise (Jacobs, 2009). To get ot his level of proficiency in analyzing Big Data sets and databases, enterprises need Business Intelligence (BI) and analytics tools that can parse through terabytes quickly, finding patterns and analyzing massive amounts of data, then distilling it down to key

Across Industry verticals there is increasing adoption of Hadoop for Information Management and Analytics. Many have realized that in addition to new business related capabilities Hadoop also offers a host of options for IT Simplification and cost reduction. Initiatives such as Offloads are at the heart of such optimization. That said, capacity planning is the first step that needs to be carried out successfully for either a IT driven use case or a business driven use case. This paper takes a look at why Big Data processing frameworks such as Hadoop clusters require careful capacity planning for the timely launch of Big Data based capabilities. Additionally, it discusses how capacity planning can facilitate appropriate service level agreement (SLA) guarantees and ensure deliveries within defined budgets. These types of guarantees with sets of standard hardware configuration are the key for effective capacity planning. The key constituent of overall capacity management strategy for the Hadoop eco system is Cluster Capacity Planning. It is this part of the strategy that caters to the troublesome and unavoidable

Today ‘Big data’ is more popular than before, the performance of a database is becoming extremely important, including family life, school studying, office work and all business. Providing reliable and faster database services is the goal of organizations including any business, schools, and

Altiscale was established to give associations access to the main cloud reason worked for Apache Hadoop, and additionally the operational aptitude expected to execute complex Hadoop ventures. The Altiscale group has been on the bleeding edge of Apache Hadoop – from its hatching at Yahoo to working more than 40,000 Hadoop hubs. As an organization that comprehends both the transformative energy of this innovation and its difficulties, no other association is better situated to convey dependable and versatile Apache Hadoop.

Big data is an extensive collection of structured and unstructured data. It is a modern day technology which is applied to store, manage and analyze data that are not possible to manage, store and analyze by using the commonly used software or tools. Since all of our daily tasks are overtaken by the modern technologies and all the businesses and organizations are using internet system to operate, the production of data has increased significantly in past

Once the data isdata are stored in Hadoop, the big data applications can be used to analyze the data. Figure 4.3 shows a simple standalone Hadoop architecture.

Data has always been analyzed within companies and used to help benefit the future of businesses. However, the evolution of how the data stored, combined, analyzed and used to predict the pattern and tendencies of consumers has evolved as technology has seen numerous advancements throughout the past century. In the 1900s databases began as “computer hard disks” and in 1965, after many other discoveries including voice recognition, “the US Government plans the world’s first data center to store 742 million tax returns and 175 million sets of fingerprints on magnetic tape.” The evolution of data and how it evolved into forming large databases continues in 1991 when the internet began to pop up and “digital storage became more cost effective than paper. And with the constant increase of the data supplied digitally, Hadoop was created in 2005 and from that point forward there was “14.7 Exabytes of new information are produced this year" and this number is rapidly increasing with a lot of mobile devices the people in our society have today (Marr). The evolution of the internet and then the expansion of the number of mobile devices society has access to today led data to evolve and companies now need large central Database management systems in order to run an efficient and a successful business.

Data gathered in the 80s and 90s, commonly called “traditional data” was measured in gigabytes and was mainly structured, organized and analyzed using SQL; (SQL stands for structured query language, it is a the standard language to communicate with RDBMS) (American National Standards Institute), as opposed to now where data includes huge volume, high velocity and variety, it is now measured in Petabytes (1 Petabyte = 1000000 Gigabyte). New technologies have been developed to analyze the new types of data (semi - structured, unstructured) using Hadoop systems.

This chapter introduces an industrial and technical review for Hadoop framework with other technologies used with Hadoop system to process bigdata. Hadoop project originally was built and supervised by Apache community. In addition to Apache many other companies whose businesses run on Hadoop are adding more interesting features to Hadoop, some of them announced their own Hadoop distributions replying on the original core distribution distributed by Apache.