Real Time Big Data Processing With Machine Learning

This paper proposes backup task mechanism to improve the straggler tasks which are the final set of MapReduce tasks that take unusually longer time to complete. The simplified programming model proposed in this paper opened up the parallel computation field to general purpose programmers. This paper served as the foundation for the open source distributing computing software – Hadoop as well as tackles various common error scenarios that are encountered in a compute cluster and provides fault tolerance solution on a framework

This is one of the greatest challenge that is acting as a key barrier in widespread adoption and pervasive use of the tools fostering for development and progress in this field of study. The real time of the big data analytics would require some of the unique features and special computing powers and potentials (Gantz, 2012). Tools have to made specially advanced so as to incorporate process terms in real time (Chen, 2012). Every business oriented organization should be transformed into information centric (Kaisler, 2013) to focus upon the real time data analysis in terms of both input and output.

Big Data gives one the ability to collect, manage and analyze large amounts of data, leading to changes in the way companies deal with customers, resulting in increased profits and a competitive edge over other companies. Big data is

Big data refers to the collection and subsequent analysis of any significantly large collection of data that may contain hidden insights or intelligence (user data, sensor data, machine data). when analyzed properly, big data can

Big data is not a hype, but it is the future. The big data industry continues to advance, and big data service providers are making it easier for companies to work with big data in driving their businesses. Progressively, greater volumes and varieties of data will be incorporated with more business processes to support better decision making and greater insight. Moreover,

Big Data is the data that surpasses the processing limit of the customary database frameworks. The data is huge, moves with more speed and does not fit the structure of the conventional database frameworks.

In this section we descus the expected properties of a system designed for performing data analysis at the cloud environment and how parallel database systems and MapReduce-based systems achieve these properties.

With 3.2 billion internet users [1] and 6.4 billion internet connected devices by 2016 [2], unprecedented amount of data is being generated and process daily and increasing every year. The advent of web 2.0 has fueled the growth and creation of new and more complex types of data which creates a natural demand to analyze new data sources in order to gain knowledge. This new data volume and complexity of the data is being called Big Data, famously characterised by Volume, Variety and Velocity; has created data management and processing challenges due to technological limitations, efficiency or cost to store and process in a timely fashion. The large volume and complex data is unable to be handled and/or processed by most current information systems in a timely manner and the traditional data mining and analytics methods developed for a centralized data systems may not be practical for big data.

With 3.2 billion internet users [1] and 6.4 billion internet connected devices by 2016 [2], unprecedented amount of data is being generated and process daily and increasing every year. The advent of web 2.0 has fueled the growth and creation of new and more complex types of data which creates a natural demand to analyze new data sources in order to gain knowledge. This new data volume and complexity of the data is being called Big Data, famously characterised by Volume, Variety and Velocity; has created data management and processing challenges due to technological limitations, efficiency or cost to store and process in a timely fashion. The large volume and complex data is unable to be handled and/or processed by most current information systems in a timely manner and the traditional data mining and analytics methods developed for a centralized data systems may not be practical for big data.

The Apache-Hadoop open-source program that provides a layer of software that can turn a grid of servers into a single unit has become known as NoSQL, or Not-Only SQL database. It provides a parallel storage and processing framework to run MapReduce, an application module that runs in two phases, first mapping data then reducing or transforming it. Using the HDFS or Hadoop Distributed File System, huge data sets are spread across servers, thus enabling programmers to write application modules and run them in parallel on the same cluster that stores the data, thus using the power and capacity of all the CPUs and hard drives. The advantage of Hadoop is that it processes large amounts of data in seconds, one

In Hadoop environment, data from various sources is first loaded and stored in the HDFS framework from the local file system. Map Reduce provides supports distributed and parallel I/O scheduling, it is fault tolerant and supports scalability and has inbuilt processes for status and monitoring of heterogeneous and large datasets as in Big Data. To fetch data from Hadoop File System, MapReduce program is used which is written in Java. The MapReduce program is executed and run in the system. The MapReduce program can be written in many ways as required by the programmer which is used for analyzing the data in different ways. The Mapper function can be run as

Chris Twogood from Teradata [50] and Timothy Mallalieu from Microsoft talked about how important integrating Hadoop with traditional data technologies to get the most out of the new opportunities.

The data is too large, moves too fast or does not meet the constraints of the database

In modern times, the amount of data being stored is terrifically large. Companies must deal with such abundance of data on a daily basis in both storing and analyzing as

There are lots of opportunity associated to big data that help any organization to handle their large amount of data, like in financial sector it store data related to finance, healthcare sector it store health related patient records, doctors detail and medicine ,medical equipment related details . In retail sector it is also used [5]. Web/social media/mobile companies also use it for storing their user detail and data like their likes, search pattern, calling and messaging records. Manufacturing and government sectors also use it.