Data And Processing Of Data Essay

Every day, we produce 2.5 quintillion bytes of data. 90% of all data in the world was produced in the past two years. Data has been around forever; we have always gathered information. Paleolithic cavemen recorded their activities by carving them in stone or notching them in sticks. Egyptians used hieroglyphics to record significant events in history. The Library of Alexandria was home to half-a-million scrolls of the ancient world. Less than hundred years ago, we used punch cards to record and store information. As technology continues to evolve, the amount of data we store continues to grow. We’ve come a long way since stone tablets, scrolls, and punch cards. It’s important to understand the concept of big data and the impact is has created. This paper will define the classifications of data, explain the challenges of big data, and describe how big data analytics is being used in today’s data driven world.

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

Big data is nothing but collecting of datasets. Organizations in current world demands data to be broken down which can used to get more high effectiveness and benefit. Big data refers to the large amounts of data which collected from various devices such as mobiles, sensors and social media etc. Generally, large amount of data have been regenerating by IT industry such as satellite data, mobile devices and etc. This data is being growing rapidly day by day and it would be referred as Big Data.

The rise of Big Data and its attendant complexities has spawned a whole ecosystem to support the ever growing requirements of a 24x7 world. One of the key technologies coming out of the initial stages of Big Data has been Hadoop. Conceived in response to the rapidly growing needs of Yahoo!’s search engine, Hadoop provides a mechanism to store and collect vast amounts of data across a highly distributed environment using commodity hardware.

Materials Requirement Planning (MRP) is one of the integral production control techniques used in Enterprise resource planning(ERP). It calculates the net requirements for materials based on its dependence on other materials and develop procurement proposals which are often in the form of purchase requisitions and planned orders. The MRP creates planned orders for all finished and semi-finished products and aims to make sure that the company has sufficient availability of material to meet its requirements. In ERP, the Master Data is a key factor in the successful implementation of the production process. It sorts and stores data automatically, based on their functions and distinctive characteristics. The Material Master Data

Hadoop is an open source framework that could be very resourceful in data processing of the complex data systems, and has been reverently used in the recent past for query processing in the complex databases that contains millions of records. The major advantage of Hadoop is that it clusters the entire records to few blocks and the query is run on each cluster and the compiled information is displayed in effective terms.

Hadoop excels with managing and processing file-based data, especially when the data is voluminous in the extreme and the data would not benefit from transformation and loading into a DBMS. In fact, for the kinds of discovery analytics involved with Hadoop, it’s best to keep the data in its raw, source form. This is why Hadoop has such a well-deserved reputation with big data analytics.

Raw data used in generating this report mainly came from 2 sources: United States Department of Agriculture and Federal Reserve Bank of St. Louis Economic Data. All Data manipulated are national, time series data. The frequency of the dada is monthly. All Data were de-trended; the base year is 2007.

The emergence of new technologies, applications and network systems makes it hard to run the current business models and huge data types, and thus emerged various types of analytic tools like Big Data, which make this work easier by way of proper organization of data. Big Data is all about analyzing different forms of data (Structured, Semi-structured and Un-structured) and it is not about the procedure, creation or consumption of data.

arouse mainly because data is asset to Organization , analyzing data is inexpensive and data

Big data is an extensive collection of structured and unstructured data. It is a modern day technology which is applied to store, manage and analyze data that are not possible to manage, store and analyze by using the commonly used software or tools. Since all of our daily tasks are overtaken by the modern technologies and all the businesses and organizations are using internet system to operate, the production of data has increased significantly in past

Data storage and management technologies have recently begun to surge in popularity. Businesses want to learn how to implement the best ways to store, maintain, capitalize on the copious amounts of data that their products, consumers, services, etc. generate. With that being said, organizing and measuring data has proven to be quite difficult despite present-day technological innovations. The term “Big data” has emerged and Apache Hadoop, or Hadoop, technology uses a set of algorithms to process large clusters of data (Kelly, 2014).

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

This provides an overview of the steps that were used for the processing of the data. In general all of the processes were executed using batch macros in ImageJ and the resulting files were saved at each step.

In some sense, the unstructured information is the intriguing information, however it 's hard to syn the Big information is a generally new term yet the definition demonstrates that it has been around for quite a while. Huge information is can be summed up by the three V 's, Volume, Velocity and Variety. Volume is when associations gather information from an assortment of sources, including business exchanges, online networking and data from sensor or machine-to-machine information. Prior to recent advances data storage was not the problem, but the rate at which they could process the data. For example the company Hadoop, an open source software, has combined the processes of storying the data and processing it. “Hadoop transfers packaged code for nodes to process in parallel based on the data that needs to be processed.”(Wiki 1) The speed of information streams in at a remarkable speed and is managed in an efficient way that is necessary in today’s world. RFID labels, sensors and keen metering are driving the need to manage deluges of information in close ongoing. The diverse assortment of Data comes in a wide range of organizations – from organized, numeric information in conventional databases to unstructured content reports, email, video, sound, stock ticker information and money related exchanges. ("What Is Big Data?"; 1)