Notes On Hadoop And Mark Logic

Every day, we produce 2.5 quintillion bytes of data. 90% of all data in the world was produced in the past two years. Data has been around forever; we have always gathered information. Paleolithic cavemen recorded their activities by carving them in stone or notching them in sticks. Egyptians used hieroglyphics to record significant events in history. The Library of Alexandria was home to half-a-million scrolls of the ancient world. Less than hundred years ago, we used punch cards to record and store information. As technology continues to evolve, the amount of data we store continues to grow. We’ve come a long way since stone tablets, scrolls, and punch cards. It’s important to understand the concept of big data and the impact is has created. This paper will define the classifications of data, explain the challenges of big data, and describe how big data analytics is being used in today’s data driven world.

The concept of Big Data management is becoming more important to modern organizations by giving them the ability, from its analytics, to predict future trends and make better decisions. The advancements in modern technology has greatly revolutionized Big Data management by providing better techniques for data collection, storage, transmission, presentation, and analytics. The use of the internet has boosted availability of data for many organizations, giving a speedier growth to data.

Since the 1970’s databases and report generators have been used to aid business decisions. In the 1990’s technology in this area improved. Now technology such as Hadoop has gone another step with the ability to store and process the data within the same system which sparked new buzz about “big data”. Big Data is roughly the collection of large amounts of data – sourced internally or externally - applied as a tool – stored, managed, and analyzed - for an organization to set or meet certain goals.

When a business started, the related data has been generated. Along with the development of the business, the data becomes larger and accumulates day by day. The accumulated data can be transferred to meaningful information for the managers’ decision makings. Storing and protecting the data is essential to the development of the enterprise business. Today, almost all enterprises use computer storage to store their data instead of the hard copy papers. In the past, hard copy papers stored the data for the enterprise, but it existed too many potential risks, such as data loss, data stolen, data destroy, or cannot sharing with others. In 1990s, more companies started to use computer hard drive to store their business data; this change impacted the later development of the storage strategies. Along with the technology development and the data becomes bigger and bigger, big data problems appear. How can enterprise store such large data and manage such large data effectively? Currently there are several classic storage strategies to help enterprise solve their data storage problems, such as spinning media, flash, and cloud. The enterprise also can select onsite or offsite option as their considerations based on their business environment. Security issues are popular discussion topics as well in this paper.

With the buzz around many big data applications, privacy concerns regarding their uses have also grown. With the personal data has been mined and published every day, the battle to reclaim the privacy starts vigorously. E-commerce websites harvests information about all the online searches of customers. Social Media exposes the likes and preferences of people, their photos and all their daily activities. Video surveillance monitors the movement of people. The data gets published from health care, censes and other government agencies. With such potential of harvesting, mining and publishing data, the risks of using big data is more than anything [29].

Data is a forever growing thing and is not looking to slow down anytime soon. With that be said, companies must be able to manage data for their day to day operations as well as for predicting future trend. Now there are many tools for managing data and extremely high cost for managing it as well. One of the costs for managing data is data security, but this paper will touch on that topic in, but not as much as other topics. Also, this paper will demonstrate various business cases to serve as testimonial for just how useful big data management it in to day society.

Hadoop is one of the open source frameworks, is used as extension to big data analytics framework which are used by a large group of vendors. This type of framework makes work easy for the companies how they?re going to store and can use the data within the digital products as well as physical products (James, M. et al. 2011). We can analyze data using Hadoop, which is emerging as solution to

The main purpose of this report is to provide a critical review of the processes and own experiences of Hadoop within the context of the assignment which was given to us. The review concentrates on the discussion and evaluation of the overall steps followed during the progress of the project and the reasons for which we have chosen these particular steps. It also draws attention at the main points that were accomplished, both with respect to individual, and with respect to the group 's perspectives. Finally, it concentrates on the project 's progress in terms of changes for a future implementation.

Big Data has many challenges and opportunities associated with it, which necessitates us to rethink on aspects such as data management in order to attain desirable outputs. The next generation of BDA lies in its data management and its associated systems, principles and platforms. This will indeed make Big Data in creating a new wave of technological advancements.

According to the article, 2.8ZB of data has been created and replicated in 2012. The proliferation of devices such as PCs and smartphones worldwide, increased Internet access within emerging markets and the boost in data from machines such as surveillance cameras or smart meters has contributed to the doubling of the digital universe. IDC projects that the digital universe will reach 40 ZB by 2020, an amount that exceeds previous forecasts by 14%. Thus, data is not narrowed to big only; it is actually huge. Like, 40 ZB data is the equivalent of 1.7 MBs of new information created by the every single human for every second of the day. Developing countries like China and India are currently covering 36% of digital universe; the prediction says it will be increased up to 62% by 2020. So the companies will have numerous scopes to dig out more data and analyze them as per their requirement. Despite the unprecedented expansion of the digital universe due to the massive amounts of data being generated daily by people and machines, IDC estimates that only 0.5% of the world’s data is being analyzed.

Big data is an extensive collection of structured and unstructured data. It is a modern day technology which is applied to store, manage and analyze data that are not possible to manage, store and analyze by using the commonly used software or tools. Since all of our daily tasks are overtaken by the modern technologies and all the businesses and organizations are using internet system to operate, the production of data has increased significantly in past

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

Data has always been analyzed within companies and used to help benefit the future of businesses. However, the evolution of how the data stored, combined, analyzed and used to predict the pattern and tendencies of consumers has evolved as technology has seen numerous advancements throughout the past century. In the 1900s databases began as “computer hard disks” and in 1965, after many other discoveries including voice recognition, “the US Government plans the world’s first data center to store 742 million tax returns and 175 million sets of fingerprints on magnetic tape.” The evolution of data and how it evolved into forming large databases continues in 1991 when the internet began to pop up and “digital storage became more cost effective than paper. And with the constant increase of the data supplied digitally, Hadoop was created in 2005 and from that point forward there was “14.7 Exabytes of new information are produced this year" and this number is rapidly increasing with a lot of mobile devices the people in our society have today (Marr). The evolution of the internet and then the expansion of the number of mobile devices society has access to today led data to evolve and companies now need large central Database management systems in order to run an efficient and a successful business.

Data or information is the kernel of the modern company. Its availability, integrity and confidentiality are basic to the long-term survival of any 21st-century company. Each company need a complete and scientific method to protect its data, otherwise, the company would be weak in front of risks. These risks are not limited to internet companies. In fact, any company that have secret information have to counter with the risks of data security. In a word, both public and private data could be affected by risks. In UK, almost every company run some external guidance or

Allows hardware infrastructure to scale up and down with no need to change data formats