Taking a Look at MapReduce

Cloud computing is an attracting technology in the field of computer science. Cloud computing is an on demand service in which shared resources, information, software and other devices are provided according to the clients requirement at specific time. It’s a term which is generally used in case of Internet. The whole Internet can be viewed as a cloud.

The main purpose of the study is to provide the information of cloud computing. Here the definitions of cloud computing is described including the briefing about what cloud computing and why it is used. Moreover, it includes brief overview of deployment models and service models of cloud computing. It will help to get a better and clear idea about how organizations can access cloud. It includes advantages and drawbacks of using cloud computing in organizations.

Optimization of existing data warehouse using Hadoop (HDFS) can be implemented by following below seven steps or fundamental processes which are divided into 2 phases.

Several years after the emergence of cloud computing, most of people still confuse with its concept because different research institutes defined the term differently. With the development of cloud computing, its concept evolves gradually in years. According to the Wikipedia, the most recent definition of cloud computing is “the delivery of services of the computing a service rather than a product, whereby shared resources, software, and information are provided to computers and other devices as a utility over a network (typically the Internet).” Cloud computing, in one word, was termed as the many different types of virtualized applications and services that are moved into the “cloud” – a metaphor for the internet. Cloud computing providers maintain and control physical servers, and data is processed in the “cloud” provider’s environment which also delivers hosted services. For the user, the entire computing infrastructures virtually reside “in the cloud.” In many cases, devices that used to access these services and

cloud computing means to deliver services in rapid speed on demand at any time and from any location. Access can be reached from any device. Could computing covers both office based surface and also the out-sourced platform. A lot of survey has been taken place to determine the benefits of cloud computing. As one survey postulate that based on written statement from CIO chief information officers. They reported in their survey that nearly 50 countries in year 2014 the greater number of information technologies’ survives were operating in the cloud computing platform. Also

Cloud technology is attracting increasing attention from industry and scientific areas. It has been treated as one of the most important technologies and shown promising future. Cloud is the technology to provide a quick, secure, convenient net computing and data storage service and implemented over the Internet, which is capable of sharing a pool of configurable computing resources with on-demand network access. Cloud technology provides various specific services such as software application, data storage, hardware, and infrastructure. People access Cloud via the internet through web browsers to finish on-demand services, while this was implemented on traditional deploying on-premises servers in old days.

Over the past several years, cloud computing has been getting attention as a new computing model for providing flexible and on-demand infrastructures, platforms, and software as services. Cloud computing is a technology through exchange of information and software management. Cloud computing allows users to use computer technology without the installation in their computers. It allows users to access their files or data from any computer having an internet connection. Cloud computing provides more efficient computing by centralizing data storage, processing and bandwidth. This technology emphases on virtualization of the host server or the controlling computer. The server acts as a communication network through which information could be shared. The information can be stored, retrieved and shared as and when required through a widespread global platform.

Abstract—Hadoop framework is a solution for big data problem. In a large cluster, thousands of servers both host directly attached storage and execute user application tasks. Big data is only not about storing the data, it is also about execution and analyzing of data.

Modern Data Centers always interested in the new technology for various web search analysis, web log, bigdata analysis, social networking, so in this tasks new technology implemented using parallel processing for large-scale database analysis, so the MapReduce is one of new technology to get amounts of data, perform massive computation, and extract critical knowledge out of big data for business intelligence, proper analysis of large scale of datasets, it requires accurate input output capacity from the large server systems to perform and analyze weblog data which is derived from two steps called mapping and reducing. Between these two steps, MapReduce requires a on important phase called shuffling phase which exchange the intermediate data. So at the point of data shuffling, by physically changing the location(moving) segments of intermediate data across disks, causes major I/O contention and generate the Input/Output problem such as large power consumption, high heat generation which accounts for a large portion of the operating cost of data centers in analyzing such big data. So in this synopsis we introduce the new virtual shuffling approach to enable well-organized data movement and reduce I/O problem for MapReduce shuffling, thereby reducing power consumption and conserving energy. Virtual shuffling is achieved through a combination of three techniques including a three-level segment table, near-demand merging, and dynamic and balanced merging subtrees. Our

The Basic idea in Hadoop is all about parallelization. Parallelization can be easily accomplished, if a processing (a work) can be easily split into n units. Hence the core focus of MapReduce programming framework have been to solve the partition problem.

Hortonworks is a business computer software company based in Palo Alto, California. The company focuses on the development and support of Apache Hadoop, a framework that allows for the distributed processing of large data sets across clusters of computers. Architected, developed, and built completely in the open, Hortonworks Data Platform (HDP) provides Hadoop designed to meet the needs of enterprise data processing.HDP is a platform for multi-workload data processing across an array of processing methods from batch through interactive to real-time - all supported with solutions for governance, integration, security and operations.

Big data has received popularity as datasets which are large that cannot be easily managed by the traditional relational databases. Big data deals with high volume, variety and velocity of data. However, if traditional relational database schemas are applied to big data, the large volume of datasets cannot be processed and managed by these traditional techniques. The notable solution to manage and process the data is MapReduce framework. MapReduce is programming model which programming model where a large data sets is divided into small data sets; these small data sets are processed in parallel by nodes(computers). This is called as batch processing, where each worker node applies map function and reduce function to the associated data to

This technology named as Cloud Technology or Cloud Computing was just a buzzword during the early 2010 but became more prominent and was widely adopted by 2012.Internet in other terms can be termed as a “cloud”. There are two words that define Cloud computing the most: Web Applications. Any organization using huge clusters of computers which require high performance through the host of services available use cloud computing. [1, 2]

Business analysts, Data scientists, Data analysts, Statisticians, want to analyze the data for collecting the important characteristics of the data set. In 2004 Google introduced MapReduce. It simplified the data processing on a large cluster. But many organizations have only few developers who can write good MapReduce code, which is written in java. (MapReduce can be written in other languages).

“MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster” [1]. Before the introduction of MapReduce, biggest challenge in front of Google was processing large input data and distributed computations. There was no solution for parallelizing the computations, distributing the data, and handling failures. In order to solve these problems, Google introduced the concept of MapReduce for parallel computation in a distributed system for synthesizing large datasets.