Zero Hop Distributed Key Value Storage System

Hadoop \cite{white2012hadoop} is an open-source framework for distributed storage and data-intensive processing, first developed by Yahoo!. It has two core projects: Hadoop Distributed File System (HDFS) and MapReduce programming model \cite{dean2008mapreduce}. HDFS is a distributed file system that splits and stores data on nodes throughout a cluster, with a number of replicas. It provides an extremely reliable, fault-tolerant, consistent, efficient and cost-effective way to store a large amount of data. The MapReduce model consists of two key functions: Mapper and Reducer. The Mapper processes input data splits in parallel through different map tasks and sends sorted, shuffled outputs to the Reducers that in turn groups and processes them using a reduce task for each group.

0166: In spite of this efficiency in MPI, handling Big Data applications is a challenge job for MPI where fault-tolerance by checkpointing can be impractical at very large-scale data because of its excessive disk access and limited scalability.

Permission to make digital or hard copies of all or part of this work for

With the increasing focus on information and big data the storage capability of datacenters is in a state of flux and confusion. Consequently, many storage problems can be solved with a software-centric infrastructure and move storage from proprietary hardware devices to software-led services that become storage pools in the software-defined datacenter (SDC). HaaS is the answer with its software-defined storage (SDS).

For the purpose of this paper, we are going to focus on these three type of NoSQL database BigTable, Cassandra, DynamoDB.

Internet has been developing very rapidly in wide range. It has become available and accessible for everyone. However, other issues such as storage size, the power consumed by equipment and hardware cost have been constantly increasing. The storage space in data center is no longer meeting our increasing demands. The innovation of cloud computing has emerged in an attempt to solve these and other environmental problems. This make software even more attractive as a service and shaping the way hardware is designed and purchased. According to Architectural requirements for cloud computing systems: An enterprise cloud approach, “The overarching goal of Cloud Computing is to provide on-demand computing services with high reliability, scalability, and availability in distributed environments. Despite this common goal, Cloud Computing has been described in many different ways and no standard definition has been adopted until now.”

The Hadoop disseminated record framework (HDFS) is a dispersed, adaptable, and compact document framework written in Java for the Hadoop system. Some consider HDFS to rather be an information store because of its absence of POSIX consistence and powerlessness to be mounted, however it provides shell orders and Java API strategies that are like other record frameworks. A Hadoop bunch has ostensibly a solitary namenode in addition to a group of datanodes, in spite of the fact that excess alternatives are accessible for the namenode because of its criticality. Each datanode serves up pieces of information over the system utilizing a square convention particular to HDFS. The record framework utilizes TCP/IP attachments for correspondence. Customers utilize remote methodology call (RPC) to convey between each other.

A key-value store database has a set of keys and values, and each value is associated with a key. The implementation of key-value store database is actually a distributed hash table (Stonebraker, 04/2010). Key-Value Stores(KV), which are normally known as a model of NoSQL databases, are widely deployed for data operation and management in purpose of enhancing Internet services due to better scalability, higher efficiency and more availability than existing relational databases systems (Wang, et al., 2014). Because KV stores sacrifice relation model in exchange for fast writing, and they are often featured with simple methods like “put()”,”delete()” and “get()”.

There is a lot of data which is processed daily over the computer network which is very complicated. Therefore it is difficult to handle all these data because this

Development of cloud computing started during 1980’s from the complex and extended roots of Information Technology (IT). With the rise of network and internet technologies in the 1990’s, users could

Over the years it has become very essential to process large amounts of data with high precision and speed. This large amounts of data that can no more be processed using the Traditional Systems is called Big Data. Hadoop, a Linux based tools framework addresses three main problems faced when processing Big Data which the Traditional Systems cannot. The first problem is the speed of the data flow, the second is the size of the data and the last one is the format of data. Hadoop divides the data and computation into smaller pieces, sends it to different computers, then gathers the results to combine them and sends it to the application. This is done using Map Reduce and HDFS i.e., Hadoop Distributed File System. The data node and the name node part of the architecture fall under HDFS.

As mentioned in the introduction, ensuring files confidentiality is a major issue in cloud computing. The authors of this paper introduce a new storage scheme called the HourGlass. In this scheme the cloud server is forced to save files in a way that enable clients challenges, the server’s response ensures clients that their files are safely stored.

Abstract. Cloud computing is the upcoming generation of computation. Internet based computing is known as cloud computing, where software and hardware resources provided to the user on demand. Cloud computing is new paradigm that uses dynamically scalable shared resources e.g. networks, servers, storage, applications and services. It provides different cloud services through which resources can be swiftly providing and released with minimum management efforts. In this paper, we aim to pinpoint the types of cloud, services and difference that shows how effective cloud computing from other paradigm (Grid computing, Utility computing and Mainframe). Last, we will provide an overview on the various challenges and risk associated to cloud computing that deserves future research.

This paper broadly talks about co-locating of compute and storage nodes in HPC architecture to overcome the I/O challenges in scientific applications. In a conventional HPC architecture compute nodes and storage nodes are located far apart and are interconnected via shared network. The bottleneck of the system is the gap between compute node and I/O which continues to widen up with increasing storage. The idea of co-locating distributed storage nodes with compute nodes seems to be an eye catcher, as it allows the applications to perform their computation or manipulate their intermediate results and checkpoints, the data only needed to be transferred over the network for archival purpose only. This paper also discusses the

Imagine how does the Google’s world of database look like? Therefore, nothing is small because google provide everything a user need to find through the database. GFS is implemented to encounter the rapid growing of demands of Google’s data processing requirements. However, Google have difficulties when it comes to managing large amount of data. Depending on the average number of comparable small servers, GFS is mainly designed as a distributed file system that can be run on clusters for more than a thousands of machines. To ease the GFS application development, the file system includes a programming interface used to abstract the management and distribution aspect. While commodity hardware is being tested, GFS does not only being challenged by managing not only on the distribution but also needed to cope with the increased danger of hardware problems. Developers of GFS made an assumptions during the design of GFS is to consider handling the disk faults, machine faults and network faults as being the model rather than the exception. The key challenges faced by GFS is the security of data while scaling up to more than a thousands of computers while managing the multiple terabytes of data