Data De-Duplication Essay

Storage of data plays a major role in improving the performance of a company and this can happen either offline or online and in various formats.

Data mining has become an important process in many business applications and decisions. So to preserve the quality of data on which data mining is performed becomes essential. Data cleansing therefore becomes a task of critical importance for the effective as well as efficient data mining. Duplicate detection is a sub task of data cleansing. Duplicates are the multiple representations of same real world object in data sources. Duplicates may occur due to various reasons such as typographical errors or due to inconsistent representations of same real world object. This problem of duplicate detection has been studied comprehensively for structured data such as relational databases and lots of work has been done on the same. But this work cannot be applied

The repetition of data across multiple sites computer is one natural form of distributed data bases.

A company allows its staffs in the same groupor department to store and share files in the cloud. By utilizing the cloud, the staffs can be completely released from the troublesome local data storage and maintenance.However, it also poses a significant risk to the confidentialityof those stored files. Specifically, the cloud serversmanaged by cloud providers are not fully trusted by userswhile the data files stored in the cloud may be sensitive andconfidential, such as business plans. To preserve dataprivacy, a basic solution is to encrypt data files, and thenupload the encrypted data into the cloud. Unfortunately,designing an efficient and secure data sharing scheme forgroups in the cloud is not an easy task due to the followingchallenging issues.First, identity privacy is one of the most significantobstacles for the wide

It is not only responsible for storing data but also to provide security along with it. It also allows accessing data effectively; integrating data to enhance service intelligence, analysis based on the services required and most importantly increases the storage efficiency.

Whatever the data that is produced is not useful, it could be mostly useful or completely waste of data storage it can be filtered a compressed by other methods.

This causes an inconsistency in the data since it was only updated in one location and not all locations. Insertion anomalies occur due to the impossibility to store some information unless other information is stored as well. Deletion anomalies are caused by information being deleted that may cause other information to be deleted as well. This means it may not be possible to delete some information without deleting other information as well. That is why it is important to use normalization when creating databases to help alleviate these problems.

Reduce time to access the required data: DDBMS allows to store copies of a data in multiple branches.

[15] W. K. Ng, Y. Wen, and H. Zhu. Private data deduplicationprotocols in cloud storage. In S. Ossowski and P. Lecca, editors,Proceedings of the 27th Annual ACM Symposium on Applied Computing,pages 441–446. ACM, 2012.

Data Redundancy: Data redundancy is where a duplicate of information is sorted into different tables/databases. Sometimes data redundancy is done on purpose as a backup of data as a precaution just in case something happens and the data gets deleted. Data redundancy creates a new piece of data so that any modifications, addition of new data or deletion of data will be done on a new piece so that you will always have the

Having a backup of data is important because if the data becomes corrupted or lost you will have it copied and stored so that you will still have it in future for if something goes wrong.

Databases allow us to easily store and retrieve data in a purely digital format. The strength of this is that large amounts of data can be stored and retrieved with minimal effort on the part of the user. Opposed to manually flipping through files, one can quickly pull up the requested data through a computer program. Many systems that were conventionally paper and file based have been converted to a digital format which are now stored in one or more databases.

From the viewpoint of data security, which has always been an important form of quality of service, Cloud computing surely poses new challenging security threats for number of reasons. Firstly, traditional cryptographic primitives for the purpose of data security protection cannot be straight accepted due to users loss control of data under cloud computing. Therefore, verification of correct data storage in the cloud must be attended without accurate knowledge of the whole data. Considering different kind of data for every user stored in the cloud and the requirement of long term continuous security of their data safety, the problem of checking correctness of data storage in the cloud becomes even more challenging.[1]

one of the important data compression techniques for eliminating the duplicate copies of repeating data is the data deduplication, and has been widely used in cloud storage to reduce the amount of storage space and save bandwidth. This method is useful when there are number of copies that having the duplication in private cloud, which will reduce the amount of repeated files after applying this technique we will be obtaining the files with no duplication .To protect the confidentiality of sensitive data while supporting deduplication, the convergent encryption technique has been proposed to encrypt the data before outsourcing. To better protect data security, this paper makes the first attempt to formally address the problem of authorized data deduplication. Different from the traditional deduplication systems, the differential privileges of users are further considered in duplicate check besides the data itself. We also present several new deduplication constructions supporting authorized duplicate check in private cloud architecture. As we are using the private cloud architecture which makes our processing removal of the duplicate files easily. more than that security that is been provided in this is mostly high . Security analysis demonstrates that our scheme is secure in terms of the definitions specified in the proposed security model. As a proof of concept, we implement a prototype of our proposed authorized duplicate check scheme and conduct testbed

The paper presents the idea of dynamic data deduplication model. The main motivation of authors is to use deduplication on dynamic nature of cloud data. The current approaches on deduplication on cloud are mostly focused on static nature of data. They emphasized that we are not utilizing full power of deduplication as most volume of data on cloud are dynamic in nature. It starts with detailed discussion on cloud computing and deduplication. Authors then analyze existing work on cloud