Data Analysis : Data Cleansing

very important is the elimination of duplicate data, also known as redundant data, which not only

A well-designed database should be able to provide accurate information on-demand, especially if your data is subjected to frequent changes or updates. For this reason, carefully organizing data as structured content in your database is a critical process. One way to do this is through data normalization.

The concepts are the significant advancement in examination and access to quality information. The new aspect of big data that is essential for its application is the variety formats it receive data. The data streams in different format of structured and unstructured (Groves et al., 2016).The data comes in numeric database, videos, audios and email. The data also comes from multiple sources. It is therefore ease to detect the trend and nature of the information in the system. Data management is one of the key aspect of data mining. It involves examination of data and identification based on source and quality (Crockett, & Eliason, 2016). These elements create utility of the information in the system. The health care will therefore get quality information for improving its

BIRCH (balanced iterative reducing and clustering using hierarchies) is an unsupervised data mining algorithm used to achieve hierarchical clustering over particularly huge data-sets. An advantage of Birch is its capacity to incrementally and dynamically cluster incoming, multi-dimensional metric data points in an effort to generate the best quality clustering for a given set of resources (memory and time constraints). In most cases, Birch only requires a single scan of the database. In addition, Birch is accepted as the, "first clustering algorithm proposed in the database area to handle 'noise' (data points that are not part of the underlying pattern) efficiently.

in the above given data lists table case for example (1,2) are unique to y, (3,4) are common, and (5,6) are unique to z.

Data is very important thing in every business, especially in today’s dynamic world where optimal use of data leads to success in shorter span of time as lots of companies are struggling for truthful and accurate data. These data must be analyzed in exact time and in a proper way so that the decision is more effective, but the data we receive are very redundant and carry lot of space in our system. This creates a challenge for the Analytics people to remove the redundancy and bring out only those relevant data that aids in decision making process. Master Data Management is a solution for such Analyst who wants to eliminate the redundant and inconsistent data of the organization (Vinculum, 2016).

In order to collect data for my CSP class, my classmates and I used google forms to answer survey questions such as, how many hours did you sleep, how many hours did you relax, and more. The Springbrook students took this survey for two weeks. It should be noted that on Monday’s survey, there was some confusion because some people thought the hours they slept and hours they worked was a total of both days of the weekend. For example, people said they worked for nine, ten, and even eleven hours, which no one usually works in a day because they totaled the hours they worked both days of the weekend. Also, some people did not take the surveys seriously and didn’t use accurate answers. The raw data was stored in a spreadsheet on google sheets that was a

“Databases, Data Mining and Beyond” is the title of the article by Shamsul Chowdhurry. Professor Chowdhurry is a professor of Information Systems at Roosevelt University. The article focuses on case based reasoning, data mining and databases and their uses and application in various industries. First, let us define case based reasoning. Case based reasoning is a procedure to solve current problems by adapting solutions from past problems and applying it to the current problem to be solved. Case base database can be designed on case base reasoning. The structure of the database would have four steps. The steps include: 1) retrieval of similar cases; 2) adaptation; 3) evaluation and 4) update the case base. Each of the steps is a process that assists in the solving

Relational database normalization entails organizing database and it includes the creation of tables as well as relationships establishment between the tables using designed rules intended to protect the data as well as make the database to be flexible. This is achieved through the elimination of redundancy as well as inconsistent dependency. Redundant data is known to waste the space of the disk thereby creating problems of

Data Cleaning: The dataset consist of set project number and effort multipliers which further is segmented into a set of 15 parameters, development effort and line of code. Relevant information will be fetched from the initial dataset and dataset will be converted into a subset which will help us to get relevant results.

Before a data set can be mined, it first has to be ?cleaned?. This cleaning process removes errors, ensures consistency and takes missing values into account. Next, computer algorithms are used to ?mine? the clean data looking for unusual patterns. Finally, the patterns are interpreted to produce new knowledge.3

Data is often far from perfect. Most of the data mining technique can tolerate and bare some level of imperfection and inconsistency in the data. This imperfection and inconsistency may cause loss of individual’s private data. Preserving privacy data mining focuses on obtaining positive result analysis. The data quality issues that often need to be addressed includes the presence of noise and outliers, missing inconsistent or duplicate data, biased and unrepresentative phenomenon and that the data is supposed to be. For preprocessing and pattern recognition it is mandatory to build the data in representable form before providing it to data mining techniques. In this paper we introduce Noise cure framework as a bridge between data preprocessing and data mining for preprocessing the raw data to bring in a form suitable for pattern recognition in preserving privacy data mining

The era of digitalization considered the data as an important information asset that is required to be safeguarded and stored in an effective manner to maintain the confidentiality, integrity and availability of the information (Conklin, White, Williams, Davis & Cothren, 2012). The effective managing of data helps in accessing the accurate information at right time for organizational decision making purposes to achieve the long term business goals (Conklin et al., 2012). New innovations in the field of technology are used for carrying out research that resulted in entirely new categories of data in boundlessly growing amounts (Strasser, 2014). Research projects includes data in the physical and the digital form that are required to

In every moment of our lives, we handle a tremendous amount of large data, this large data will become a store of values if we could be turned it into searchable information involving analysis steps. The big challenge is that 90% of the large data are unstructured data which is growing faster than structured data, unstructured data as a data warehouse come without any predefined data structure and not appropriate with any relational database schema.

Data de-duplication is a method for sinking the amount of storage space an organization needs to save its data. In most organizations, the storage systems contain duplicate copies of many pieces of data. For example, the identical file may be