Data Cleaning Case Study

Data mining has become an important process in many business applications and decisions. So to preserve the quality of data on which data mining is performed becomes essential. Data cleansing therefore becomes a task of critical importance for the effective as well as efficient data mining. Duplicate detection is a sub task of data cleansing. Duplicates are the multiple representations of same real world object in data sources. Duplicates may occur due to various reasons such as typographical errors or due to inconsistent representations of same real world object. This problem of duplicate detection has been studied comprehensively for structured data such as relational databases and lots of work has been done on the same. But this work cannot be applied

The enterprise data repository (EDR) project at InsuraCorp was developed to be the data warehouse for customer and product data for all InsuraCorp business units. There is a school of thought that data management responsibilities should fall to IT and to the business units themselves. The collaboration between the IT and business users together could produce higher quality data and administer data management more effectively. Everyone who receives or accesses information within an organization is responsible for data integrity so it only stands to reason all parties have a responsibility. Both the information system managers and the business managers, as data stewards, are duty-bound to monitor and control data accuracy. With data, it is as important to have accurate input so that the information that is shared will be useful to other users. Storing data in a holding tank will not solve a bad data problem.

Data can be stored in data warehouses. However, they need data cleansing to standardise the data.

Data Cleaning: The dataset consist of set project number and effort multipliers which further is segmented into a set of 15 parameters, development effort and line of code. Relevant information will be fetched from the initial dataset and dataset will be converted into a subset which will help us to get relevant results.

Be it validating the data of a medical device or a database or an instrument, assuring data completeness and accuracy is not just pertained to individual components. It is more to do with managing the entire lifecycle of enterprise-data of an organization and ensuring data integrity throughout the IT systems. So is the case with SharePoint®.

A data warehouse is a large databased organized for reporting. It preserves history, integrates data from multiple sources, and is typically not updated in real time. The key components of data warehousing is the ability to access data of the operational systems, data staging area, data presentation area, and data access tools (HIMSS, 2009). The goal of the data warehouse platform is to improve the decision-making for clinical, financial, and operational purposes.

analysis) lies in the kind of information that we want to estimate or unearth from the data. The

Essentially all Applications have dirty data that calls for cleansing. This cleansing might involve manual data cleansing, automated data cleansing or even a combination of the two. Some contributors to Faulty data may include:

ETL Tools provide facility to extract data from different non-coherent systems, transform (cleanse & merge it) and load into target systems. The main goal of maintaining an ETL process in an organization is to migrate and transform data from the source OLTP systems to feed a data warehouse and form data marts. ETL process is the basis of BI and it is a prime decisive factor for success or failure of BI.

After collecting the data, you will be faced with challenges to clean and analyze it with appropriate methods in order to derive meaningful conclusions from the data. For example, you will need to structure the individual data processing and analysis steps and to automate and eventually provide the results for implementation. Without the right data analysis methods, tools and other necessary resources, you will not be able to exploit the potential knowledge in the data.

Always access and process the data you need. Improve data integrity at the source with automatic processes that consolidate, cleanse and standardize your data directly in your operational environments. Offer a collaborative environment with a common set of tools that promote the reuse and sharing of data to achieve faster results and lower costs. Deliver consistent, trusted and verifiable

The make-or-buy analysis is heavily depending on the accuracy of the company’s database. Therefore, I need to make sure that I have maintained the database and updated the information correctly. Moreover, I also need to perform several analysis from numerous files in the database and required the ability to analyze a huge amount of data effectively and in a timely manner.

Data Quality provides business intelligence for research, fraud detection, and planning. Better Data Quality leads to better analysis in research, fraud detection and in the part of planning.

Data Transformation are often very complex and is the most costly section of the ETL process. Transformations are often achieved outside the database using flat files, but mostly occurs within an Oracle database. The transform step applies rules or functions to the extracted data. These rules or functions will decide on the analysis of data and can involve transformations like the following:

Data practices have been very difficult to define, organize and store when it comes to following a certain methodology. I currently work for a company that is heavy on data analytics and has had a difficult time defining itself on a certain methodology. Yu Qian and Kang Zhang talk a lot about the booming of data mining and its natural relation to the waterfall method in The Role of Visualization in Effective Data Cleaning. At first, the title doesn’t necessarily tell you that this article will be heavy on waterfall and modeling. If you have or are currently working in the data world, then yes, you have probably already run through the different topics of what data mining naturally consists of. Data is not naturally connected with the