I am a newbie to python. I am working on a CSV file where it has over a million records. In the data, every Location has a unique ID (SiteID). I want to filter for and remove any records where there is no value or mismatch between SiteID and Location in my CSV file. (Note: This script should print the lines number and mismatch field values for each record.) lines = [] count = 0 # read line with open(r"air-quality-data-continuous.csv",'r') as fp:          # read an store all lines into list     lines = fp.readlines()

COMPREHENSIVE MICROSOFT OFFICE 365 EXCE
1st Edition
ISBN:9780357392676
Author:FREUND, Steven
Publisher:FREUND, Steven
Chapter6: Creating, Sorting, And Querying A Table
Section: Chapter Questions
Problem 12AYK
icon
Related questions
Question

I am a newbie to python. I am working on a CSV file where it has over a million records. In the data, every Location has a unique ID (SiteID). I want to filter for and remove any records where there is no value or mismatch between SiteID and Location in my CSV file. (Note: This script should print the lines number and mismatch field values for each record.)
lines = []
count = 0

# read line
with open(r"air-quality-data-continuous.csv",'r') as fp:
    
    # read an store all lines into list
    lines = fp.readlines()
    
    print(str(len(lines)) + 'lines in input file')
    
with open(r"crop.csv", 'w') as fp:
    
    fp.write(lines.pop(0))
    
    #iterate each line
       for number, line in enumerate(lines):
        if (line[4] == 'NaN'):
            print('Empty Site ID found in line:' + str(number))
            continue
        if(str(line[4]) != line[17]):
            fp.write(line)
            count +=1
        
print(str(count + 1) + 'lines written to filter.csv')
        
print(str(count + 1) + 'lines written to crop.csv')

Expert Solution
steps

Step by step

Solved in 2 steps with 2 images

Blurred answer
Knowledge Booster
File Input and Output Operations
Learn more about
Need a deep-dive on the concept behind this application? Look no further. Learn more about this topic, computer-science and related others by exploring similar questions and additional content below.
Similar questions
  • SEE MORE QUESTIONS
Recommended textbooks for you
COMPREHENSIVE MICROSOFT OFFICE 365 EXCE
COMPREHENSIVE MICROSOFT OFFICE 365 EXCE
Computer Science
ISBN:
9780357392676
Author:
FREUND, Steven
Publisher:
CENGAGE L