Data validation library in Python and user defined rules

Question

I'm starting a solution that matches data rows (importing a csv file initially) from created rules. It would be like an Excel filter solution with some specific options for date and number formats. But my intended output is a list of matches for these created rules, then those results would be made available for a notification system.

I'm wondering if there is some python library/solution that already does this or something half-through.

Answer 1

Pandas can do a lot of the work for you and is well worth looking at.

• Reading .csv files, (also .xlsx & others)
• Format conversions for date & value columns
• Filtering

Are all available.

Answer 2

Python's built in CSV support lets you import a CSV file and manipulate it.

You can do this as rows to iterate over (and filter):

import csv

def filter_file(filename: str):
  with open(filename, newline='') as csvfile:
    spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')

    return [
      row
      for row in spamreader
      if row[2] != "skipme"
    ]

Or as a list of dictionaries, to make lookups easier via column headers:

import csv

def filter_file(filename: str):
  with open(filename, newline='') as csvfile:
    reader = csv.DictReader(csvfile)

    return [
      row
      for row in spamreader
      if row["should_skip"] != "skipme"
    ]