I have a database that stores HIPAA protected pdf files. It is a repository for all scanned documents performed by data entry staff at my organization for the legacy EMR we are phasing out. The DB holds over 300k in possible files to extract. These files will be moved to our new hosted EMR using the vendor's data extraction process. We have identified as best we can the files that will go over using the vendor's indexing requirement of attaching a "document type" category to each file that will go over. This is stored in a metadata file where each line is a file with data separators:



We are getting caught up on the following asks thought:

What is the final file count/size of the expected data to be extracted? This is difficult for us to determine because:

The scanned documents over the life of the legacy EMR system had poor oversight with regards to naming conventions and auditing. While the legacy system has static categories that staff were required to allocate a scanned document to the was no auditing procedure in place to account for whether this was being properly applied. E.g. The category "Miscellaneous" was overused and scanned documentation that should go in

  • (a) Progress Notes or
  • (b) Clinical Assessments

was performed possibly thousands of times.

The scanned documents also were given a file name by data entry staff however, again, there was no standard naming convention applied, so there is no obvious "rule" to best identify which scanned documents are desired. So looping back to my original question:

Given that the metadata file that my team generated is likely inaccurate and 2) it is separate from the DB that holds the physical data what, if any, application can perform an automated process that uses some list of keywords provided that can identify the file count/size of scanned data and possibly copy the the desired files to a new DB folder which can then used to generate a more accurate metadata file? That's my logic for this project but I admit it may not be the best approach so any alternate suggestions are appreciated.

  • 1
    if nobody comes up with any useful idea here, maybe re-post this with question to the datascience.SE site (or rather, remove and repost . conforming to site terms of service)
    – knb
    Nov 3, 2020 at 16:04
  • reposted to Data Science datascience.stackexchange.com/questions/84958/…
    – jam320
    Nov 4, 2020 at 21:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.