The purpose for this document is to provide basic instructions on how to align the source files and get them prepared for loading into the Azure Blob, with further instructions on how to run the process (they requested that it not run automatically), and how to check the validation report for found issues.
The second part of the document is pseudo-code on the process that the data goes through to get to reporting… including cleansing, alignment and other steps.
With this process we were able to quickly identify many records that didn’t meet set criteria. What took 1+ weeks to manually view and process for reporting, we were able to reduce to about an hour with a 90%+ confidence. We then could send those records back to the originators for correction.
