lakehouse-engine
adidas
293étoiles
The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.
- post-mortem (static) data, using the DQ Validator algorithm (execute_dq_validation)
- data in-motion, using the dq_specs keyword in the Data Loads, to add it as one more step while loading data. On the usage section you will find an example using this type of Data Quality validations.
