Скачать с ютуб Enhancing data governance in Trino with the OpenLineage integration в хорошем качестве

Enhancing data governance in Trino with the OpenLineage integration

Many organizations use the Trino engine to perform complex data transformations that result in data flows between multiple data sources. Understanding data linage, the data's entire lifecycle, from its source to its final destination and every transformation in between, gives critical insight around data observability. In this talk, Alok Prusty and Amogh Margoor will discuss how they used OpenLineage, an open framework for data lineage collection and analysis, and built a Trino open-lineage plugin to publish OpenLineage complaint events that can be viewed in Marquez, an open source metadata service. Some of the events that establish the lineage include, but are not limited to CTAS, UPDATE, and INSERT. Using this OpenLineage integration, data governance teams can now see the end to end lineage of all critical datasets that are mutated via Trino queries.