A key information pipeline functionality is to trace information lineage, together with methodologies and instruments that expose information’s life cycle and assist reply questions on who, when, the place, why, and the way information adjustments. Knowledge pipelines remodel information, which is a part of the information lineage’s scope, and monitoring information adjustments is essential in regulated industries or when human security is a consideration. Platforms which have information lineage capabilities embrace Alex Options, Alation, Atlan, Boomi, Collibra, Erwin, IBM, Informatica, Manta, Microsoft, Octopai, Oracle, Exactly, Secoda, Solidatus, SAP, SAS, and Talend. Different information catalog, information governance, and AI governance platforms might also have information lineage capabilities.
“Enterprise and technical stakeholders should equally perceive how information flows, transforms, and is used throughout sources with end-to-end lineage for deeper affect evaluation, improved regulatory compliance, and extra trusted analytics,” says Felix Van de Maele, CEO of Collibra.
While you deploy pipelines, how have you learnt whether or not they obtain, remodel, and ship information precisely? Are information errors captured, and do single-record information points halt the pipeline? Are the pipelines performing persistently, particularly underneath heavy load? Are transformations idempotent, or are they streaming duplicate data when information sources have transmission errors?
