Common Metadata Framework: Integrated Framework for Trustworthy Artificial Intelligence Pipelines

In this article, we present the capabilities of the Common Metadata Framework (CMF) to enable trustworthy artificial intelligence (AI). CMF is a decentralized framework for tracking metadata and lineages of datasets and machine learning (ML) models in AI pipelines. The framework provides a few unique features for ML practitioners, such as effortless management of distributed AI pipelines that span across the edge, high-performance systems, and public and private clouds. It provides an unbreakable audit trail and model provenance, resulting in trustworthy models. It ensures reproducibility by versioning artifacts and source code. CMF bridges the gap between the pipeline- and model-centric views of the AI metadata. This end-to-end approach to metadata logging unlocks a comprehensive understanding of ML workflows, enabling more efficient management and optimization of AI pipelines.