Phase 18 · Lesson 26 of 30

Model, System, and Dataset Cards

Three documentation formats structure AI transparency. Model Cards (Mitchell et al. 2019) — nutrition labels for models: training data, quantitative disaggregated analyses, ethical considerations, caveats; only 0.3% of Hugging Face model cards document ethical considerations (Oreamuno et al. 2023). Datasheets for Datasets (Gebru et al. 2018, CACM) — motivation, composition, collection process, labeling, distribution, maintenance; electronics-datasheet analogy. Data Cards (Pushkarna et al., Google 2022) — modular layered detail (telescopic, periscopic, microscopic) as boundary objects for diverse readers. 2024-2025 developments: automated generation via LLMs (CardGen, Liu et al. 2024); model-card detail correlates with up to 29% download increase on HF (Liang et al. 2024); verifiable attestations (Laminator, Duddu et al. 2024); sustainability reporting additions for carbon/water (Jouneaux et al. July 2025); EU/ISO regulatory cards emerging. System Cards (Sidhpurwala 2024; Meta system-level transparency; "Blueprints of Trust" arXiv:2509.20394) — end-to-end AI system documentation covering security capabilities, prompt-injection protection, data-exfiltration detection, alignment with human values.

Lesson 26 of 30Build~60 minutesPython (stdlib, model-card + datasheet + system-card generator)

Before this: Lesson 18 of Phase 18 (safety frameworks), lesson 24 of Phase 18 (regulatory)

On this page

Learning Objectives

  • Describe the original Mitchell et al. 2019 model card and the Gebru et al. 2018 datasheet.
  • Describe Data Cards' telescopic/periscopic/microscopic layering.
  • Describe System Cards and their end-to-end coverage.
  • State three 2024-2025 developments (automated generation, verifiable attestations, sustainability reporting).

The Problem

Regulatory frameworks (Lesson 24) and lab safety policies (Lesson 18) both require documentation. Documentation formats evolved from model-specific (model cards) to dataset-specific (datasheets) to system-specific (system cards). Each addresses a different scope of transparency. The 2024-2025 automation and verifiable-attestation work addresses the long-standing adoption problem.

The Concept

Model Cards (Mitchell et al. 2019)

Sections:

  • Model details.
  • Intended use.
  • Factors (relevant demographic or environmental factors for evaluation).
  • Metrics.
  • Evaluation data.
  • Training data.
  • Quantitative analyses (disaggregated by factors).
  • Ethical considerations.
  • Caveats and recommendations.

Adoption problem: Oreamuno et al. 2023 audit of Hugging Face model cards found only 0.3% document ethical considerations.

Datasheets for Datasets (Gebru et al. 2018)

Electronics-datasheet analogy. Sections:

  • Motivation (why was the dataset created).
  • Composition (what is in it).
  • Collection process (how was it assembled).
  • Labeling (if applicable).
  • Uses (intended, prohibited, risks).
  • Distribution.
  • Maintenance.

Published in CACM 2021. The datasheet is the upstream documentation; the model card depends on the datasheet being accurate.

Data Cards (Pushkarna et al., Google 2022)

Modular layered detail. Three zoom levels:

  • Telescopic. High-level summary for non-experts.
  • Periscopic. Middle-level overview for ML practitioners.
  • Microscopic. Detailed feature-level documentation for auditors.

Boundary-object framing: different readers extract different information from the same document.

System Cards

Scope: end-to-end AI system including model + safety stack + deployment context. Sections typically include:

  • Security capabilities.
  • Prompt-injection protection.
  • Data-exfiltration detection.
  • Alignment with stated human values.
  • Incident response.

Sidhpurwala 2024 and Meta system-level transparency work. "Blueprints of Trust" (arXiv:2509.20394) formalizes the System Card as the deployment-layer complement to Model Cards.

2024-2025 developments

  • CardGen (Liu et al. 2024). Automated model-card generation via LLMs; reports higher objectivity than many human-authored cards on the standardized Mitchell 2019 fields.
  • Download correlation (Liang et al. 2024). Detailed model cards correlate with up to 29% higher download rates on HF — adoption pressure is now market-driven, not only compliance-driven.
  • Laminator (Duddu et al. 2024). Verifiable attestations via hardware TEE / cryptographic signatures — allows the model card to carry a proof-of-claim, not just a claim.
  • Sustainability (Jouneaux et al. July 2025). Additions for carbon, water, and compute-energy footprint; emerging ISO standards.
  • Regulatory cards. EU AI Act (Lesson 24) GPAI Code of Practice Transparency chapter requires model cards as a compliance artifact.

Where this fits in Phase 18

Lessons 24-25 are regulatory and CVE layers. Lesson 26 is the documentation layer. Lesson 27 is training-data governance, which is the datasheet's upstream. Lesson 28 is the research ecosystem that produces evaluations referenced in cards.

an-card-scopes

Use It

code/main.py generates a minimal model card, datasheet, and system card for a toy deployment. Each follows the canonical section structure. You can inspect the format and compare the three scopes.

Ship It

This lesson produces outputs/skill-card-audit.md. Given a model card, datasheet, or system card, it audits section coverage, numerical disaggregation, and whether verifiable attestations are present.

Exercises

  1. Run code/main.py. Inspect the generated cards. Identify sections that are weak (placeholder-only) and specify what evidence would strengthen them.

  2. Extend the model card with a quantitative disaggregated analysis across two demographic groups (Lesson 20).

  3. Read Oreamuno et al. 2023 on the 0.3% adoption rate. Propose one structural change to the model card specification that would increase ethical-considerations adoption.

  4. Laminator (Duddu et al. 2024) uses TEEs for verifiable attestations. Design a model-card field that carries a cryptographic attestation of an evaluation result and describe the verifier's role.

  5. Write a System Card (System Card, not Model Card) for one of your past projects or a hypothetical deployment. Identify the highest-value section for third-party auditors.

Key Terms

Term What people say What it actually means
Model Card "the Mitchell card" Mitchell et al. 2019 standard documentation for ML models
Datasheet "the Gebru datasheet" Gebru et al. 2018 standard documentation for datasets
Data Card "the Pushkarna card" Google 2022 modular layered data documentation
System Card "the deployment card" End-to-end AI system documentation including safety stack
Boundary object "different readers, one doc" Data Cards framing: same document serves diverse audiences
Verifiable attestation "the Laminator attestation" Cryptographic or TEE proof attached to a documentation claim
Sustainability field "carbon / water footprint" Emerging 2025 addition for environmental accounting

Further Reading

Check yourself

Quick quiz

  1. Which best describes the original Mitchell et al. (2019) Model Card?

    1. A model checkpoint manifest only
    2. A regulatory legal contract
    3. A 'nutrition label' for ML models: training data, intended use, factors, metrics, evaluation data, quantitative disaggregated analyses, ethical considerations, caveats
    4. An OpenAPI specification
    Show the answer

    C. A 'nutrition label' for ML models: training data, intended use, factors, metrics, evaluation data, quantitative disaggregated analyses, ethical considerations, caveats

  2. What did Oreamuno et al. (2023) find about ethical-considerations adoption in Hugging Face model cards?

    1. No model card has any text at all
    2. Approximately 0.3% document ethical considerations
    3. Approximately 80% document ethical considerations
    4. All model cards document them
    Show the answer

    B. Approximately 0.3% document ethical considerations

  3. What does Pushkarna et al. (Google 2022) Data Cards' telescopic / periscopic / microscopic layering provide?

    1. A pre-RLHF audit checklist
    2. A telescope-themed visualization library
    3. A boundary-object framing where the same document serves different readers (non-experts, ML practitioners, auditors) via three zoom levels of detail
    4. A privacy-preserving query interface
    Show the answer

    C. A boundary-object framing where the same document serves different readers (non-experts, ML practitioners, auditors) via three zoom levels of detail

  4. What scope does a System Card cover that Model Cards do not?

    1. Only the tokenizer
    2. End-to-end AI system: model + safety stack + deployment context, including security capabilities, prompt-injection protection, data-exfiltration detection, alignment, and incident response
    3. Only the prompt template
    4. Only the dataset
    Show the answer

    B. End-to-end AI system: model + safety stack + deployment context, including security capabilities, prompt-injection protection, data-exfiltration detection, alignment, and incident response

  5. What did Liang et al. (2024) report about model-card detail and Hugging Face download rate?

    1. Only ethical-considerations sections affected downloads
    2. Detail decreased download rates
    3. Detailed model cards drive a +29.0% weekly download lift on HF for the treatment group vs control (diff-in-diff); adoption pressure is now market-driven, not only compliance-driven
    4. Detail had no effect on downloads
    Show the answer

    C. Detailed model cards drive a +29.0% weekly download lift on HF for the treatment group vs control (diff-in-diff); adoption pressure is now market-driven, not only compliance-driven

  6. What does Laminator (Duddu et al. 2024) contribute to documentation?

    1. An automated benchmarking harness
    2. Verifiable attestations via hardware TEE / cryptographic signatures, so the card can carry a proof-of-claim instead of just a claim
    3. An LLM-based summary generator
    4. A replacement for the Datasheets standard
    Show the answer

    B. Verifiable attestations via hardware TEE / cryptographic signatures, so the card can carry a proof-of-claim instead of just a claim

Questions people ask

What is a model card?

A model card is a short document published with an AI model. It covers what the model is for, how it was trained, how it performs across different groups and tasks, known limits and ethical concerns. It helps users decide whether the model fits their case.

What is the difference between a model card and a system card?

A model card describes one model. A system card describes a full deployed product, including the model, its safety filters, tools and policies, and usually reports safety testing. Large AI labs publish system cards with major releases.

Run the code Read it on GitHub

Next lessonData Provenance and Training-Data GovernancePrevious: EchoLeak and the Emergence of CVEs for AI

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes