From c8e96f2172a5c91deafd2c80908f4e797024c357 Mon Sep 17 00:00:00 2001 From: Seth Michael Larson Date: Mon, 18 Dec 2023 11:04:37 -0600 Subject: [PATCH 1/6] Add initial guide for maintaining Software Bill-of-Materials --- developer-workflow/index.rst | 1 + developer-workflow/sbom.rst | 45 ++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 developer-workflow/sbom.rst diff --git a/developer-workflow/index.rst b/developer-workflow/index.rst index 23f1909c9..7b069021b 100644 --- a/developer-workflow/index.rst +++ b/developer-workflow/index.rst @@ -13,3 +13,4 @@ Development workflow lang-changes grammar porting + sbom diff --git a/developer-workflow/sbom.rst b/developer-workflow/sbom.rst new file mode 100644 index 000000000..83e58319f --- /dev/null +++ b/developer-workflow/sbom.rst @@ -0,0 +1,45 @@ +Software Bill-of-Materials (SBOM) +================================= + +Software Bill-of-Materials (abbreviated as "SBOM") is a document for sharing information about +software and how it's been composed. This format is used most often in the security space for +checking software and its dependencies for vulnerabilities using vulnerability databases like +`CVE `_ and `OSV `_. The SBOM format that the CPython project uses is `SPDX `_ +which can be transformed into other formats if necessary by consumers. + +There are multiple sources of third-party dependencies for CPython. +Some are vendored into the source code of CPython itself (like ``mpdecimal`` vendored at ``Modules/_decimal/libmpdec``) +or they could be optionally pulled in during builds like Windows using dependencies from the `python/cpython-source-deps `_ repository. + +Whenever adding or updating a third-party dependency, an update will likely need to be done +to the SBOM in order to track the version and software identifiers. + +Updating a dependency +--------------------- + +The SBOM for CPython's bundled dependencies is kept at ``Misc/sbom.spdx.json``. Instead of updating this document +directly, there is a tool for automatically updating the document at ``Tools/build/generate_sbom.py``. +The recommended workflow is: + +* Update the vendored code in-place. Take note of the new version, download URL, and checksum of the downloaded archive. +* Edit ``Misc/sbom.spdx.json`` to add the new ``versionInfo``, ``downloadLocation``, ``checksums``, and ``externalReferences`` for the corresponding ``package``. + Don't update any information in ``files`` and ``relationships`` as this will be generated automatically later. +* Run ``$ make regen-sbom`` or ``Tools/build/generate_sbom.py``. Ensure that this doesn't fail with validation errors. +* Run ``$ git diff Misc/sbom.spdx.json`` and check the diff matches the expected changes. +* Commit the changes to ``Misc/sbom.spdx.json`` along with the update to the dependency code. + +Adding or removing a dependency +------------------------------- + +When adding a dependency it's important to have the following information: + +* Name, version, and download URL of the project +* License of the project as an `SPDX License Expression `_ +* Software identifiers that match values in vulnerability databases (`CPE `_ and `Package URLs `_ or "PURLs") + +Add the information into a new entry in ``packages`` in the file ``Misc/sbom.spdx.json``. +If a new license ID is to be used, add the license expression to ``ALLOWED_LICENSE_EXPRESSIONS`` +in the SBOM generation tool. + +When removing a dependency, remove the entry from the ``Misc/sbom.spdx.json`` under the ``packages`` field +and then remove the corresponding entry in ``Tools/build/generate_sbom.py`` and then run ``make regen-sbom``. From 4e60f1e3df70680f0d0478c301d8ee0b5d39e3f3 Mon Sep 17 00:00:00 2001 From: Seth Michael Larson Date: Mon, 18 Dec 2023 14:14:16 -0600 Subject: [PATCH 2/6] Incorporate review suggestions --- developer-workflow/sbom.rst | 69 +++++++++++++++++++++++-------------- 1 file changed, 43 insertions(+), 26 deletions(-) diff --git a/developer-workflow/sbom.rst b/developer-workflow/sbom.rst index 83e58319f..9d0db54d5 100644 --- a/developer-workflow/sbom.rst +++ b/developer-workflow/sbom.rst @@ -1,45 +1,62 @@ Software Bill-of-Materials (SBOM) ================================= -Software Bill-of-Materials (abbreviated as "SBOM") is a document for sharing information about -software and how it's been composed. This format is used most often in the security space for -checking software and its dependencies for vulnerabilities using vulnerability databases like -`CVE `_ and `OSV `_. The SBOM format that the CPython project uses is `SPDX `_ +Software Bill-of-Materials (abbreviated as "SBOM") is a document for sharing +information about software and how it's been composed. This format is used +most often in the security space for checking software and its dependencies +for vulnerabilities using vulnerability databases like +`CVE `_ and `OSV `_. The SBOM format +that the CPython project uses is `SPDX `_ which can be transformed into other formats if necessary by consumers. There are multiple sources of third-party dependencies for CPython. -Some are vendored into the source code of CPython itself (like ``mpdecimal`` vendored at ``Modules/_decimal/libmpdec``) -or they could be optionally pulled in during builds like Windows using dependencies from the `python/cpython-source-deps `_ repository. +Some are vendored into the source code of CPython itself (like ``mpdecimal`` +vendored at :cpy-file:`Modules/_decimal/libmpdec`) or they could be optionally pulled +in during builds like Windows using dependencies from the +`python/cpython-source-deps `_ +repository. -Whenever adding or updating a third-party dependency, an update will likely need to be done -to the SBOM in order to track the version and software identifiers. +Whenever adding or updating a third-party dependency, an update will likely +need to be done to the SBOM in order to track the version and software identifiers. Updating a dependency --------------------- -The SBOM for CPython's bundled dependencies is kept at ``Misc/sbom.spdx.json``. Instead of updating this document -directly, there is a tool for automatically updating the document at ``Tools/build/generate_sbom.py``. -The recommended workflow is: - -* Update the vendored code in-place. Take note of the new version, download URL, and checksum of the downloaded archive. -* Edit ``Misc/sbom.spdx.json`` to add the new ``versionInfo``, ``downloadLocation``, ``checksums``, and ``externalReferences`` for the corresponding ``package``. - Don't update any information in ``files`` and ``relationships`` as this will be generated automatically later. -* Run ``$ make regen-sbom`` or ``Tools/build/generate_sbom.py``. Ensure that this doesn't fail with validation errors. -* Run ``$ git diff Misc/sbom.spdx.json`` and check the diff matches the expected changes. -* Commit the changes to ``Misc/sbom.spdx.json`` along with the update to the dependency code. +The SBOM for CPython's bundled dependencies is kept at +:cpy-file:`Misc/sbom.spdx.json`. Instead of updating this document directly, +there is a tool for automatically updating the document at +:cpy-file:`Tools/build/generate_sbom.py`. The recommended workflow is: + +1. Update the vendored code in-place. Take note of the new version, download + URL, and checksum of the downloaded archive. +2. Edit :cpy-file:`Misc/sbom.spdx.json` to add the new ``versionInfo``, + ``downloadLocation``, ``checksums``, and ``externalReferences`` for the + corresponding ``package``. + Don't update any information in ``files`` and ``relationships`` as this will + be generated automatically later. +3. Run ``$ make regen-sbom`` or :cpy-file:`Tools/build/generate_sbom.py`. + Ensure that this doesn't fail with validation errors. +4. Run ``$ git diff Misc/sbom.spdx.json`` and check the diff matches the + expected changes. +5. Commit the changes to :cpy-file:`Misc/sbom.spdx.json` along with the + update to the dependency code. Adding or removing a dependency ------------------------------- When adding a dependency it's important to have the following information: -* Name, version, and download URL of the project -* License of the project as an `SPDX License Expression `_ -* Software identifiers that match values in vulnerability databases (`CPE `_ and `Package URLs `_ or "PURLs") +1. Name, version, and download URL of the project +2. License of the project as an `SPDX License Expression `_ +3. Software identifiers that match values in vulnerability databases + (`CPE `_ and + `Package URLs `_ + or "PURLs") -Add the information into a new entry in ``packages`` in the file ``Misc/sbom.spdx.json``. -If a new license ID is to be used, add the license expression to ``ALLOWED_LICENSE_EXPRESSIONS`` -in the SBOM generation tool. +Add the information into a new entry in ``packages`` in the file +:cpy-file:`Misc/sbom.spdx.json`. If a new license ID is to be used, add the +license expression to ``ALLOWED_LICENSE_EXPRESSIONS`` in the SBOM generation tool. -When removing a dependency, remove the entry from the ``Misc/sbom.spdx.json`` under the ``packages`` field -and then remove the corresponding entry in ``Tools/build/generate_sbom.py`` and then run ``make regen-sbom``. +When removing a dependency, remove the entry from the :cpy-file:`Misc/sbom.spdx.json` +under the ``packages`` field and then remove the corresponding entry in +:cpy-file:`Tools/build/generate_sbom.py` and then run ``make regen-sbom``. From e1e0fda126938760479823346141bef3b79acf5d Mon Sep 17 00:00:00 2001 From: Seth Michael Larson Date: Tue, 19 Dec 2023 11:06:07 -0600 Subject: [PATCH 3/6] Flatten embedded lists, add more guidance for SBOM fields --- developer-workflow/sbom.rst | 95 +++++++++++++++++++++++++++---------- 1 file changed, 71 insertions(+), 24 deletions(-) diff --git a/developer-workflow/sbom.rst b/developer-workflow/sbom.rst index 9d0db54d5..cfd68fca4 100644 --- a/developer-workflow/sbom.rst +++ b/developer-workflow/sbom.rst @@ -23,40 +23,87 @@ Updating a dependency --------------------- The SBOM for CPython's bundled dependencies is kept at -:cpy-file:`Misc/sbom.spdx.json`. Instead of updating this document directly, -there is a tool for automatically updating the document at -:cpy-file:`Tools/build/generate_sbom.py`. The recommended workflow is: +:cpy-file:`Misc/sbom.spdx.json`. When updating a dependency to a new version +you'll need to edit the version and other metadata about this dependency in +the SBOM. -1. Update the vendored code in-place. Take note of the new version, download +The recommended workflow is: + +1. Download the new dependency as an archive. Take note of the new version, download URL, and checksum of the downloaded archive. -2. Edit :cpy-file:`Misc/sbom.spdx.json` to add the new ``versionInfo``, +2. Update the vendored code in the CPython source tree. +3. Edit :cpy-file:`Misc/sbom.spdx.json` to add the new ``versionInfo``, ``downloadLocation``, ``checksums``, and ``externalReferences`` for the - corresponding ``package``. + corresponding ``package``. For most of these updates all that's needed is to + update the embedded version within URLs and project identifiers. Don't update any information in ``files`` and ``relationships`` as this will - be generated automatically later. -3. Run ``$ make regen-sbom`` or :cpy-file:`Tools/build/generate_sbom.py`. + be generated automatically by the SBOM tool. +4. Run ``$ make regen-sbom`` or :cpy-file:`Tools/build/generate_sbom.py`. Ensure that this doesn't fail with validation errors. -4. Run ``$ git diff Misc/sbom.spdx.json`` and check the diff matches the +5. Run ``$ git diff Misc/sbom.spdx.json`` and check the diff matches the expected changes. -5. Commit the changes to :cpy-file:`Misc/sbom.spdx.json` along with the +6. Commit the changes to :cpy-file:`Misc/sbom.spdx.json` along with the update to the dependency code. -Adding or removing a dependency -------------------------------- +Adding a new dependency +----------------------- When adding a dependency it's important to have the following information: -1. Name, version, and download URL of the project -2. License of the project as an `SPDX License Expression `_ -3. Software identifiers that match values in vulnerability databases - (`CPE `_ and - `Package URLs `_ - or "PURLs") +* Name, version, and download URL of the project +* License of the project as an `SPDX License Expression `_ +* Software identifiers that match values in vulnerability databases + (`CPE `_ and + `Package URLs `_ + or "PURLs") +* Path(s) to include and exclude in the CPython source tree corresponding to this dependency. + +After gathering this information: + +1. Add the information into a new entry in ``packages`` in the file + :cpy-file:`Misc/sbom.spdx.json`. Don't worry about formatting, the tool will + auto-format your manually written JSON. The fields to fill out include: + + * ``name`` for the project name. + * ``SPDXID`` which will be ``"SPDXRef-PACKAGE-{name}"`` + * ``licenseConcluded`` for the SPDX license identifier of the project license. + * ``versionInfo`` for the version of the project. + * ``downloadLocation`` should be an HTTPS URL for the project download as an archive. + * ``checksums[0].checksumValue`` and ``.algorithm`` will be the SHA-256 + checksum of the downloaded archive. + * ``originator`` for the original author information, prefix with either an + ``Organization:`` or ``Person:`` depending on the author/maintenance situation. + * ``primaryPackagePurpose`` will likely be ``"SOURCE"``. + * ``externalReferences`` is a list of one or more project identifiers, + either CPE or Package URL. The value for ``referenceLocator`` must include + the value in ``versionInfo`` to ensure the identifier + corresponds to the correct release of the software. You can read more about + external references in the `SPDX SBOM specification`_. +2. If a new license ID is to be used, add the license expression to + ``ALLOWED_LICENSE_EXPRESSIONS`` in the :cpy-file:`Tools/build/generate_sbom.py`. +3. Add the paths to include and exclude into a ``PackageFiles`` instance + with a key corresponding to the SBOM ID for the package (``SPDXID`` without the + ``SPDXRef-PACKAGE-*`` prefix) in :cpy-file:`Tools/build/generate_sbom.py`. +4. Run the tool with ``make regen-sbom`` or ``python Tools/build/generate_sbom.py``. + Ensure that the tool doesn't fail with any validation errors. +5. Compare the changes to :cpy-file:`Misc/sbom.spdx.json` with ``$ git diff``, check + that all information appears correct. +6. Commit the changes to :cpy-file:`Misc/sbom.spdx.json` and + :cpy-file:`Tools/build/generate_sbom.py`. + +.. _SPDX SBOM specification: https://spdx.github.io/spdx-spec/v2-draft/external-repository-identifiers/ + +Removing a dependency +--------------------- -Add the information into a new entry in ``packages`` in the file -:cpy-file:`Misc/sbom.spdx.json`. If a new license ID is to be used, add the -license expression to ``ALLOWED_LICENSE_EXPRESSIONS`` in the SBOM generation tool. +When removing a dependency: -When removing a dependency, remove the entry from the :cpy-file:`Misc/sbom.spdx.json` -under the ``packages`` field and then remove the corresponding entry in -:cpy-file:`Tools/build/generate_sbom.py` and then run ``make regen-sbom``. +1. Remove the entry from the :cpy-file:`Misc/sbom.spdx.json` + under the ``packages`` field. +2. Remove the corresponding ``PackageFiles`` entry in :cpy-file:`Tools/build/generate_sbom.py` +3. Run the tool with ``make regen-sbom`` or ``python Tools/build/generate_sbom.py``. + Ensure that the tool doesn't fail with any validation errors. +4. Compare the changes to :cpy-file:`Misc/sbom.spdx.json` with ``$ git diff``, check + that correct package is removed from the SBOM. +5. Commit the changes to :cpy-file:`Misc/sbom.spdx.json` and + :cpy-file:`Tools/build/generate_sbom.py`. From 3361cb2b19105c9b0d6b314deb8d1a9429b6ddeb Mon Sep 17 00:00:00 2001 From: Seth Michael Larson Date: Tue, 19 Dec 2023 11:28:38 -0600 Subject: [PATCH 4/6] Apply suggestions from code review Co-authored-by: Hugo van Kemenade --- developer-workflow/sbom.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/developer-workflow/sbom.rst b/developer-workflow/sbom.rst index cfd68fca4..639a843fe 100644 --- a/developer-workflow/sbom.rst +++ b/developer-workflow/sbom.rst @@ -38,9 +38,9 @@ The recommended workflow is: update the embedded version within URLs and project identifiers. Don't update any information in ``files`` and ``relationships`` as this will be generated automatically by the SBOM tool. -4. Run ``$ make regen-sbom`` or :cpy-file:`Tools/build/generate_sbom.py`. +4. Run ``make regen-sbom`` or `` python Tools/build/generate_sbom.py``. Ensure that this doesn't fail with validation errors. -5. Run ``$ git diff Misc/sbom.spdx.json`` and check the diff matches the +5. Run ``git diff Misc/sbom.spdx.json`` and check the diff matches the expected changes. 6. Commit the changes to :cpy-file:`Misc/sbom.spdx.json` along with the update to the dependency code. @@ -56,7 +56,7 @@ When adding a dependency it's important to have the following information: (`CPE `_ and `Package URLs `_ or "PURLs") -* Path(s) to include and exclude in the CPython source tree corresponding to this dependency. +* Paths to include and exclude in the CPython source tree corresponding to this dependency After gathering this information: @@ -86,7 +86,7 @@ After gathering this information: ``SPDXRef-PACKAGE-*`` prefix) in :cpy-file:`Tools/build/generate_sbom.py`. 4. Run the tool with ``make regen-sbom`` or ``python Tools/build/generate_sbom.py``. Ensure that the tool doesn't fail with any validation errors. -5. Compare the changes to :cpy-file:`Misc/sbom.spdx.json` with ``$ git diff``, check +5. Compare the changes to :cpy-file:`Misc/sbom.spdx.json` with ``git diff``, check that all information appears correct. 6. Commit the changes to :cpy-file:`Misc/sbom.spdx.json` and :cpy-file:`Tools/build/generate_sbom.py`. @@ -103,7 +103,7 @@ When removing a dependency: 2. Remove the corresponding ``PackageFiles`` entry in :cpy-file:`Tools/build/generate_sbom.py` 3. Run the tool with ``make regen-sbom`` or ``python Tools/build/generate_sbom.py``. Ensure that the tool doesn't fail with any validation errors. -4. Compare the changes to :cpy-file:`Misc/sbom.spdx.json` with ``$ git diff``, check +4. Compare the changes to :cpy-file:`Misc/sbom.spdx.json` with ``git diff``, check that correct package is removed from the SBOM. 5. Commit the changes to :cpy-file:`Misc/sbom.spdx.json` and :cpy-file:`Tools/build/generate_sbom.py`. From af05a5a9b575fb1aed0f1885efe4991ea524f7a8 Mon Sep 17 00:00:00 2001 From: Seth Michael Larson Date: Tue, 19 Dec 2023 11:30:22 -0600 Subject: [PATCH 5/6] Fix inline code example Co-authored-by: Hugo van Kemenade --- developer-workflow/sbom.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer-workflow/sbom.rst b/developer-workflow/sbom.rst index 639a843fe..290dd71af 100644 --- a/developer-workflow/sbom.rst +++ b/developer-workflow/sbom.rst @@ -38,7 +38,7 @@ The recommended workflow is: update the embedded version within URLs and project identifiers. Don't update any information in ``files`` and ``relationships`` as this will be generated automatically by the SBOM tool. -4. Run ``make regen-sbom`` or `` python Tools/build/generate_sbom.py``. +4. Run ``make regen-sbom`` or ``python Tools/build/generate_sbom.py``. Ensure that this doesn't fail with validation errors. 5. Run ``git diff Misc/sbom.spdx.json`` and check the diff matches the expected changes. From 1090a1afa7d4bb9bbf1990192b3e1918d2f92e1a Mon Sep 17 00:00:00 2001 From: Seth Michael Larson Date: Tue, 19 Dec 2023 11:49:49 -0600 Subject: [PATCH 6/6] Add period to end of list item Co-authored-by: Ezio Melotti --- developer-workflow/sbom.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer-workflow/sbom.rst b/developer-workflow/sbom.rst index 290dd71af..756c17570 100644 --- a/developer-workflow/sbom.rst +++ b/developer-workflow/sbom.rst @@ -65,7 +65,7 @@ After gathering this information: auto-format your manually written JSON. The fields to fill out include: * ``name`` for the project name. - * ``SPDXID`` which will be ``"SPDXRef-PACKAGE-{name}"`` + * ``SPDXID`` which will be ``"SPDXRef-PACKAGE-{name}"``. * ``licenseConcluded`` for the SPDX license identifier of the project license. * ``versionInfo`` for the version of the project. * ``downloadLocation`` should be an HTTPS URL for the project download as an archive.