SCB relies on many third-party scanners, some of which will perform major version upgrades occasionally (e.g., gitleaks, sslyze, ...). At the same time, we also make changes to the SCB codebase, which can also occasionally include major changes (updates to CRDs, ...).
At the moment, the policy for what consitutes a major release (SCB v2.X => v3.X) vs. a minor release (v3.5 => 3.6) vs. a patch release (v3.6.0 => 3.6.1) is unclear (wrt. included scanners) and inconsistently followed (wrt. changes to CRDs, for example). This issue tracks our process for agreeing on a new, explicit versioning scheme.
We discussed this issue at length today. From my recollection, this is what we came up with.
General Points
The purpose of (semantic) versioning is that users can see directly "how worried they have to be" when installing an update. A single look at the old and new version number should tell them what kind of changes to expect, and if updating will require manual actions to keep the system running. We have always tried to follow semantic versioning principles, but have in some cases not done this correctly, and never really defined how it interacts with the updates released by external scanners. Thus, we now explicitly define what these terms mean in the context of the secureCodeBox.
Major releases (v3.X => 4.0)
A major release should happen every time we change something that will cause failures on updates unless additional, manual actions are taken. This includes:
- Backwards-incompatible changes to scanners (e.g., requiring new parameters / changes to the scan YAML, as is the case for Gitleaks v8)
- Changes to CRDs (since they have to be installed manually)
- Breaking changes to the finding format of a scanner (i.e., changing the contents and meaning of an existing field, removing a field)
- ...
A major release does not mean that every single user of the system will have to make changes - it is enough if users using a specific scanner (gitleaks) will have to make changes, either to how they parameterize the scan or for working with the output format. The patchnotes should highlight exactly what part of the system contains a breaking change, and what actions are required to address it (either directly in the patch notes, or in a linked pull request / documentation page).
This policy implies that major releases will be a lot more common in the future, as especially the rule about changing CRDs was not followed in the last few months. However, it also implies that minor/patch releases should be installable with a lot more confidence as soon as this policy is in effect.
Minor releases (v3.1.X => 3.2.0)
Following the principles of semver, minor releases contain changes in functionality that are backwards-compatible. This includes:
- Updates to scanners, hooks or operators that add new functionality, but do not significantly alter the behavior of existing functionality
- Changes to the findings format that add new information without changing what is already there
- ...
A major release should, generally, be installable without reading the release notes (although we of course do not recommend this 😁), as it by definition does not require making manual changes after installation and does not significantly alter the functioning of whatever you have already configured.
Patch releases (v3.1.0 => 3.1.1)
Patch releases only contain bug fixes that do not add new functionality, only repair what is already there. Importantly, a patch release can alter the behavior of the system, including the contents of findings, as long as what was previously being returned was incorrect. For example, if a finding listed a URL as https:///google.com, changing it to https://google.com (without the extra, incorrect slash) would be considered a patch, not a minor or major change.
We are seeking input on this proposed policy, both from the rest of the team that wasn't present in the meeting today and from external contributors and users of the system like @EndPositive and @sanmai-NL. Please give your input in this issue. @Weltraumschaf @rseedorff, if I misremembered anything or you have more examples to add, please feel free.
Once we have agreed on a specific policy, the following changes will have to be made:
SCB relies on many third-party scanners, some of which will perform major version upgrades occasionally (e.g., gitleaks, sslyze, ...). At the same time, we also make changes to the SCB codebase, which can also occasionally include major changes (updates to CRDs, ...).
At the moment, the policy for what consitutes a major release (SCB v2.X => v3.X) vs. a minor release (v3.5 => 3.6) vs. a patch release (v3.6.0 => 3.6.1) is unclear (wrt. included scanners) and inconsistently followed (wrt. changes to CRDs, for example). This issue tracks our process for agreeing on a new, explicit versioning scheme.
We discussed this issue at length today. From my recollection, this is what we came up with.
General Points
The purpose of (semantic) versioning is that users can see directly "how worried they have to be" when installing an update. A single look at the old and new version number should tell them what kind of changes to expect, and if updating will require manual actions to keep the system running. We have always tried to follow semantic versioning principles, but have in some cases not done this correctly, and never really defined how it interacts with the updates released by external scanners. Thus, we now explicitly define what these terms mean in the context of the secureCodeBox.
Major releases (v3.X => 4.0)
A major release should happen every time we change something that will cause failures on updates unless additional, manual actions are taken. This includes:
A major release does not mean that every single user of the system will have to make changes - it is enough if users using a specific scanner (gitleaks) will have to make changes, either to how they parameterize the scan or for working with the output format. The patchnotes should highlight exactly what part of the system contains a breaking change, and what actions are required to address it (either directly in the patch notes, or in a linked pull request / documentation page).
This policy implies that major releases will be a lot more common in the future, as especially the rule about changing CRDs was not followed in the last few months. However, it also implies that minor/patch releases should be installable with a lot more confidence as soon as this policy is in effect.
Minor releases (v3.1.X => 3.2.0)
Following the principles of semver, minor releases contain changes in functionality that are backwards-compatible. This includes:
A major release should, generally, be installable without reading the release notes (although we of course do not recommend this 😁), as it by definition does not require making manual changes after installation and does not significantly alter the functioning of whatever you have already configured.
Patch releases (v3.1.0 => 3.1.1)
Patch releases only contain bug fixes that do not add new functionality, only repair what is already there. Importantly, a patch release can alter the behavior of the system, including the contents of findings, as long as what was previously being returned was incorrect. For example, if a finding listed a URL as
https:///google.com, changing it tohttps://google.com(without the extra, incorrect slash) would be considered a patch, not a minor or major change.We are seeking input on this proposed policy, both from the rest of the team that wasn't present in the meeting today and from external contributors and users of the system like @EndPositive and @sanmai-NL. Please give your input in this issue. @Weltraumschaf @rseedorff, if I misremembered anything or you have more examples to add, please feel free.
Once we have agreed on a specific policy, the following changes will have to be made: