Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: update licenses to including custom content when SPDX expressions are unable to be determined #3366

Merged
merged 39 commits into from
Jan 28, 2025
Merged

feat: update licenses to including custom content when SPDX expressions are unable to be determined #3366

merged 39 commits into from
Jan 28, 2025

Conversation

HeyeOpenSource
Copy link
Contributor

@HeyeOpenSource HeyeOpenSource commented Oct 22, 2024 •
edited
Loading

Description

Implements support for custom licenses.
Should probably be reviewed by @wagoodman and / or @spiffcs.

Type of change

  • New feature (non-breaking change which adds functionality)

Checklist:

  • I have added unit tests that cover changed behavior
  • I have tested my code in common scenarios and confirmed there are no regressions
  • I have added comments to my code, particularly in hard-to-understand sections
  • I have run make test locally without receiving any failures
  • 'Validations' workflow returns no errors

HeyeOpenSource and others added 11 commits October 17, 2024 10:33
@HeyeOpenSource
- Implement support for custom licenses.
c3d4306 
Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource
- Make sure custom licenses are also exported for the CycloneDX format.
19611de 
Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource
- Export license URL for custom licenses in the CycloneDX format, if …
6ebc5fa 
…available.

Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource
- Retain custom license contents for golang licenses.
1ba2393 
Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource
- Harmonize custom license line endings to UNIX style before registra…
64a433e 
…tion in order to avoid incompatibilities (and multiple instances of the same license).

Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource
Merge branch 'main' into Custom_Licenses
b6b8ca5
@HeyeOpenSource
- Fix for static analysis.
48254a2 
Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource
Merge branch 'anchore:main' into Custom_Licenses
243ecb4
@HeyeOpenSource
Merge commit '14355aac2157482fc6d4dc34a608f779912e6d2b' into Custom_L…
a67d7b3 
…icenses
@HeyeOpenSource
Merge branch 'main' into Custom_Licenses
9ae8aad 
Signed-off-by: HeyeOpenSource <[email protected]>

# Conflicts:
#	internal/licenses/parser.go
@HeyeOpenSource
- Ignore custom license contents in hash.
cbb9ddd 
Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource HeyeOpenSource changed the title Allow for custom licenses (including content) Allow for custom non SPDX licenses (including content) Oct 23, 2024
@github-actions github-actions bot added json-schema Changes the json schema and removed json-schema Changes the json schema labels Oct 23, 2024
@HeyeOpenSource
Revert "- Adapt JSON schema."
52e1fab 
This reverts commit beda1b6.

Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource
- Add schema-16.0.19.json again.
bb5a0eb 
Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource
Revert "- Add schema-16.0.19.json again."
ef8875f 
This reverts commit 9b90378.

Signed-off-by: HeyeOpenSource <[email protected]>
@HeyeOpenSource
- Added new schema as created by 'make generate-json-schema'.
3e546de 
Signed-off-by: HeyeOpenSource <[email protected]>
wagoodman
wagoodman reviewed Oct 23, 2024
View reviewed changes
syft/pkg/license.go
@@ -29,6 +29,7 @@ type License struct {
Type license.Type
URLs []string `hash:"ignore"`
Locations file.LocationSet `hash:"ignore"`
Contents string `hash:"ignore"` // The optional binary contents of the license file
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we add support for this we should make this configurable. It could be a simple on/off (collect or do not collect contents) but there is some argument for more options here. Specifically, when we can get the ID from contents and we have a high degree of confidence that it matches then including the contents is redundant / not necessary (one of multiple possible middle of the road options).

I think the default for this option should either be "off" or the middle of the road option.

Copy link
Contributor Author

@HeyeOpenSource HeyeOpenSource Oct 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My implementation only ever fills the Contents field if a non-empty file identified as potential license file (by name) exists and no SPDX-expression can be determined for it with the required confidence.
Otherwise the status quo implementation is retained.

Hence I would assume that the option as you describe it is currently "middle of the road" by default. 😉

Copy link
Contributor Author

@HeyeOpenSource HeyeOpenSource Oct 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would advocate for a later implementation of the configurability, as my current implementation mitigates the fact, that packages with an identified (but non-SPDX-expression-compatible) license file will be erroneously handled as unlicensed. (cf. #3412)

@HeyeOpenSource
Copy link
Contributor Author

The Validations workflow finally throws no more errors at me. 👍
cf. Validations #22

@HeyeOpenSource HeyeOpenSource mentioned this pull request Nov 2, 2024
Closed
@HeyeOpenSource HeyeOpenSource changed the title Allow for custom non SPDX licenses (including content) Allow for custom non SPDX-expression licenses (including content) Nov 3, 2024
@spiffcs spiffcs assigned spiffcs and unassigned wagoodman Dec 12, 2024
HeyeOpenSource and others added 3 commits January 2, 2025 02:23
@HeyeOpenSource
Merge branch 'main' into Custom_Licenses
a487ce3 
# Conflicts:
#	syft/format/internal/cyclonedxutil/helpers/licenses.go
@spiffcs
Merge branch 'main' into Custom_Licenses
77db223 
* main: (54 commits)
  chore(deps): update CPE dictionary index (anchore#3620)
  chore(deps): bump github.com/bmatcuk/doublestar/v4 from 4.8.0 to 4.8.1 (anchore#3621)
  chore(deps): bump github/codeql-action from 3.28.4 to 3.28.5 (anchore#3622)
  chore(deps): bump github/codeql-action from 3.28.3 to 3.28.4 (anchore#3618)
  chore(deps): bump anchore/sbom-action from 0.17.9 to 0.18.0 (anchore#3619)
  chore(deps): update tools to latest versions (anchore#3607)
  chore(deps): bump github/codeql-action from 3.28.2 to 3.28.3 (anchore#3608)
  chore(deps): bump github.com/go-git/go-git/v5 from 5.13.1 to 5.13.2 (anchore#3609)
  chore(deps): bump github.com/docker/docker (anchore#3610)
  chore(deps): bump actions/setup-go in /.github/actions/bootstrap (anchore#3612)
  chore(deps): bump actions/cache in /.github/actions/bootstrap (anchore#3613)
  chore(ci): fix composite GitHub action path in dependabot config (anchore#3611)
  chore(deps): update tools to latest versions (anchore#3602)
  chore(deps): bump github/codeql-action from 3.28.1 to 3.28.2 (anchore#3604)
  chore(deps): bump github.com/hashicorp/hcl/v2 from 2.22.0 to 2.23.0 (anchore#3605)
  chore(deps): bump github.com/aquasecurity/go-pep440-version (anchore#3606)
  chore: bump stereoscope to v0.0.13 (anchore#3601)
  feat(cataloger): add a terraform provider cataloger (anchore#3378)
  chore(deps): update tools to latest versions (anchore#3597)
  chore(deps): update CPE dictionary index (anchore#3599)
  ...

Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
chore: bump schema to 16.0.21
4af799e 
Signed-off-by: Christopher Phillips <[email protected]>
@github-actions GitHub Actions
Copy link

Warning

Detected modification or removal of existing json schemas:

  • schema/json/schema-16.0.19.json

@spiffcs
Copy link
Contributor

spiffcs commented Jan 28, 2025

rebased and pull in upstream main - I'll work through the CI, add my review, and add @wagoodman's requested functionality about contents configuration.

Thanks for the contribution @HeyeOpenSource! Sorry this one has taken a little while.

@spiffcs
Copy link
Contributor

spiffcs commented Jan 28, 2025 •
edited
Loading

PR looks great! Adding some new tests around the format model changes to double check behavior and will merge 👍

After reading the behavior again I agree with @HeyeOpenSource that we don't need to make contents configurable as they are only exposed when a valid license ID can not be determined by the scanner.

I'll make sure to update the documentation in syft to express this mutually exclusive behavior.

Apologies for breaking the tests using TestingOnlyScanner I was under the impression it was a local construct and not consumed on by other tests. I've added a new constructor to the license package to account for this.

@spiffcs spiffcs changed the title Allow for custom non SPDX-expression licenses (including content) feat: update licenses to including custom content when SPDX expressions are unable to be determined Jan 28, 2025
@spiffcs spiffcs mentioned this pull request Jan 28, 2025
Open
5 tasks
@spiffcs spiffcs enabled auto-merge (squash) January 28, 2025 17:55
@spiffcs spiffcs merged commit f7e767f into anchore:main Jan 28, 2025
12 checks passed
This was referenced Jan 28, 2025
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@spiffcs spiffcs spiffcs approved these changes

@wagoodman wagoodman Awaiting requested review from wagoodman

Assignees

@spiffcs spiffcs

Labels
json-schema Changes the json schema
Projects
OSS
Status: Done
Milestone
No milestone
3 participants
@HeyeOpenSource @spiffcs @wagoodman