Compute summary for all detected packages.

Organization: AboutCode

Project: Scancode Toolkit

Mentee: Swastik Sharma (swastkk)

Mentors: Philippe Ombredanne, Ayan Sinha Mahapatra, Avishrant Sharma, Jonathan Yang, Jay Kumar

Overview

Previously, we computed the summary at the codebase level, which included elements like the license_clarity_score, declared_holder, other_license_expressions, and more. This project aims to improve scanning accuracy by computing summaries and license clarity scores for each package and its files, rather than for the entire scan. This involves enhancing package models and ensuring accurate attribute collection across all package ecosystems.

Implementation

• Added a new command-line option called --package-summary:

  • Provides a package-level summary within a single codebase.

  • Involves the license_clarity_score calculation.

  • Populates package attributes like copyright, holder, other_license_expression, notice_text.

• The --package-summary option must be used with:

  • --classify: Helps ScanCode further classify scanned files/directories into categories like legal, readme, top-level, manifest.

  • --package or -p: Detects various package manifests, lockfiles, and package-like data, assembles codebase-level packages and dependencies, and tags files as part of the packages.

• Benefits of the change:

  • Allows users to obtain a more refined summary for each individual package in a codebase.

  • Improves package assembly for various package ecosystems like npm, python-whl, rust, rubygems, etc. Since the package-level summary heavily depends on the package assembly, there were several scenarios where key files for top-level packages were not properly tagged. To address this, a method called get_top_level_resources was implemented. This method retrieves the resources for top-level packages, which helps in correctly tagging the key files.

• Testing:

  • All changes are tested through multiple full scan tests.

  • Validated both correct behavior and error handling.

Linked Pull Requests

Sr. no	Link	Status
1	https://github.com/aboutcode-org/scancode-toolkit/pull/3792	Open

Related Issues

Sr. no	Name	Link
1	Improve Package models to have license_clarity_score	#3817
2	Post Scan option –package-summary	#3802
3	Look in package-ecosystem specific key-files for referenced licenses	#3707
4	Compute summary and clarity for EACH package in a codebase	#3287
5	Provide data values in scan results to correspond with license_clarity_score elements	#1395
6	Populate package instance attributes from it’s files	#3862
7	Improve Ruby Package Ecosystem/Datafile Handler to tag key_files properly	#3881
8	Rust Members files are not detected properly	#3895
9	Add Tests for Package Level Summary computation	#3889

Post GSoC

I would like to merge this PR into Scancode Toolkit, hopefully allowing users to leverage this feature to expand their package/codebase scanning capabilities.

Links

• Project Idea

• Official GSoC project page

• GSoC Proposal

• Project Board

• Reference Issue

Acknowledgements

I would like to thank my mentors:

• Philippe Ombredanne

• Ayan Sinha Mahapatra

• Avishrant Sharma

• Jay Kumar

• Jonathan Yang

Weekly Status calls were greatly helpful and those special 1:1 calls with Ayan Sinha Mahapatra and Philippe Ombredanne were so amazing. Thank you for your time and your patience!