Shared success in building a safer open source community

Today we joined the Open Source Security Foundation (OpenSSF), Linux Foundation and industry leaders for a meeting to continue progressing the open source software security initiatives discussed during January’s White House Summit on Open Source Security. During this meeting, Google announced the creation of its new “Open Source Maintenance Crew” — a dedicated staff of Google engineers who will work closely with upstream maintainers on improving the security of critical open source projects. In addition to this initiative, we contributed ideas and participated in discussions on improving the security and trustworthiness of open source software.

Amid all this momentum and progress, it is important to take stock on how far we’ve come as a community over the past year and a half. In this post we will provide an update on some major milestones and projects that have launched and look towards the future and the work that still needs to be done.

Know, Prevent, Fix

A little over a year ago we published Know, Prevent, Fix, which laid out a framework for how the software industry could address vulnerabilities in open source software. At the time, there was a growing interest in the topic and the hope was to generate momentum in the cause of advancing and improving software supply-chain security.

The landscape has changed greatly since then:

• Prominent attacks and vulnerabilities in critical open source libraries such as Log4j and Codecov made headline news, bringing a new level of awareness to the issue and unifying the industry to address the problem.
• The US government formalized the push for higher security standards in the May 2021 Executive Order on Cybersecurity. The release of the Secure Software Development Framework, a set of guidelines for national security standards on software development, sparked an industry-wide discussion about how to implement them.
• Last August, technology leaders including Google, Apple, IBM, Microsoft, and Amazon invested in improving cybersecurity — and Google alone pledged $10 billion over the next five years to strengthen cybersecurity, including $100 million to support third-party foundations, like OpenSSF, that manage open source security priorities and help fix vulnerabilities.

In light of these changes, the Know, Prevent, Fix framework proved prescient: beyond just the increased discussion about open source security, we’re witnessing real progress in the industry to act on those discussions. In particular, the OpenSSF has become a community town hall for driving security engineering efforts, discussions, and industry-wide collaboration.

These successes have also surfaced new challenges, though, and we believe the next step is to increase accessibility. Security tools should be more easily adopted into common developer workflows, more integrated across the ecosystem, and simpler to connect into projects. Underlying all of this is a need to streamline the process of matching projects with available funds and resources to enable security improvements.

This follow-up blog post discusses Google’s efforts in collaboration with the open source community to make progress on security goals in the past year, the lessons learned and how the industry can build on this momentum.

Know

Our goals for “Know” were to capture more precise data about vulnerabilities, establish a standard schema to track vulnerabilities across databases, and create tooling for better tracking of dependencies.

Over the past year the community’s investment in Open Source Vulnerabilities (OSV) has resulted in a new vulnerability format. The format was developed and adopted by several open source ecosystems (Python, Rust, Go), as well as vulnerability databases such as GitHub’s Security Advisories (GHSA) and the Global Security Database. Google also worked closely with MITRE on the new CVE 5.0 JSON schema to simplify future interoperability. OSV.dev also supports a searchable vulnerability database that, thanks to the standardized format, aggregates vulnerabilities from all other databases into one easily searched location.

During the Log4j vulnerability response, the Google-supported Open Source Insights project helped the community understand the impact of the vulnerability. This project analyzes open source packages and provides detailed graphs of dependencies and their properties. With this information, developers can understand how their software is put together and the consequences to changes in their dependencies—which, as Log4j showed, can be severe when affected dependencies are many layers deep in the dependency graph. Today, we’re also making the data powering Open Source Insights available as a public Google Cloud Dataset.

The OSV project showed that connecting a CVE to the vulnerability patch development workflow can be difficult without precise vulnerability metadata. It will take cooperation across disparate development communities to reap the full benefits of the progress, but with collaboration OSV can scale quickly across language and project ecosystems.

We believe the next major goal is to lower the barrier of entry for users. Integrating with developer tools and processes will bring high quality information to where it is most useful. For instance, OSV findings can be everywhere from code editors (e.g., when deciding whether to include a library) to deployment (e.g., stopping vulnerable workloads from deploying).

Prevent

“Prevent” was conceived to help users understand the risks of new dependencies so they can make informed decisions about the packages and components they consume.

We’ve seen strong community involvement in the prevention of vulnerabilities, particularly in the Security Scorecards project. Scorecards evaluates a project’s adherence to security best practice and assigns scores that developers can consult before consuming a dependency. Users can choose to avoid projects that, for example, don’t use branch protection settings or employ dangerous workflows (which make projects vulnerable to malicious commits), and gravitate to projects that follow strong security practices like signing their releases and using fuzzing. Thanks to contributions from Cisco, Datto and several other open source contributors, there are now regular Scorecard scans of 1 million projects, and Scorecards has developed from a command line tool into an automated GitHub Actions that runs after any change to GitHub project. More organizations are adopting Scorecards, and the Scorecard GitHub Action has been installed on over 1000 projects, with continued growth. With increased adoption, overall security will improve across entire ecosystems.

Additionally, Sigstore is helping prevent attacks by creating new tools for signing, verifying and protecting software. Recently, Kubernetes announced that it is using sigstore to sign its releases, showing that artifact signing at a large scale is now within reach. As adoption expands, we can expect stronger links between published source code and the binaries that run it.

Community collaborators like Citi, Chainguard, DataDog, VMWare and others have actively contributed to the OpenSSF’s SLSA framework. This project is based on Google’s internal Binary Authorization for Borg (BAB), which for more than a decade has been mitigating the risk of source and production attacks at Google. SLSA lays out an actionable path for organizations to increase their overall software supply-chain security by providing step-by-step guidelines and practical goals for protecting source and build system integrity. The SLSA framework addresses a limitation of Software Bills of Materials (SBOMs), which on their own do not provide sufficient information about integrity and provenance. An SBOM created using SLSA provenance and metadata is more complete and addresses both source code and build threat vectors. Using SLSA may also help users implement Secure Software Development Framework (SSDF) requirements.

Continued improvements to the OSS-Fuzz service for open source developers have helped get over 2300 vulnerabilities fixed across 500+ projects in the past year. Google has also been heavily investing in expanding the scope of fuzzing through adding support for new languages such as Java and Swift and developing bug detectors to find issues like Log4shell.

Through the Linux Kernel Self-Protection Project, Google has been providing a steady stream of changes to overhaul internal kernel APIs so that the compiler can detect and stop buffer overflows in fragile areas that have seen repeated vulnerabilities. For everyone in the ecosystem staying current on Linux kernel versions, this removes a large class of flaws that could lead to security exploits.

Looking ahead, this area’s rapid growth highlights the community’s concern about integrity in software supply chains. Users are searching for solutions that they can trust across ecosystems, such as provenance metadata that connects deployed software to its original source code. Additionally, we expect increased scrutiny of development processes to ensure that software is built in the most secure way possible.

The next goals for open source software security should involve broad adoption of best practices and scalability. Increasing the use of these tools will multiply the positive effects as more projects become secured, but adoption needs to happen in a scalable way across ecosystems (e.g., via the OpenSSF Securing Package Repositories Working Group focused on improving security in centralized package managers). Education will be a driving force to speed the shift from project-by-project adoption to broadscale ecosystem conversion: greater awareness will bring greater momentum for change.

Fix

“Fix” was conceived to help users understand their options to remove vulnerabilities, enable notifications that help speed repairs, and fix widely used versions of affected software, not just the most recent versions.

Google supported open source innovation, security, collaboration, and sustainability through our programs and services by giving $15 million to open source last year. This includes $7.5 million to targeted security efforts in areas such as supply chain security, fuzzing, kernel security and critical infrastructure security. For example, $2.5 million of the security funding went to the Alpha-Omega project, which made its first grant to the OpenJS foundation to strengthen its security team and support vulnerability remediation.

Other security investments include $1 million to SOS Rewards, and $300,000 to the Internet Security Research Group to improve memory safety by incorporating Rust into the Linux kernel. The remaining funding supports security audits, package signing, fuzzing, reproducible builds, infrastructure security and security research.

Beyond financial investments, Google employees contribute their hours, effort, and code to tens of thousands of open source repositories each year. One issue frequently cited by open source maintainers is limited time. Since under-maintained, critical open source components are a security risk, Google is starting a new Open Source Maintenance Crew, a dedicated staff of Google engineers who will work closely with upstream maintainers on improving the security of critical open source projects. We hope that other enterprises that rely on open source will invest in similar efforts to help accelerate security improvements in the open source ecosystem.

Up Next

The amount of progress in the past year is very encouraging: we as an industry have come together to discuss, fund, and make headway on many of the difficult problems that affect us all. The solutions are not just being talked about, but also built, refined, and applied. Now we need to magnify this progress by integrating these solutions with tooling and language ecosystems: every open source developer should have effortless access to end-to-end security by default.

Google is committed to continuing our work with the OpenSSF to achieve these goals. To learn more about the OpenSSF foundation and join its efforts, check it out here.