-
-
Notifications
You must be signed in to change notification settings - Fork 168
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* WIP OSS patching blog post * small post edits * oss patching blog post drafted * update date and truncate * get license out of PR template and add to ignore file * explain more why people need to know to security patch * update intro wording * edits to include githubs tooling * more credit to google * nits * change a link Co-authored-by: breadchris <chris@lunasec.io>
- Loading branch information
1 parent
998c69d
commit a33566d
Showing
13 changed files
with
237 additions
and
7 deletions.
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,191 @@ | ||
--- | ||
title: "How to Discuss and Fix Vulnerabilities in Your Open Source Library" | ||
description: Using the Log4Shell disaster to explain how to properly discuss and patch vulnerabilities in Open Source software. | ||
slug: how-to-mitigate-open-source | ||
date: 2021-12-18 | ||
keywords: [open-source, vulnerability, patching, PR, pull-release] | ||
tags: [zero-day, security, data-security, data-breaches, guides] | ||
authors: | ||
- name: Forrest Allison | ||
title: Developer at LunaSec | ||
url: https://github.com/factoidforrest | ||
image_url: https://github.com/factoidforrest.png | ||
- name: Free Wortley | ||
title: CEO at LunaSec | ||
url: https://github.com/freeqaz | ||
image_url: https://github.com/freeqaz.png | ||
--- | ||
|
||
<!-- | ||
~ Copyright by LunaSec (owned by Refinery Labs, Inc) | ||
~ | ||
~ Licensed under the Creative Commons Attribution-ShareAlike 4.0 International | ||
~ (the "License"); you may not use this file except in compliance with the | ||
~ License. You may obtain a copy of the License at | ||
~ | ||
~ https://creativecommons.org/licenses/by-sa/4.0/legalcode | ||
~ | ||
~ See the License for the specific language governing permissions and | ||
~ limitations under the License. | ||
~ | ||
--> | ||
|
||
The worldwide scramble to fix the Log4Shell vulnerability has resulted in a number of paths to resolution without a single | ||
trusted, clear, and concise recommendation. Ever since our initial report on the Log4Shell vulnerability, we have made | ||
countless updates to our posts over the past week as more information is uncovered and new vulnerabilities are found. | ||
Ideally, the narrative would be much clearer for a library which is [so widely used](https://www.contrastsecurity.com/security-influencers/log4shell-by-the-numbers?utm_campaign=WD-Log4jOffer-Q4FY22&utm_content=191438199&utm_medium=social&utm_source=twitter&hss_channel=tw-453210489). | ||
|
||
We feel that the chaos which has followed the initial report of the Log4Shell vulnerability started with how the public patching | ||
of Log4j was handled on GitHub. This is a step-by-step guide for Open Source maintainers on how to handle the security | ||
patching process properly, using the Log4Shell incident as a retrospective. | ||
|
||
<!--truncate--> | ||
|
||
Maybe some are more careful than others, but something like Log4Shell could happen to many of us who work in the Open Source world. | ||
Putting aside vulnerability prevention for a moment, let's talk about what to do when you find out that you released something insecure. | ||
Staying calm and knowing what to do when the time comes could save a lot of damage down the road. | ||
|
||
### Things they don't teach you in school | ||
As a "normal" developer who works alongside security engineers, I hear them mention all kinds of best practices that I've never heard of. | ||
It turns out that how to properly deal with vulnerabilities in a project you use or contribute to, | ||
right down to what you say on GitHub, can be just as important as the patch itself. | ||
|
||
Log4j is a great case study because even though people had the best intentions, things went catastrophically wrong. At the | ||
center of the story we have a couple of developers donating their spare time, doing their best. This is absolutely not | ||
meant to be a hit piece against them or Open Source software, but instead a guide to help the next person handle things the right way. | ||
Together, let's learn how to do this properly. | ||
|
||
### Hackers are crawling GitHub looking for security patches | ||
|
||
Finding a vulnerability in a project and contributing a fix can be pretty exciting. Unfortunately, the fact | ||
that something is being patched can reveal to outsiders that a vulnerability must have existed in the first place. | ||
It's trivially easy to write a webcrawler that looks through GitHub for the words | ||
"vulnerability", "security", "injection", "exploit", etc... | ||
and find a package where the last version, if not the current one, has a vulnerability ready to be turned into an exploit. | ||
Those crawlers are up and running _right now_, reading your comments and looking for the next 0day. | ||
|
||
## It's about being discreet | ||
|
||
### Reporting the vulnerability | ||
|
||
How the vulnerability first gets reported is critical. In the case of Log4j, contrary to what you might think, | ||
it looks like everything went right. According to the Log4Shell [Wikipedia page](https://en.wikipedia.org/wiki/Log4Shell), the vulnerability was reported | ||
*privately* to the apache team way back on November 24th by Chen Zhaojun of | ||
Alibaba Cloud Security Team. | ||
|
||
If you are someone spotting a vulnerability for the first time, this is exactly what you should do. | ||
We can also see that Mitre site created the CVE, **in private**, two days later, november 26th. | ||
|
||
![Picture of CVE date](/img/cve-date.png) | ||
|
||
Up until this point, it was security professionals following a well documented practice called [Responsible Disclosure](https://en.wikipedia.org/wiki/Responsible_disclosure). | ||
Let's move on to what happened next: | ||
|
||
### Patching the vulnerability | ||
|
||
Often just the nature of the patch can alert those who know what to look for, even without | ||
anyone spelling out that it's a vulnerability. | ||
|
||
Let's look at the first [Log4j PR](https://github.com/apache/logging-log4j2/pull/608). | ||
|
||
![PR header](/img/log4j-pr-header.png) | ||
|
||
At first glance it doesn't look too bad, at least to an average dev. Unfortunately, bad actors probably noticed this PR | ||
almost right away. | ||
Let's look at the files committed: | ||
|
||
![exploit patch file name](/img/log4j-exploit-file.png) | ||
|
||
In the PR, there is literally a file named `JndiExploit.java`. Even without using the word exploit, the word JNDI could | ||
have been enough to attract bad actors. | ||
|
||
![person on github asking publicly if its a vuln](/img/security-question-log4j.png) | ||
|
||
Cat's out of the bag. It's almost certain this PR, made with the best intentions, made things far worse. | ||
|
||
In the end, the vulnerability went public before Log4j was ready to ship a fix, and they were left scrambling. | ||
Their first release of version `2.15.0`, which much of the community installed, [proved to still have issues](https://www.lunasec.io/docs/blog/log4j-zero-day-severity-of-cve-2021-45046-increased/). | ||
They had to release a second patch in `2.16.0` which we seriously hope is seeing widespread adoption. | ||
|
||
## How to do it right | ||
|
||
### Step 1: Mitigate in private | ||
Open source software is public, which means eventually the patch will need to be public. We want that to happen | ||
when we are as ready for it as possible, **not before**, so: | ||
|
||
#### Use GitHub's Security Advisories and Temporary Private Forks | ||
Developers will need a private space to work on a patch and get ready before that happens. | ||
This will give them space to make the change, discuss it, prepare a release, and get documentation ready. | ||
|
||
GitHub actually has a new | ||
built-in feature for this scenario, called a [Github Security Advisory](https://docs.github.com/en/code-security/security-advisories/creating-a-security-advisory). | ||
This tooling walks you through the process of creating a private space to talk about the issue. It can link to or create a CVE, notify your users when the time comes, | ||
and even helps you create a linked [Temporary Private Fork](https://docs.github.com/en/code-security/security-advisories/collaborating-in-a-temporary-private-fork-to-resolve-a-security-vulnerability) | ||
in which to work. You should absolutely use this feature and make a private fork. | ||
CI won't run in this fork which means you will need to be careful and test locally, but this is worth the trade-off for privacy. | ||
|
||
![github security advisory creation](/img/security-advisory-new-draft-security-advisory-button.png) | ||
|
||
As the maintainer, it's also your responsibility to request a CVE. GitHub is a CVE Numbering Authority, meaning you can create a | ||
CVE through them. [Request a CVE](https://docs.github.com/en/code-security/security-advisories/about-github-security-advisories#cve-identification-numbers) | ||
from the Security Advisory you created above. Being a part of the process helps make sure the CVE is accurate and not confusing | ||
and miscategorized like some of the log4j CVEs. | ||
|
||
Ideally, a security person will be brought into this private process to audit the change, if you can get one. Side note: We are considering | ||
founding a non-profit to give projects this help, and if that's something you'd like to see please [let us know](https://docs.google.com/forms/d/e/1FAIpQLSd1yaB-3ob4yNnHBDzmFvrnl0VyViT234kpdcvadX-9G2M3eQ/viewform?usp=sf_link). | ||
|
||
Once a release has been drafted, this is an excellent time to bring in some of your biggest users to try it out. This has | ||
the benefit of testing your patch, getting more eyes on the problem, and making sure those consumers have their | ||
own patched releases ready to go. Let them know they should do this work in private. | ||
|
||
In the case of Apache, it would have been better if the biggest users of log4j (like Apache Struts) were alerted privately. They could have | ||
had their own privately prepared releases ready to go public when log4shell did. | ||
|
||
In summary, doing it in private gives space and time to go through and fix things properly. Use GitHub's built-in tool to help | ||
you manage the process. | ||
|
||
### Step 2: Release Simultaneously | ||
Now it's time to rip the bandage off. Merge all changes, releases, and documentation at the same time. Publish the GitHub | ||
Security Advisory you created above. Coordinate with any consumers | ||
that took your change early. This process is called **coordinated disclosure**. | ||
|
||
Hopefully the documentation and Security Advisory contain all the information needed by users to patch the change, right away. | ||
Generally, this is just explaining the vulnerability and pointing to the new release. | ||
|
||
Making noise and getting the word out about the vulnerability is going to make for more people updating. | ||
In the case of larger projects and more serious vulnerabilities, it might be a good idea to contact the press directly. | ||
This can help you keep a handle on the reporting and let you control the narrative around how to fix it. | ||
|
||
## Educate your contributors | ||
The above process is something that your contributors need to know about ahead of time. Personally, I would recommend a short warning | ||
about the proper way to patch security problems in your Issue and Pull Request templates. | ||
|
||
Ours looks like: | ||
|
||
**STOP**: Is this a **security vulnerability**? If so, follow Responsible Disclosure and email us at email@email.com instead of opening an issue. | ||
|
||
It's a good idea to say something similar in your README and to create a `SECURITY.md` file in the root of your project with disclosure instructions. | ||
Here is React's [`SECURITY.md`](https://github.com/facebook/react/security/policy), for example. | ||
|
||
If you want to customize the process further, here are some useful [templates](https://github.com/google/oss-vulnerability-guide/tree/main/templates) | ||
for opening security policies and reporting vulnerabilities. | ||
|
||
It might not be a bad idea to link to this blog post or the resources below if you are a security researcher reporting a vulnerability to a project. | ||
Maybe if that had happened with log4j, it would have been handled differently. | ||
|
||
## Other resources | ||
* Github's guide on [coordinated disclosure](https://docs.github.com/en/code-security/security-advisories/about-coordinated-disclosure-of-security-vulnerabilities) | ||
* Google has compiled a very good but little-known [comprehensive guide](https://github.com/google/oss-vulnerability-guide) on dealing with exactly these situations. Big thank you to them for publishing this. | ||
* OpenSSF has a [Vulnerability Disclosures Working Group](https://github.com/ossf/wg-vulnerability-disclosures) that works to improve vulnerability disclosures in Open Source Software. | ||
|
||
## That's it | ||
These steps aren't that hard to follow, but they're different from the normal development flow of Open Source software. | ||
Coordinated disclosure is common knowledge in security communities, but now we can say for sure | ||
that not every Open Source developer knows how to do it. | ||
Not everyone has a background in security, and that's okay. It's [our goal](https://www.lunasec.io/docs/) to make security | ||
as easy and accessible as possible for everyday developers. | ||
|
||
### Stay in the loop | ||
import ContactForm from '../src/components/ContactForm.jsx' | ||
|
||
<ContactForm/> |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+26.4 KB
docs/static/img/security-advisory-new-draft-security-advisory-button.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
**STOP**: Is this a **security vulnerability**? If so, follow Responsible Disclosure and email us at security@lunasec.io | ||
instead of opening a public PR. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters