Third-Party Audits and Penetration Tests

[brianerdelyi]

Hello all,

I'm still going through the CCSS but wanted to comment on Third-Party Audits and Penetration Tests. I'd like to help clarify what the intent is of this requirement. This may also help to clarify the overall tone of CCSS as a voluntary standard for developers and service providers to adopt.

From NIST's Technical Guide to Information Security Testing and Assessment (SP800-115): "An information security assessment is the process of determining how effectively an entity being assessed (e.g., host, system, network, procedure, person—known as the assessment object) meets specific security objectives". Audits, vulnerability scanning and penetration tests are merely a type of security assessment. I'd like to suggest CCSS requirement focus on "determining how effective" a particular Bitcoin product or service is rather then the need for third-party testing or verification for the following reasons:

1. As a voluntary standard this can help improve adoption by reducing the burden (and potential cost) to comply. In the US (and other jurisdictions) there are existing laws and regulations that govern false advertising once an organization voluntarily states compliance with CCSS. Perhaps higher levels could include independent verification.
2. To help ensure quality of the assessment I believe we should focus on the assessment methodology and that it is performed by a qualified individual. Perhaps we could publish an assessment methodology or compliance report (similar to PCI's RoC). Perhaps list qualifications that the assessor should have like Security+, CISSP, GIAC, CISA or CEH in addition to being a Certified Bitcoin Professional or Certified Bitcoin Expert?

As a start, I propose we name this section "Security Testing and Assessment" to clearly focus on the need to perform security testing. Levels might then be something like:

Level 1: One-time security assessment. Self-assessment questionnaire.
Level 2: Annual security assessment (by non-developer?). Or assessed during changes to the product or services?
Level 3: Third-party assessment. Annual. To include penetration testing.

[mperklin]

Brian,

Great comments.
You're absolutely right that an assessment methodology is needed in order to bring this draft up to a level where it can begin to be compared with actual standards. We can begin drafting these methodologies in a git repo and fill them out; we'd love your assistance in contributing to these documents once we get that ball rolling.

I like your renaming of the Security Testing and Assessment section. I agree it does focus on the need to perform security testing. The levels you have suggested are a good start of a discussion. We're compiling all of the suggestions made so-far and trying to package them all together.

Thanks for your participation in this ambitious project!

[brianerdelyi]

I’ve been giving it more thought at a high-level.

To put the levels into perspective, I think we should consider it based on risk. Level 1 for the lowest level of risk and level 3 for the highest type of risk. This may align with the different scenarios of who/what may choose to obtain certification.

Level 1: This level of certification is intended for a desktop or mobile based application who’s functions are to: generate keys/seeds, store keys and sign transactions.
Level 2: This level of certification is intended for third-party websites or services that provides the following features/capabilities: . This level of certification does not apply to third-party services that have 100%
Level 3: This level of certification is intended for third-party websites or services that f[provide the following features/capabilities: the ability to buy/sell bitcoins (i.e. an exchange), accepts payments (via wires, ACH, payment card, PayPal or other payment processors) and/or has control of ALL the keys necessary to sign transactions on behalf of users.

What do you think? If we understand the scope of each level, we can assign requirements in each of the categories you have listed.

Regards,
Brian

[Abstrct]

Hi Brian,

I'm kind of conflicted here. On one hand, I agree that adding a cleaner definition of scope to the levels would help focus what components are needed for each level.

On the other hand, the standard right now is designed to consider the entire ecosystem as a whole. It isn't just about how one organization handles their security, but how related pieces fit together. Security for a consumer still goes far beyond what kind of application they use on their desktop and, on the other end, MSBs still need to have procedures in place about how individuals in their organization interact with cryptocurrencies from their desktop.

The standard needs to be able to address these differences and overlap. I'm not sure if designating a level for each type of actor or organization would accomplish this.

We should discuss this further but let's create a new issue for this topic.

-Josh

[Abstrct]

Getting back on topic, I am going to assign this task to myself for now. I will formalize what Brian has proposed into a pull request and submit that once its ready for further discussion.