Skip to content

Commit

Permalink
ACCEPT/REJECT instead of VERIFY SUCCESS/FAILURE
Browse files Browse the repository at this point in the history
  • Loading branch information
@aleksiklasila
aleksiklasila committed Jul 27, 2016
1 parent 7905c8d commit b93c7cb
Show file tree
Hide file tree
Showing 39 changed files with 396 additions and 398 deletions.
8 changes: 4 additions & 4 deletions cases/osx-10.11.5.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,24 +72,24 @@ OpenSSL 0.9.8zh 14 Jan 2016

```
$ /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 stubs/python-urllib2/run.py sha256.badssl.com 443
VERIFY SUCCESS
ACCEPT
```

## CA cert bundle defined - succeeds unexpectedly
```
$ /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 stubs/python-urllib2/run.py sha256.badssl.com 443 pki/certs/theonlycertitrust.crt
VERIFY SUCCESS
ACCEPT
```

## Apple's TEA-patch disabled - fails as expected

```
$ env OPENSSL_X509_TEA_DISABLE=1 /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 stubs/python-urllib2/run.py sha256.badssl.com 443
VERIFY FAILURE
REJECT
```
## Running with brew-installed third party python interpreter - fails as expected

```
$ /usr/local/bin/python stubs/python-urllib2/run.py sha256.ssllabs.com 443 pki/certs/theonlycertitrust.crt
VERIFY FAILURE
REJECT
```
4 changes: 2 additions & 2 deletions popularisation/workshop/Run-example.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ public static void main(String[] args)
url = new URL(https_url);
HttpsURLConnection con = (HttpsURLConnection)url.openConnection();
con.getResponseCode();
System.out.println("VERIFY SUCCESS");
System.out.println("ACCEPT");
} catch (javax.net.ssl.SSLHandshakeException e) {
System.out.println("VERIFY FAILURE"); // did not accept connection
System.out.println("REJECT"); // did not accept connection
} catch (Exception e) {
System.out.println(e.getCause().getMessage());
System.exit(3); //some other error?
Expand Down
4 changes: 2 additions & 2 deletions popularisation/workshop/presentation.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,8 @@ line arguments (`<host> <port> [ca-bundle]`):

All stubs should return one of the following strings to the standard output:

* `VERIFY SUCCESS` when connection was established in a secure way
* `VERIFY FAILURE` when connection failed to establish in a secure way
* `ACCEPT` when connection was established in a secure way
* `REJECT` when connection failed to establish in a secure way
* `UNSUPPORTED` if the example has not implemented the requested behaviour (e.g. setting
CA certificate bundle)

Expand Down
66 changes: 33 additions & 33 deletions runners/bashtls/shared/simplerunner/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,40 +68,40 @@ against badssl (run the code on your computer and you will see the colorcoded ve
```console
$ bash run mono '../trytls/stubs/cSharp-Net/run.exe' 'conf/badssl-all' _ _ CSharp-Net | sort

[cSharp-Net][ PASS ][VERIFY FAILURE][ dh480 ][dh480.badssl.com]
[cSharp-Net][ PASS ][VERIFY FAILURE][ dsdtestprovider ][dsdtestprovider.badssl.com]
[cSharp-Net][ PASS ][VERIFY FAILURE][ edellroot ][edellroot.badssl.com]
[cSharp-Net][ PASS ][VERIFY FAILURE][ self-signed ][self-signed.badssl.com]
[cSharp-Net][ PASS ][VERIFY FAILURE][ superfish ][superfish.badssl.com]
[cSharp-Net][ PASS ][VERIFY FAILURE][ untrusted-root ][untrusted-root.badssl.com]
[cSharp-Net][ PASS ][VERIFY FAILURE][ wrong host ][wrong.host.badssl.com]
[cSharp-Net][ PASS ][VERIFY SUCCESS][ sha-256 ][sha256.badssl.com]
[cSharp-Net][ PASS ][VERIFY SUCCESS][ supports SNI ][badssl.com]
[cSharp-Net][ PASS ][REJECT][ dh480 ][dh480.badssl.com]
[cSharp-Net][ PASS ][REJECT][ dsdtestprovider ][dsdtestprovider.badssl.com]
[cSharp-Net][ PASS ][REJECT][ edellroot ][edellroot.badssl.com]
[cSharp-Net][ PASS ][REJECT][ self-signed ][self-signed.badssl.com]
[cSharp-Net][ PASS ][REJECT][ superfish ][superfish.badssl.com]
[cSharp-Net][ PASS ][REJECT][ untrusted-root ][untrusted-root.badssl.com]
[cSharp-Net][ PASS ][REJECT][ wrong host ][wrong.host.badssl.com]
[cSharp-Net][ PASS ][ACCEPT][ sha-256 ][sha256.badssl.com]
[cSharp-Net][ PASS ][ACCEPT][ supports SNI ][badssl.com]
[cSharp-Net][ OK? ][ UNSUPPORTED ][ disable ca-bundles ][badssl.com]
[cSharp-Net][ OK? ][VERIFY FAILURE][ dh1024 ][dh1024.badssl.com]
[cSharp-Net][ OK? ][VERIFY FAILURE][ dh-small-subgroup ][dh-small-subgroup.badssl.com]
[cSharp-Net][ OK? ][VERIFY FAILURE][ incomplete-chain ][incomplete-chain.badssl.com]
[cSharp-Net][ OK? ][VERIFY FAILURE][ mozilla-intermediate ][mozilla-intermidiate.badssl.com]
[cSharp-Net][ OK? ][VERIFY FAILURE][ mozilla-modern ][mozilla-modern.badssl.com]
[cSharp-Net][ OK? ][VERIFY FAILURE][ subdomain.preloaded-hsts ][subdomain.preloaded-hsts.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ cbc ][cbc.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ hsts ][hsts.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ mixed ][mixed.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ mixed-favicon ][mixed-favicon.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ mixed-script ][mixed-script.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ mozilla-old ][mozilla-old.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ pinning-test ][pinning-test.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ preloaded-hsts ][preloaded-hsts.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ rc4 ][rc4.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ rsa8192 ][rsa8192.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ sha1-2016 ][sha1-2016.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ sha1-2017 ][sha1-2017.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ upgrade ][upgrade.badssl.com]
[cSharp-Net][ OK? ][VERIFY SUCCESS][ very ][very.badssl.com]
[cSharp-Net][ FAIL ][VERIFY FAILURE][ 10000-sans (Bad in ten years) ][10000-sans.badssl.com]
[cSharp-Net][ FAIL ][VERIFY FAILURE][ 1000-sans ][1000-sans.badssl.com]
[cSharp-Net][ FAIL ][VERIFY FAILURE][ dh2048 ][dh2048.badssl.com]
[cSharp-Net][ FAIL ][VERIFY SUCCESS][ expired ][expired.badssl.com]
[cSharp-Net][ OK? ][REJECT][ dh1024 ][dh1024.badssl.com]
[cSharp-Net][ OK? ][REJECT][ dh-small-subgroup ][dh-small-subgroup.badssl.com]
[cSharp-Net][ OK? ][REJECT][ incomplete-chain ][incomplete-chain.badssl.com]
[cSharp-Net][ OK? ][REJECT][ mozilla-intermediate ][mozilla-intermidiate.badssl.com]
[cSharp-Net][ OK? ][REJECT][ mozilla-modern ][mozilla-modern.badssl.com]
[cSharp-Net][ OK? ][REJECT][ subdomain.preloaded-hsts ][subdomain.preloaded-hsts.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ cbc ][cbc.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ hsts ][hsts.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ mixed ][mixed.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ mixed-favicon ][mixed-favicon.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ mixed-script ][mixed-script.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ mozilla-old ][mozilla-old.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ pinning-test ][pinning-test.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ preloaded-hsts ][preloaded-hsts.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ rc4 ][rc4.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ rsa8192 ][rsa8192.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ sha1-2016 ][sha1-2016.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ sha1-2017 ][sha1-2017.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ upgrade ][upgrade.badssl.com]
[cSharp-Net][ OK? ][ACCEPT][ very ][very.badssl.com]
[cSharp-Net][ FAIL ][REJECT][ 10000-sans (Bad in ten years) ][10000-sans.badssl.com]
[cSharp-Net][ FAIL ][REJECT][ 1000-sans ][1000-sans.badssl.com]
[cSharp-Net][ FAIL ][REJECT][ dh2048 ][dh2048.badssl.com]
[cSharp-Net][ FAIL ][ACCEPT][ expired ][expired.badssl.com]


...
Expand Down
4 changes: 2 additions & 2 deletions runners/bashtls/shared/simplerunner/run
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,8 +121,8 @@ handleresponse()

}

FAILURE="VERIFY FAILURE"
SUCCESS="VERIFY SUCCESS"
FAILURE="REJECT"
SUCCESS="ACCEPT"

connect()
{
Expand Down
4 changes: 2 additions & 2 deletions runners/trytls/runner.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@ def run_one(args, host, port, cafile=None):
lines = out.splitlines()
if lines:
verdict = lines.pop()
if verdict == b"VERIFY SUCCESS":
if verdict == b"ACCEPT":
return True, "".join(lines)
elif verdict == b"VERIFY FAILURE":
elif verdict == b"REJECT":
return False, "".join(lines)
elif verdict == b"UNSUPPORTED":
raise Unsupported("".join(lines))
Expand Down
4 changes: 2 additions & 2 deletions stubs/FSharp-net/Run.fs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ let main(args) = //host, port, no support for ca-bundle at the moment
let url = String.Format("https://{0}:{1}", host, port)
try
let req = HttpWebRequest.Create(url).GetResponse()
printfn "VERIFY SUCCESS"; 0
printfn "ACCEPT"; 0
with
| :? System.Net.WebException as ex ->
if ex.Message.Contains("NameResolutionFailure") then
printfn "%s" ex.Message; 1
else
printfn "VERIFY FAILURE"; 0
printfn "REJECT"; 0
| _ as ex->
printfn "%s" ex.Message; 1
| [|_; _; _|] ->
Expand Down
74 changes: 37 additions & 37 deletions stubs/FSharp-net/results.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,45 +37,45 @@ platform: Linux (Ubuntu 16.04)
runner: simplerunner
stub: FSharp-Net 'Run.exe'
[F#-Net][ PASS ][VERIFY SUCCESS][ Valid cert ][google.com]
[F#-Net][ PASS ][ACCEPT][ Valid cert ][google.com]
[F#-Net][ PASS ][VERIFY FAILURE][ dh480 ][dh480.badssl.com]
[F#-Net][ PASS ][VERIFY FAILURE][ dsdtestprovider ][dsdtestprovider.badssl.com]
[F#-Net][ PASS ][VERIFY FAILURE][ edellroot ][edellroot.badssl.com]
[F#-Net][ PASS ][VERIFY FAILURE][ expired ][expired.badssl.com]
[F#-Net][ PASS ][VERIFY FAILURE][ self-signed ][self-signed.badssl.com]
[F#-Net][ PASS ][VERIFY FAILURE][ superfish ][superfish.badssl.com]
[F#-Net][ PASS ][VERIFY FAILURE][ untrusted-root ][untrusted-root.badssl.com]
[F#-Net][ PASS ][VERIFY FAILURE][ wrong host ][wrong.host.badssl.com]
[F#-Net][ PASS ][VERIFY SUCCESS][ sha-256 ][sha256.badssl.com]
[F#-Net][ PASS ][VERIFY SUCCESS][ supports SNI ][badssl.com]
[F#-Net][ PASS ][REJECT][ dh480 ][dh480.badssl.com]
[F#-Net][ PASS ][REJECT][ dsdtestprovider ][dsdtestprovider.badssl.com]
[F#-Net][ PASS ][REJECT][ edellroot ][edellroot.badssl.com]
[F#-Net][ PASS ][REJECT][ expired ][expired.badssl.com]
[F#-Net][ PASS ][REJECT][ self-signed ][self-signed.badssl.com]
[F#-Net][ PASS ][REJECT][ superfish ][superfish.badssl.com]
[F#-Net][ PASS ][REJECT][ untrusted-root ][untrusted-root.badssl.com]
[F#-Net][ PASS ][REJECT][ wrong host ][wrong.host.badssl.com]
[F#-Net][ PASS ][ACCEPT][ sha-256 ][sha256.badssl.com]
[F#-Net][ PASS ][ACCEPT][ supports SNI ][badssl.com]
[F#-Net][ OK? ][ UNSUPPORTED ][ disable ca-bundles ][badssl.com]
[F#-Net][ OK? ][VERIFY FAILURE][ dh1024 ][dh1024.badssl.com]
[F#-Net][ OK? ][VERIFY FAILURE][ dh-small-subgroup ][dh-small-subgroup.badssl.com]
[F#-Net][ OK? ][VERIFY FAILURE][ incomplete-chain ][incomplete-chain.badssl.com]
[F#-Net][ OK? ][VERIFY FAILURE][ mozilla-intermediate ][mozilla-intermidiate.badssl.com]
[F#-Net][ OK? ][VERIFY FAILURE][ mozilla-modern ][mozilla-modern.badssl.com]
[F#-Net][ OK? ][VERIFY FAILURE][ subdomain.preloaded-hsts ][subdomain.preloaded-hsts.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ cbc ][cbc.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ hsts ][hsts.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ mixed ][mixed.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ mixed-favicon ][mixed-favicon.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ mixed-script ][mixed-script.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ mozilla-old ][mozilla-old.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ pinning-test ][pinning-test.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ preloaded-hsts ][preloaded-hsts.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ rc4 ][rc4.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ rsa8192 ][rsa8192.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ sha1-2016 ][sha1-2016.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ sha1-2017 ][sha1-2017.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ upgrade ][upgrade.badssl.com]
[F#-Net][ OK? ][VERIFY SUCCESS][ very ][very.badssl.com]
[F#-Net][ FAIL ][VERIFY FAILURE][ 10000-sans (Bad in ten years) ][10000-sans.badssl.com]
[F#-Net][ FAIL ][VERIFY FAILURE][ 1000-sans ][1000-sans.badssl.com]
[F#-Net][ FAIL ][VERIFY FAILURE][ dh2048 ][dh2048.badssl.com]
[F#-Net][ OK? ][REJECT][ dh1024 ][dh1024.badssl.com]
[F#-Net][ OK? ][REJECT][ dh-small-subgroup ][dh-small-subgroup.badssl.com]
[F#-Net][ OK? ][REJECT][ incomplete-chain ][incomplete-chain.badssl.com]
[F#-Net][ OK? ][REJECT][ mozilla-intermediate ][mozilla-intermidiate.badssl.com]
[F#-Net][ OK? ][REJECT][ mozilla-modern ][mozilla-modern.badssl.com]
[F#-Net][ OK? ][REJECT][ subdomain.preloaded-hsts ][subdomain.preloaded-hsts.badssl.com]
[F#-Net][ OK? ][ACCEPT][ cbc ][cbc.badssl.com]
[F#-Net][ OK? ][ACCEPT][ hsts ][hsts.badssl.com]
[F#-Net][ OK? ][ACCEPT][ mixed ][mixed.badssl.com]
[F#-Net][ OK? ][ACCEPT][ mixed-favicon ][mixed-favicon.badssl.com]
[F#-Net][ OK? ][ACCEPT][ mixed-script ][mixed-script.badssl.com]
[F#-Net][ OK? ][ACCEPT][ mozilla-old ][mozilla-old.badssl.com]
[F#-Net][ OK? ][ACCEPT][ pinning-test ][pinning-test.badssl.com]
[F#-Net][ OK? ][ACCEPT][ preloaded-hsts ][preloaded-hsts.badssl.com]
[F#-Net][ OK? ][ACCEPT][ rc4 ][rc4.badssl.com]
[F#-Net][ OK? ][ACCEPT][ rsa8192 ][rsa8192.badssl.com]
[F#-Net][ OK? ][ACCEPT][ sha1-2016 ][sha1-2016.badssl.com]
[F#-Net][ OK? ][ACCEPT][ sha1-2017 ][sha1-2017.badssl.com]
[F#-Net][ OK? ][ACCEPT][ upgrade ][upgrade.badssl.com]
[F#-Net][ OK? ][ACCEPT][ very ][very.badssl.com]
[F#-Net][ FAIL ][REJECT][ 10000-sans (Bad in ten years) ][10000-sans.badssl.com]
[F#-Net][ FAIL ][REJECT][ 1000-sans ][1000-sans.badssl.com]
[F#-Net][ FAIL ][REJECT][ dh2048 ][dh2048.badssl.com]
[F#-Net][ PASS ][VERIFY FAILURE][ OS X vulnerability ][www.ssllabs.com]
[F#-Net][ PASS ][VERIFY FAILURE][ Freak ][www.ssllabs.com]
[F#-Net][ PASS ][VERIFY FAILURE][ Logjam ][www.ssllabs.com]
[F#-Net][ PASS ][REJECT][ OS X vulnerability ][www.ssllabs.com]
[F#-Net][ PASS ][REJECT][ Freak ][www.ssllabs.com]
[F#-Net][ PASS ][REJECT][ Logjam ][www.ssllabs.com]
```
12 changes: 6 additions & 6 deletions stubs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ operating system.
Stubs should attempt to establish a **secure** connection to the given
service(host + port) and catch possible errors and exceptions to determine if the connection was successful.

The last string the stub should print is the verdict (UNSUPPORTED, VERIFY SUCCESS etc.). If you want the stub to print additional context such as the reason to accept/reject connection or an error message, the stub should print them before the verdict.
The last string the stub should print is the verdict (UNSUPPORTED, ACCEPT etc.). If you want the stub to print additional context such as the reason to accept/reject connection or an error message, the stub should print them before the verdict.

The data outputted by the stub should follow the following set of instructions or a similar one.

Expand All @@ -43,11 +43,11 @@ The data outputted by the stub should follow the following set of instructions o
print "UNSUPPORTED"
return zero
3.0 else if [the stub could connect to the service] then
print "VERIFY SUCCESS"
print "ACCEPT"
return zero
4.0 else if [the stub could not connect to the service] then
4. 1 if [the stub could not connect due to reasons closely related to TLS/SSL (certificate, cipher suites, etc..)] then
print "VERIFY FAILURE"
print "REJECT"
return zero
4.2 else (the stub could not connect due to reasons unrelated to TLS/SSL (Name resolution, etc..))
goto "fatal error" (5.0, see one line below for more info)
Expand All @@ -74,14 +74,14 @@ Connecting to `google.com` on HTTPS port should be success:

```sh
$ run.test google.com 443
VERIFY SUCCESS
ACCEPT
```

Connecting to `badssl.com`'s `untrusted-root` should be failure:

```sh
$ run.test untrusted-root.badssl.com 443
VERIFY FAILURE
REJECT
```

If these simple tests work, your stub is ready to be tested with
Expand All @@ -97,7 +97,7 @@ https://mkcert.org/generate/`) and then test:

```sh
$ run.test google.com 443 ca-bundle.pem
VERIFY SUCCESS
ACCEPT
```

---
Expand Down
Loading

0 comments on commit b93c7cb

Please sign in to comment.