switch to ssllabs v3 api endpoint #34

jhoblitt · 2018-03-08T15:46:44Z

The v2 endpoint has been dead for the last ~60 days.

Pull Request Checklist

Resolves #33

General

Update Changelog following the conventions laid out here
RuboCop passes
Existing tests pass

The v2 endpoint has been dead for the last ~60 days.

jhoblitt · 2018-03-08T16:03:27Z

There is a failing rspec test but it is unrelated to the PR and is failing on master.

jhoblitt · 2018-03-08T17:56:58Z

Eh, actually, it looks like v2 isn't dead but has started taking over 2 minutes to return checks on most of my hosts. Hmm.

majormoses · 2018-03-15T05:48:32Z

I can confirm the tests are failing on master. I will take a closer look at the tests. Makes sense that they would keep the old api running for a while even if its slower. I am all for updating it as long as it does not break anything else. Can you provide a manual or automated testing artifact? It can be a simple display of redacted input and output. I will try to take a look at the test and see why it is failing.

majormoses · 2018-03-15T06:15:19Z

I am seeing both of them as unbearably slow although sometimes it returns in a somewhat reasonable time period:

$ time ./bin/check-ssl-qualys.rb --api-url https://api.ssllabs.com/api/v2/ -d google.com
CheckSSLQualys OK: google.com rated A

real	0m10.268s
user	0m0.080s
sys	0m0.008s

$ time ./bin/check-ssl-qualys.rb --api-url https://api.ssllabs.com/api/v2/ -d google.com
CheckSSLQualys OK: google.com rated A

real	2m51.734s
user	0m0.104s
sys	0m0.016s

$ time ./bin/check-ssl-qualys.rb --api-url https://api.ssllabs.com/api/v3/ -d google.com
CheckSSLQualys OK: google.com rated A

real	1m30.956s
user	0m0.088s
sys	0m0.024s

$ time ./bin/check-ssl-qualys.rb --api-url https://api.ssllabs.com/api/v3/ -d google.com
CheckSSLQualys OK: google.com rated A

real	2m51.834s
user	0m0.104s
sys	0m0.020s

From a bit of research v2 is indeed deprecated: https://github.com/ssllabs/ssllabs-scan/blob/master/ssllabs-api-docs-v2-deprecated.md
v3 api docs: https://github.com/ssllabs/ssllabs-scan/blob/master/ssllabs-api-docs-v3.md

When I use curl it returns super quickly:

$ time curl 'https://api.ssllabs.com/api/v3/analyze?host=google.com&publish=off'
{
  "host": "google.com",
  "port": 443,
  "protocol": "http",
  "isPublic": false,
  "status": "READY",
  "startTime": 1521093609962,
  "testTime": 1521093777957,
  "engineVersion": "1.31.0",
  "criteriaVersion": "2009p",
  "endpoints": [
    {
      "ipAddress": "2607:f8b0:4005:807:0:0:0:200e",
      "serverName": "sfo07s16-in-x0e.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 83862,
      "delegation": 1
    },
    {
      "ipAddress": "216.58.195.78",
      "serverName": "sfo07s16-in-f78.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 84054,
      "delegation": 1
    }
  ]
}
real	0m0.197s
user	0m0.032s
sys	0m0.008s

$ time curl 'https://api.ssllabs.com/api/v3/analyze?host=google.com&publish=off'
{
  "host": "google.com",
  "port": 443,
  "protocol": "http",
  "isPublic": false,
  "status": "READY",
  "startTime": 1521093609962,
  "testTime": 1521093777957,
  "engineVersion": "1.31.0",
  "criteriaVersion": "2009p",
  "endpoints": [
    {
      "ipAddress": "2607:f8b0:4005:807:0:0:0:200e",
      "serverName": "sfo07s16-in-x0e.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 83862,
      "delegation": 1
    },
    {
      "ipAddress": "216.58.195.78",
      "serverName": "sfo07s16-in-f78.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 84054,
      "delegation": 1
    }
  ]
}
real	0m0.191s
user	0m0.036s
sys	0m0.004s

$ time curl 'https://api.ssllabs.com/api/v2/analyze?host=google.com&publish=off'
{
  "host": "google.com",
  "port": 443,
  "protocol": "HTTP",
  "isPublic": false,
  "status": "READY",
  "startTime": 1521093609962,
  "testTime": 1521093777957,
  "engineVersion": "1.31.0",
  "criteriaVersion": "2009p",
  "endpoints": [
    {
      "ipAddress": "2607:f8b0:4005:807:0:0:0:200e",
      "serverName": "sfo07s16-in-x0e.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 83862,
      "eta": 3,
      "delegation": 1
    },
    {
      "ipAddress": "216.58.195.78",
      "serverName": "sfo07s16-in-f78.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 84054,
      "eta": 3,
      "delegation": 1
    }
  ]
}
real	0m0.192s
user	0m0.036s
sys	0m0.008s

$ time curl 'https://api.ssllabs.com/api/v2/analyze?host=google.com&publish=off'
{
  "host": "google.com",
  "port": 443,
  "protocol": "HTTP",
  "isPublic": false,
  "status": "READY",
  "startTime": 1521093609962,
  "testTime": 1521093777957,
  "engineVersion": "1.31.0",
  "criteriaVersion": "2009p",
  "endpoints": [
    {
      "ipAddress": "2607:f8b0:4005:807:0:0:0:200e",
      "serverName": "sfo07s16-in-x0e.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 83862,
      "eta": 3,
      "delegation": 1
    },
    {
      "ipAddress": "216.58.195.78",
      "serverName": "sfo07s16-in-f78.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 84054,
      "eta": 3,
      "delegation": 1
    }
  ]
}
real	0m0.204s
user	0m0.028s
sys	0m0.012s

This leads me to believe while we should indeed switch there is something horribly wrong in the code that causes it to be that inefficient. I looked through the options and even with attempting a single check occurence it is timing out.

$ time ./bin/check-ssl-qualys.rb --api-url https://api.ssllabs.com/api/v2/ -d google.com --number-checks 1
CheckSSLQualys WARNING: Timeout waiting for check to finish

real	0m10.376s
user	0m0.076s
sys	0m0.008s

$ time ./bin/check-ssl-qualys.rb --api-url https://api.ssllabs.com/api/v2/ -d google.com --number-checks 1
CheckSSLQualys WARNING: Timeout waiting for check to finish

real	0m10.182s
user	0m0.076s
sys	0m0.008s

majormoses · 2018-03-15T06:24:01Z

I tested as far back as 1.0.0 and it is having the same issues regardless of api version.

majormoses · 2018-03-15T06:58:41Z

Because I was paranoid of caching:

$ time curl 'https://api.ssllabs.com/api/v2/analyze?host=google.com&publish=off&fromCache=false'
{
  "host": "google.com",
  "port": 443,
  "protocol": "HTTP",
  "isPublic": false,
  "status": "READY",
  "startTime": 1521096047622,
  "testTime": 1521096215300,
  "engineVersion": "1.31.0",
  "criteriaVersion": "2009p",
  "endpoints": [
    {
      "ipAddress": "2607:f8b0:4005:807:0:0:0:200e",
      "serverName": "sfo07s16-in-x0e.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 83853,
      "eta": 5,
      "delegation": 1
    },
    {
      "ipAddress": "172.217.0.46",
      "serverName": "lga15s43-in-f46.1e100.net",
      "statusMessage": "Ready",
      "grade": "A",
      "gradeTrustIgnored": "A",
      "hasWarnings": false,
      "isExceptional": false,
      "progress": 100,
      "duration": 83823,
      "eta": 5,
      "delegation": 1
    }
  ]
}
real	0m0.452s
user	0m0.032s
sys	0m0.008s

majormoses · 2018-03-15T07:52:30Z

So I did some more testing and there is something very odd it also does not seem to be very fast when clearing the cache and running the test from the browser. I added some debug code and seem to be able to get it to work pretty consistently. I noticed after putting in some debug code that there was an eta field which seems like something we should look at rather than randomly sleep for some arbitrary number. In some cases I saw the eta return over 200 which makes me feel like that is the way to go. Can you provide a bit of context on how you previously used this check? What I am looking for specifically is what kind of interval did you set this check to run on, how many check attempts, and what was your time between value? I am trying to gauge if I have unrealistic expectations for how fast it should return as it seems like it takes too long.

$ time bundle exec ./bin/check-ssl-qualys-debug.rb --api-url https://api.ssllabs.com/api/v3/ -d google.com --number-checks 5 --time-between 2
1
"start_time: 2018-03-15 00:35:51 -0700"
"response: {\"host\"=>\"google.com\", \"port\"=>443, \"protocol\"=>\"http\", \"isPublic\"=>false, \"status\"=>\"DNS\", \"statusMessage\"=>\"Resolving domain names\", \"startTime\"=>1521099351970, \"engineVersion\"=>\"1.31.0\", \"criteriaVersion\"=>\"2009p\"}"
"end_time: 2.099095709"
2
"start_time: 2018-03-15 00:35:53 -0700"
"response: {\"host\"=>\"google.com\", \"port\"=>443, \"protocol\"=>\"http\", \"isPublic\"=>false, \"status\"=>\"IN_PROGRESS\", \"startTime\"=>1521099351970, \"engineVersion\"=>\"1.31.0\", \"criteriaVersion\"=>\"2009p\", \"endpoints\"=>[{\"ipAddress\"=>\"2607:f8b0:4005:807:0:0:0:200e\", \"serverName\"=>\"sfo07s16-in-x0e.1e100.net\", \"statusMessage\"=>\"In progress\", \"statusDetails\"=>\"TESTING_PROTOCOL_INTOLERANCE_399\", \"statusDetailsMessage\"=>\"Testing Protocol Intolerance (TLS 1.152)\", \"delegation\"=>1}, {\"ipAddress\"=>\"216.58.195.78\", \"serverName\"=>\"sfo07s16-in-f78.1e100.net\", \"statusMessage\"=>\"Pending\", \"delegation\"=>1}]}"
"end_time: 2.081251269"
3
"start_time: 2018-03-15 00:35:55 -0700"
"response: {\"host\"=>\"google.com\", \"port\"=>443, \"protocol\"=>\"http\", \"isPublic\"=>false, \"status\"=>\"IN_PROGRESS\", \"startTime\"=>1521099351970, \"engineVersion\"=>\"1.31.0\", \"criteriaVersion\"=>\"2009p\", \"endpoints\"=>[{\"ipAddress\"=>\"2607:f8b0:4005:807:0:0:0:200e\", \"serverName\"=>\"sfo07s16-in-x0e.1e100.net\", \"statusMessage\"=>\"In progress\", \"statusDetails\"=>\"TESTING_RENEGOTIATION\", \"statusDetailsMessage\"=>\"Testing renegotiation\", \"delegation\"=>1}, {\"ipAddress\"=>\"216.58.195.78\", \"serverName\"=>\"sfo07s16-in-f78.1e100.net\", \"statusMessage\"=>\"Pending\", \"delegation\"=>1}]}"
"end_time: 2.086732863"
4
"start_time: 2018-03-15 00:35:57 -0700"
"response: {\"host\"=>\"google.com\", \"port\"=>443, \"protocol\"=>\"http\", \"isPublic\"=>false, \"status\"=>\"IN_PROGRESS\", \"startTime\"=>1521099351970, \"engineVersion\"=>\"1.31.0\", \"criteriaVersion\"=>\"2009p\", \"endpoints\"=>[{\"ipAddress\"=>\"2607:f8b0:4005:807:0:0:0:200e\", \"serverName\"=>\"sfo07s16-in-x0e.1e100.net\", \"statusMessage\"=>\"In progress\", \"statusDetails\"=>\"TESTING_SUITES\", \"statusDetailsMessage\"=>\"Determining available cipher suites\", \"progress\"=>2, \"eta\"=>223, \"delegation\"=>1}, {\"ipAddress\"=>\"216.58.195.78\", \"serverName\"=>\"sfo07s16-in-f78.1e100.net\", \"statusMessage\"=>\"Pending\", \"delegation\"=>1}]}"
"end_time: 223.08093911"
5
"start_time: 2018-03-15 00:39:40 -0700"
"response: {\"host\"=>\"google.com\", \"port\"=>443, \"protocol\"=>\"http\", \"isPublic\"=>false, \"status\"=>\"READY\", \"startTime\"=>1521099351970, \"testTime\"=>1521099520028, \"engineVersion\"=>\"1.31.0\", \"criteriaVersion\"=>\"2009p\", \"endpoints\"=>[{\"ipAddress\"=>\"2607:f8b0:4005:807:0:0:0:200e\", \"serverName\"=>\"sfo07s16-in-x0e.1e100.net\", \"statusMessage\"=>\"Ready\", \"grade\"=>\"A\", \"gradeTrustIgnored\"=>\"A\", \"hasWarnings\"=>false, \"isExceptional\"=>false, \"progress\"=>100, \"duration\"=>83840, \"delegation\"=>1}, {\"ipAddress\"=>\"216.58.195.78\", \"serverName\"=>\"sfo07s16-in-f78.1e100.net\", \"statusMessage\"=>\"Ready\", \"grade\"=>\"A\", \"gradeTrustIgnored\"=>\"A\", \"hasWarnings\"=>false, \"isExceptional\"=>false, \"progress\"=>100, \"duration\"=>84195, \"delegation\"=>1}]}"
CheckSSLQualys OK: google.com rated A

real	3m49.814s
user	0m0.152s
sys	0m0.036s

Here is the diff of my debug code:

$ diff bin/check-ssl-qualys.rb bin/check-ssl-qualys-debug.rb
110c110,118
<       json = ssl_check(step != 1)
---
>       p step
>       start_time = Time.now
>       p "start_time: #{start_time}"
>       json = if step == 1
>                ssl_check(false)
>              else
>                ssl_check(true)
>              end
>       p "response: #{json}"
112c120,129
<       sleep(config[:between_checks])
---
>       # sleep(config[:between_checks])
>       begin
>         # p "eta: #{json['endpoints'].first['eta']}"
>         sleep(json['endpoints'].first['eta'])
>       rescue NoMethodError
>         sleep(config[:between_checks])
>       rescue TypeError
>         sleep(config[:between_checks])
>       end
>       p "end_time: #{Time.now - start_time}"

I did one more test and when setting the cache to true it comes back very quickly. I feel like maybe that is an implementation detail best left to the user rather than hard coded to say the first request is always without cache. I realize that there is limited value in running this with cache turned on I guess depending on how often they clear their cache it might make sense.

jhoblitt · 2018-03-15T12:43:57Z

While v3 did seem to be a bit better for v2 for a while, after a few days it was back to exceeding a 5 service minute check timeout (which was increased from 2 -> 4 -> 5min while chasing the random failures).

I'm actually running the check under an old school nagios setup. The check is setup on run once per day on ~10 hosts. This had been working acceptable for about a year. I think the high rate of failure started about 2 months ago. I've been wondering if I should drop them a line to see if they are throttling requests by IP address.

define command {
        command_name    check_ssllabs
        command_line    /usr/bin/check-ssl-qualys.rb --api-url https://api.ssllabs.com/api/v3/ -d $HOSTADDRESS$ -c A
}

define service{
        name                    daily-service
        use                     generic-service
        check_interval          1440
        retry_interval          1440
        notification_interval   0
        register                0
        notifications_enabled   1
}

define service {
        host_name               ci.lsst.codes
        service_description     ssllabs
        check_command           check_ssllabs
        use                     daily-service
}

service_check_timeout=300

majormoses · 2018-03-15T17:47:07Z

Thanks for the context, from my tests when submitting without cache it seems to work OK although slower than most sensu/nagios checks I would expect to run but it is an async operation and not something we can control. I will submit a PR with those changes minus any debug bits.

throttling requests by IP address.

I don't think so, you would receive a 429 when submitting too many requests: https://github.com/ssllabs/ssllabs-scan/blob/master/ssllabs-api-docs-v3.md#access-rate-and-rate-limiting

majormoses · 2018-03-15T17:59:53Z

I created #35 to better address the issues here, thank you for your help! I will try to get to it over the weekend if I can. If you want to take a stab at it before then feel free to. You should be able to take the rough code and clean it up a bit if you want to.

switch to ssllabs v3 api endpoint

5f170b9

The v2 endpoint has been dead for the last ~60 days.

majormoses mentioned this pull request Mar 15, 2018

check-ssl-qualys.rb regularly times out #35

Closed

majormoses closed this Mar 15, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

switch to ssllabs v3 api endpoint #34

switch to ssllabs v3 api endpoint #34

jhoblitt commented Mar 8, 2018 •

edited

Loading

jhoblitt commented Mar 8, 2018

jhoblitt commented Mar 8, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

jhoblitt commented Mar 15, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

switch to ssllabs v3 api endpoint #34

switch to ssllabs v3 api endpoint #34

Conversation

jhoblitt commented Mar 8, 2018 • edited Loading

Pull Request Checklist

General

jhoblitt commented Mar 8, 2018

jhoblitt commented Mar 8, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

jhoblitt commented Mar 15, 2018

majormoses commented Mar 15, 2018

majormoses commented Mar 15, 2018

jhoblitt commented Mar 8, 2018 •

edited

Loading