Watcher: Switch to pagerduty API V2

[spinscale]

Watcher is currently using the v1 API events endpoint, available under https://v1.developer.pagerduty.com/documentation/events

This is however decommissioned at the 19th of October, so we have to switch to V2 API as soon as possible.

More info under https://v2.developer.pagerduty.com/

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

If I read this correctly:

On April 24, 2018, at 22:00 PDT (UTC+7), our v1 REST API is entering a decommissioning period. Starting on that date, you will no longer be able to generate v1 REST API keys, but the v1 REST API will remain operational.

it implies that new users would not be able to use our PagerDuty integration today.

[hub-cap]

So after finishing the PR, I got to reading the actual docs here. Look at the gem I found.

WILL THE V1 EVENTS API STILL BE SUPPORTED?
This change only affects the v1 REST API. You can continue to use the v1 Events API as you do today.Learn more about our the differences between our APIs here.

This comes directly from the linked doc in the issue description. The only place we use the PagerDuty API is here, and for brevity ill include the 2 lines below.

    static final String HOST = "events.pagerduty.com";
    static final String PATH = "/generic/2010-04-15/create_event.json";

This means we are only using the Events API, and yes we are still using the V1 Events API, which will not be affected by this.

The good news is that we can still migrate to the V2 Events API. PagerDuty has a V2 Events API already. The work is done in the PR attached to this issue. I am not sure this warrants a blocker / critical status anymore though.

[spinscale]

Agreed on this one. Events API seems safe. I think it makes sense to switch to the newer API then on master only, once verified that old tokens can be reused. But this is not a blocker anymore.

[hub-cap]

I think its worth backporting to 6.5 so future backports (which there will at least be one I know of) will be clean. Sound good @spinscale ?

[jasonslater2000]

I am running ES v6.4.3. Just tried creating a new API key. PagerDuty only creates v2 keys now, and it assigned a 20 character alphanumeric key. Plugged API key into elasticsearch.yml and created a watcher to use it. Checked .watcher-history index and found the call had failed with a 400 error and the following body.

"body": """{"status":"invalid event","message":"Event object is invalid","errors":["Service key is the wrong length (should be 32 characters)","Service key cannot contain non-alphanumeric characters."]}"""

[jasonslater2000]

Got it working, but had to create the API key in PagerDuty via Configuration/Services, rather than Configuration/API Access as is documented at https://www.elastic.co/guide/en/watcher/current/configuring-pagerduty.html

N.B.: During key creation, I selected "Events API v2" key, and this yielded a 32-character key.

[spinscale]

@hub-cap can you take a look in the docs and fix if needed?

[jasonslater2000]

@hub-cap my pagerduty trial still has 12 days remaining. let me know if I can provide screenshots or my trial credentials for you to take screenshots.

[hub-cap]

heh, i missed this ping somehow. grr. Ill update the docs here, thanks for the notificaiton.