Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PagerDutyIntegration CRD #90

Merged
merged 14 commits into from
Jul 20, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 21 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ The PagerDuty operator is used to automate integrating Openshift Dedicated clust

This operator runs on [Hive](https://github.com/openshift/hive) and watches for new cluster deployments. Hive is an API driven OpenShift cluster providing OpenShift Dedicated provisioning and management.

## How the PagerDuty Opertor works
## How the PagerDuty Operator works

* PagerDuty's reconcile function watches for the `installed` field of the `ClusterDeployment` CRD and waits for the cluster to finish installation. It also sees if `api.openshift.com/noalerts` label is set on the `ClusterDeployment` of the new cluster being provisioned.
* The `api.openshift.com/noalerts` label is used to disable alerts from the provisioned cluster. This label is typically used on test clusters that do not require immediate attention as a result of critical issues or outages. Therefore, PagerDuty does not continue its actions if it finds this label in the new cluster's `ClusterDeployment`.
* Once the `installed` field becomes true, PagerDuty creates a secret which contains the integration key required to communicate with PagerDuty Web application.
* The PagerDutyIntegration controller watches for changes to PagerDutyIntegration CRs, and also for changes to appropriately labeled ClusterDeployment CRs (and ConfigMap/Secret/SyncSet resources owned by such a ClusterDeployment).
* For each PagerDutyIntegration CR, it will get a list of matching ClusterDeployments that have the `spec.installed` field set to true and don't have the `api.openshift.com/noalerts` label set.
* For each of these ClusterDeployments, PagerDuty creates a secret which contains the integration key required to communicate with PagerDuty Web application.
* The PagerDuty operator then creates [syncset](https://github.com/openshift/hive/blob/master/config/crds/hive_v1_syncset.yaml) with the relevant information for hive to send the PagerDuty secret to the newly provisioned cluster .
* This syncset is used by hive to deploy the pagerduty secret to the provisioned cluster so that Openshift SRE can be alerted in case of issues on the cluster.
* Generally, the pagerduty secret is deployed under the `openshift-monitoring` namespace and named `pd-secret` on the new cluster.
* This syncset is used by hive to deploy the pagerduty secret to the provisioned cluster so that the relevant SRE team get notified of alerts on the cluster.
* The pagerduty secret is deployed to the coordinates specified in the `spec.targetSecretRef` field of the PagerDutyIntegration CR.

## Development

Expand All @@ -36,20 +36,17 @@ $ oc apply -f manifests/01-namespace.yaml
$ oc apply -f manifests/02-role.yaml
$ oc apply -f manifests/03-service_account.yaml
$ oc apply -f manifests/04-role_binding.yaml
$ oc apply -f deploy/crds/pagerduty_v1alpha1_pagerdutyintegration_crd.yaml
```


Create secret with pagerduty api key, for example using a [trial account](https://www.pagerduty.com/free-trial/). You can then create an API key at https://<your-account>.pagerduty.com/api_keys. Also, you need to create the ID of you escalation policy. You can get this by clicking on your policy at https://<your-account>.pagerduty.com/escalation_policies#. The ID will afterwards be visible in the URL behind the `#` character.
Create secret with pagerduty api key, for example using a [trial account](https://www.pagerduty.com/free-trial/). You can then create an API key at https://<your-account>.pagerduty.com/api_keys.
Following is an example secret to adjust and apply with `oc apply -f <filename>`.

```yaml
apiVersion: v1
data:
ACKNOWLEDGE_TIMEOUT: MjE2MDA=
ESCALATION_POLICY: MTIzNA== #echo -n <escalation-policy-id> | base64
PAGERDUTY_API_KEY: bXktYXBpLWtleQ== #echo -n <pagerduty-api-key> | base64
RESOLVE_TIMEOUT: MA==
SERVICE_PREFIX: b3Nk
kind: Secret
metadata:
name: pagerduty-api-key
Expand All @@ -70,7 +67,7 @@ Create namespace `pagerduty-operator`.
$ oc create namespace pagerduty-operator
```

Continue to `Create ClusterDeployment`.
Continue to `Create PagerDutyIntegration`.

### Option 2: Run local built operator in minishift

Expand Down Expand Up @@ -116,26 +113,22 @@ Create a copy of `manifests/05-operator.yaml` and modify it use your image from
Deploy modified operator manifest

```terminal
$ oc apply -f path/to/modified/operator.yaml
$ oc apply -f path/to/modified/operator.yaml
```
### Create PagerDutyIntegration

### Create ClusterDeployment

`pagerduty-operator` doesn't start reconciling clusters until `status.installed` is set to `true`. To be able to set this variable via `oc edit` without actually deploying a cluster to AWS, the ClusterDeployment CRD needs to be adjusted.
There's an example at
`deploy/examples/pagerduty_v1alpha1_pagerdutyintegration_cr.yaml` that
you can edit and apply to your cluster.

```terminal
$ oc edit crd clusterdeployments.hive.openshift.io
```
You'll need to use a valid escalation policy ID from your PagerDuty account. You
can get this by clicking on your policy at
https://<your-account>.pagerduty.com/escalation_policies#. The ID will be
visible in the URL after the `#` character.

Remove `subsesource` part:
### Create ClusterDeployment

```
spec:
[...]
subresources: ## delete me
status: {} ## delete me
[...]
```
`pagerduty-operator` doesn't start reconciling clusters until `spec.installed` is set to `true`.

Create ClusterDeployment.

Expand All @@ -144,7 +137,7 @@ $ oc create namespace fake-cluster-namespace
$ oc apply -f hack/clusterdeployment/fake-clusterdeployment.yml
```

If present, set `status.installed` to true.
If present, set `spec.installed` to true.
georgettica marked this conversation as resolved.
Show resolved Hide resolved

```terminal
$ oc edit clusterdeployment fake-cluster -n fake-cluster-namespace
Expand Down
12 changes: 9 additions & 3 deletions config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,8 @@ const (
PagerDutyAPISecretName string = "pagerduty-api-key"
PagerDutyAPISecretKey string = "PAGERDUTY_API_KEY"
OperatorFinalizer string = "pd.managed.openshift.io/pagerduty"
SyncSetPostfix string = "-pd-sync"
PagerDutySecretName string = "pd-secret"
ConfigMapPostfix string = "-pd-config"
SecretSuffix string = "-pd-secret"
ConfigMapSuffix string = "-pd-config"

// PagerDutyUrgencyRule is the type of IncidentUrgencyRule for new incidents
// coming into the Service. This is for the creation of NEW SERVICES ONLY
Expand All @@ -39,3 +38,10 @@ const (
// ClusterDeploymentNoalertsLabel is the label the clusterdeployment will have if the cluster should not send alerts
ClusterDeploymentNoalertsLabel string = "api.openshift.com/noalerts"
)

// Name is used to generate the name of secondary resources (SyncSets,
// Secrets, ConfigMaps) for a ClusterDeployment that are created by
// the PagerDutyIntegration controller.
func Name(servicePrefix, clusterDeploymentName, suffix string) string {
return servicePrefix + "-" + clusterDeploymentName + suffix
}
136 changes: 136 additions & 0 deletions deploy/crds/pagerduty.openshift.io_pagerdutyintegrations_crd.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: pagerdutyintegrations.pagerduty.openshift.io
spec:
group: pagerduty.openshift.io
names:
kind: PagerDutyIntegration
listKind: PagerDutyIntegrationList
plural: pagerdutyintegrations
singular: pagerdutyintegration
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
description: PagerDutyIntegration is the Schema for the pagerdutyintegrations
API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: PagerDutyIntegrationSpec defines the desired state of PagerDutyIntegration
properties:
acknowledgeTimeout:
description: Time in seconds that an incident changes to the Triggered
State after being Acknowledged. Value must not be negative. Omitting
or setting this field to 0 will disable the feature.
minimum: 0
type: integer
clusterDeploymentSelector:
description: A label selector used to find which clusterdeployment CRs
receive a PD integration based on this configuration.
properties:
matchExpressions:
description: matchExpressions is a list of label selector requirements.
The requirements are ANDed.
items:
description: A label selector requirement is a selector that contains
values, a key, and an operator that relates the key and values.
properties:
key:
description: key is the label key that the selector applies
to.
type: string
operator:
description: operator represents a key's relationship to a
set of values. Valid operators are In, NotIn, Exists and
DoesNotExist.
type: string
values:
description: values is an array of string values. If the operator
is In or NotIn, the values array must be non-empty. If the
operator is Exists or DoesNotExist, the values array must
be empty. This array is replaced during a strategic merge
patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchLabels:
additionalProperties:
type: string
description: matchLabels is a map of {key,value} pairs. A single
{key,value} in the matchLabels map is equivalent to an element
of matchExpressions, whose key field is "key", the operator is
"In", and the values array contains only "value". The requirements
are ANDed.
type: object
type: object
escalationPolicy:
description: ID of an existing Escalation Policy in PagerDuty.
type: string
pagerdutyApiKeySecretRef:
description: Reference to the secret containing PAGERDUTY_API_KEY.
properties:
name:
description: Name is unique within a namespace to reference a secret
resource.
type: string
namespace:
description: Namespace defines the space within which the secret
name must be unique.
type: string
type: object
resolveTimeout:
description: Time in seconds that an incident is automatically resolved
if left open for that long. Value must not be negative. Omitting or
setting this field to 0 will disable the feature.
minimum: 0
type: integer
servicePrefix:
description: Prefix to set on the PagerDuty Service name.
type: string
targetSecretRef:
description: Name and namespace in the target cluster where the secret
is synced.
properties:
name:
description: Name is unique within a namespace to reference a secret
resource.
type: string
namespace:
description: Namespace defines the space within which the secret
name must be unique.
type: string
type: object
required:
- clusterDeploymentSelector
- escalationPolicy
- pagerdutyApiKeySecretRef
- servicePrefix
- targetSecretRef
type: object
status:
description: PagerDutyIntegrationStatus defines the observed state of PagerDutyIntegration
type: object
grdryn marked this conversation as resolved.
Show resolved Hide resolved
version: v1alpha1
versions:
- name: v1alpha1
served: true
storage: true
18 changes: 18 additions & 0 deletions deploy/examples/pagerduty_v1alpha1_pagerdutyintegration_cr.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: pagerduty.openshift.io/v1alpha1
kind: PagerDutyIntegration
metadata:
name: example-pagerdutyintegration
spec:
acknowledgeTimeout: 21600
resolveTimeout: 0
escalationPolicy: PA12345X
servicePrefix: test
pagerdutyApiKeySecretRef:
name: pagerduty-api-key
namespace: pagerduty-operator
clusterDeploymentSelector:
matchLabels:
api.openshift.com/test: "true"
targetSecretRef:
name: test-pd-secret
namespace: test-monitoring
12 changes: 12 additions & 0 deletions hack/generate.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/bash

# Commands need to be run from project root
cd "$( dirname "${BASH_SOURCE[0]}" )"/..

operator-sdk generate k8s
operator-sdk generate crds

# This can be removed once the operator no longer needs to be run on
# OpenShift v3.11
yq d -i deploy/crds/pagerduty.openshift.io_pagerdutyintegrations_crd.yaml \
spec.validation.openAPIV3Schema.type
13 changes: 12 additions & 1 deletion manifests/02-role.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,17 @@ apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pagerduty-operator
rules:
- apiGroups:
- pagerduty.openshift.io
resources:
- pagerdutyintegrations
- pagerdutyintegrations/status
- pagerdutyintegrations/finalizers
verbs:
- get
- list
- watch
- update
- apiGroups:
- ""
resources:
Expand Down Expand Up @@ -63,4 +74,4 @@ rules:
resources:
- routes
verbs:
- '*'
- '*'
10 changes: 10 additions & 0 deletions pkg/apis/addtoscheme_pagerduty_v1alpha1.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
package apis

import (
"github.com/openshift/pagerduty-operator/pkg/apis/pagerduty/v1alpha1"
)

func init() {
// Register the types with the Scheme so the components can map objects to GroupVersionKinds and back
AddToSchemes = append(AddToSchemes, v1alpha1.SchemeBuilder.AddToScheme)
}
4 changes: 4 additions & 0 deletions pkg/apis/pagerduty/v1alpha1/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
// Package v1alpha1 contains API Schema definitions for the pagerduty v1alpha1 API group
// +k8s:deepcopy-gen=package,register
// +groupName=pagerduty.openshift.io
package v1alpha1
Loading