Feature: Expanded notification support

[wenottingham]

AWX Feature Spec - Expanded notifications

User Persona

Automation user

Usage Cases

There are a variety of ways that notifications could be made more useful for users. Among them:

• Allow for notification on job start, not just job end
• Allow for templated notification content with substitutable fields based on job parameters (requires flexible notification size, may be restricted to certain types). This should include use of filters for e.g. timzeone conversion on dates.
• Users and teams should be able to define their own notifications that can be attached to arbitrary jobs (including at launch time)
• "User who launched job" should be a selectable destination for an e-mail notification
• The notification system should notify on assorted AWX/Tower events (license related, ENOSPC, cluster health, etc.) (removed due to new metrics endpoint)

API Considerations

Significant.

UI/UX Considerations

All of this will need to be configurable.

Docs

QA Suggestions

Notes

Updates/Changelog
2017-09-08: Initial version

[matburt]

Templated notifications can get a bit tricky because some of our destinations will not allow extended information, but some definitely do and we're not taking advantage of all of them right now (Slack and Hipchat come to mind).

[matburt]

We should come up with a class of notifications that represent the system notifications and make those configurable by system administrators. This should not be made to replace an external monitoring system.

[matburt]

This also should implement websocket events for notifications (at least notification tests)

[MichalTaratuta]

I'm surprised that this has not been done yet, Tower is at version 3.2.1 and you can't modify the "template"

[pritho]

Hey, it should also be possible to allow notifications to be sent upon changes, not solely for success or failed. This would allow for security monitoring and reporting of infrastructure/configuration changes.

[wenottingham]

Playbooks only have a success and failure state; 'changed' only applies to hosts.

[pritho]

Sure, I'm aware of that, but it would be helpful to be able to receive a notification if any of the hosts of a play had a task applied that changed a configuration.
For example if somebody keeps disabling selinux on a certain host.

[eviln1]

Hi,
It would be awesome if there was a support for Grafana Annotations (http://docs.grafana.org/http_api/annotations/)

The easiest form would be basically a webhook with more features: allow selection the http method, and a field where I could write the POST payload as a Jinja template, and substitute some variables.

[MBcom]

Is it possible to get the output of the debug module/ stdout into the e-mail notfications?
We have a playbook which shows us hosts who need updates and it would be great if we could get an email containing the complete ansible output for all hosts. The mail module sends us a mail for every host and this is not the behaviour we want. And the debug module output is not shown in the notification mail.

[wenottingham]

Is it possible to get the output of the debug module/ stdout into the e-mail notfications?

This is highly unlikely to ever happen, just due to the scaling problem it causes if you're trying to schlep a 10MB log file around.

[MBcom]

@wenottingham ok your are right generally it is not a good idea but it would be nice if there were checkbox to enable this feature anyhow - because not every environment contains thousands of servers
Or could you tell me the code position where the notifactions are generated - so I could change it on my own for our company?

[matburt]

You can find the notification backends here:

https://github.com/ansible/awx/tree/devel/awx/main/notifications

If you have any other questions related to this specific topic it'd be best to take it up on the mailing list or IRC.

[MBcom]

@matburt thank you for your fast reply
Can you tell me where the json body is generated? I am new in python and I could not find the concrete code position...

I changed in /usr/lib/python2.7/site-packages/awx/main/models/jobs (in docker containers) (

awx/awx/main/models/jobs.py

Lines 685 to 692 in 07680dd

	for h in self.job_host_summaries.all():
	all_hosts[h.host_name] = dict(failed=h.failed,
	changed=h.changed,
	dark=h.dark,
	failures=h.failures,
	ok=h.ok,
	processed=h.processed,
	skipped=h.skipped)

) to

all_hosts[h.host_name] = dict(failed=h.failed,
                                          changed=h.changed,
                                          dark=h.dark,
                                          failures=h.failures,
                                          ok=h.ok,
                                          processed=h.processed,
                                          skipped=h.skipped,
                                          stdout=h.stdout)

but this does not put the stdout in the notification.

(building from source does not works for me, the playbook exits with an error: unexpected parameter: 'stream' in /usr/lib/python2.7/dist-packages/ansible/modules/cloud/docker/docker_image.py Line 602)

[ryanpetrello]

@MBcom stdout for the job is not stored in a host-specific manner in the way you're attempting to access it; if you wanted to include the stdout for the entire job, you'd need to add it here:

awx/awx/main/models/jobs.py

Line 699 in 07680dd

hosts=all_hosts))

e.g.,

...
hosts=all_hosts,
stdout=self.result_stdout_raw_handle().read().decode('utf-8'))

Keep in mind that these interfaces are subject to change (future upgrades of awx likely will break with your one-off changes), and as @wenottingham mentioned above, this is likely not the type of change we'd introduce to awx due to the performance implications.

[MBcom]

An working AWX with mail notifications containing the ansible output can be found at https://github.com/mbcom/awx/

@ryanpetrello and @matburt - thanks for helping me on this issue

[ghost]

+1 Must needed feature

[ryanpetrello]

FYI, we've just released a new version of awx (https://github.com/ansible/awx/releases/tag/3.0.0) which includes support for notifications using the Grafana annotations API.

[ryanpetrello]

Implementation Notes for "On Start" Notification Templates

Discussed implementation details of this feature w/ @beeankha today. Here are some highlights of the work to do:

1. Add a new relationship to awx.main.models.base.NotificationFieldsModel. This field is inherited by UnifiedJobTemplate and Organization.

2. Create a new migration (using awx-manage makemigrations so that the new many-to-many relationship tables are created for housing this data.

3. Add new API endpoints to support these new associate/disassociate relationships:

   • /api/v2/job_templates/N/notification_templates_start
   • /api/v2/projects/N/notification_templates_start
   • /api/v2/inventory_sources/N/notification_templates_start
   • /api/v2/organizations/N/notification_templates_start

4. Write API tests to verify the above endpoints properly affect the database (sending associate/disassociate requests actually affect the database in the anticipated manner).

At this point in time, the UI probably has what they need to start integrating with the API. @mabashian how do you feel about this?

5. In terms of actually implementing the notification under the hood, the plan is to augment the awx.main.models.notification.JobNotificationMixin().send_notification_templates() function so that it can also handle the "running" state.

[ryanpetrello]

Implementation Notes for Custom HTTP Method + Basic Auth Support

Discussed implementation details of this feature w/ @beeankha today. Here are some highlights of the work to do:

Custom HTTP Method

1. Add a new init_parameter to awx.main.notifications.webhook_backend.WebhookBackend. This field will capture a new "HTTP Method" field that will be configurable from the UI. Ideally, this field will not be free-form text, but instead will be limited to a set of choices (likely to PUT or POST). We may need to add some form of support to the init_parameter for specifying parameters, and we'll likely need code somewhere (potentially in the serializer for templates?) that ensures users don't pass data that isn't PUT or POST.
2. Users should be able to edit this method field for notification templates, and it should affect the HTTP request method utilized when the notification is sent. cc @mabashian @trahman73 the UI may need smarts to render this metadata as dropdown/selectable instead of a plain text field when adding/editing a Notification Template.
3. When upgrading AWX, any web hook notification templates created before this change will need to be verified to make sure they still fall back to using POST by default (the prior behavior).

Basic Auth Support

1. Add two new init_parameters to awx.main.notifications.webhook_backend.WebhookBackend, one for the username, and one for the password.
2. Update the code that sends HTTP requests so that it properly passes the username into password into the requests library so that the proper Authorization: Basic ... header is generated for the request.
3. Under the hood, any notification template that's saved in the database should store the password in an encrypted (non plaintext) fashion. note: this should already be implemented via eb3d663, we just need to confirm it works. We should make a web hook notification template w/ a password and verify that:
   a. the underlying data in the main_notificationtemplate table is encrypted.
   b. the API doesn't return decrypted data at /api/v2/notification_templates/N/
   c. the unencrypted value is actually what's sent to the web hook destination when the notification ends up being sent

@mabashian does the UI need special code to handle these new parameters, or is it smart enough to just draw the fields based on the fields defined on the template?

[jladdjr]

Added checklist to description so we can mark off progress on the different work items associated with this ticket. Marked 'templated notifications' and 'notifications on job start' as complete. Not sure about the status of the other items.

[jladdjr]

I removed the in_progress label and added needs_review. To my knowledge, nothing is being actively worked on for this ticket. (If I'm wrong about that, please accept my apologies .. and also leave an update 😄)

[ryanpetrello]

@jladdjr @beeankha have we finished everything we agreed on re: notifications in the next release?

If so, @wenottingham maybe we should put this in state:needs_test and track any other work we didn't do as additional issues?

[jladdjr]

@ryanpetrello - I'm still working with @tvo318 on docs, otherwise that's a yes for me

[ryanpetrello]

@wenottingham can you file new distinct issues for these if you still care about tracking them?

[wenottingham]

If they end up being needed, I can; this can be closed when it's done.

[fxfitz]

For what it's worth, we're also interested on an on_changed notification for a playbook summary. Just like someone suggested above, we really want to know when our iptables or our selinux playbooks change on any one of our hosts.

While the playbook doesn't have a "changed", you should be able to see if the playbook changed anything by filtering through the hosts/tasks the changed, no?

[matburt]

For what it's worth, we're also interested on an on_changed notification for a playbook summary.

You can probably get this by just looking at the summary event. I'm not sure if any of this is super relevant here. You can take this event as a webhook and then introspect the summary event to know if the host you were talking to changed in any way.

[Spredzy]

The three features that made this card are now tested and merged.

• Custom message for notification template
• Basic Auth and custom method for webhook notification
• Allow notification on job_start and not only on job_end

As suggested above I am closing this one card, and for any new feature a new card should be open.