How to cancel 'stuck' Builder builds

If for some reason, a build gets 'stuck', where it is not progressing for an extended period of time, it can be canceled by issuing the following command:

hab bldr job cancel <GROUP_ID>

This command may require admin credentials (eg, have WORKER privilege as part of being on the core-maintainers group in Github) if the caller is not a member of the origin for the group.

The group id can be found via a number of ways:

• From the Datadog dashboard
• From Sumologic logs
• From the DB (see steps below)

If you suspect a build group is stuck (symptoms: cannot kick off new builds of that packages), here's a way to check it via the DB.

Log into the 'builder_jobsrv' postgres instance, from the builder datastore node (which is where we are running the Postgres instances), eg:

$ sudo su hab
$ /hab/pkgs/core/postgresql/9.6.3/20180122233153/bin/psql builder_jobsrv
psql (9.6.3, server 9.6.6)
Type "help" for help.

builder_jobsrv=#

Then, you can issue more specific commands:

set search_path to shard_0;
select * from groups order by created_at desc;

Now you should be able to hopefully spot what groups are in a 'Dispatching' state - if you see your package in the group list, then make a note of the group id, and proceed to issue the cancel command.

The below steps should ONLY be run if for some reason the 'hab bldr job cancel' command is not available or working.

select project_name, project_state, updated_at from group_projects where owner_id = '<owner id>'

This should give a list of the projects and their states. If you see a project in a 'InProgress' state, that has not been updated in a while, then that is a suspect for where the stall is.

You can unstick the pipeline by issuing a DELETE on that group, eg:

http DELETE https://bldr.habitat.sh/v1/depot/pkgs/schedule/<group id> Authorization:Bearer:${HAB_AUTH_TOKEN}

Note that the DELETE can only be issued if you have worker privilege in the auth token.