Hotfix prod: Index dcp42 in prod and make dcp41 default (#6556, #6558)

[achave11-ucsc]

Connected issue: #6558, #6556

Notes

• Manually deindex the dcp40 and dcp40-it catalogs from the prod deployment
• Perform a targeted index of the dcp42 catalog

Checklist

Author

• Target branch is prod
• Name of PR branch matches hotfixes/<GitHub handle of author>/<issue#>-<slug>-prod
• On ZenHub, PR is connected to the issue it hotfixes
• PR description links to connected issue
• PR title is Hotfix prod: followed by title of connected issue
• PR title references the connected issue

Author (hotfixes)

• Added h tag to commit title _{or this PR does not include a temporary hotfix}
• Added H tag to commit title _{or this PR does not include a permanent hotfix}
• Added hotfix label to PR
• This PR is labeled partial _{or represents a permanent hotfix}

Author (before every review)

• Rebased PR branch on prod, squashed old fixups
• Ran make requirements_update _{or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile}
• Added R tag to commit title _{or this PR does not modify requirements*.txt}
• This PR is labeled reqs _{or does not modify requirements*.txt}

System administrator (after approval)

• Actually approved the PR
• Labeled PR as no sandbox
• A comment to this PR details the completed security design review
• PR title is appropriate as title of merge commit
• Moved connected issue to Approved column
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Squashed PR branch and rebased onto prod
• Sanity-checked history
• Pushed PR branch to GitHub
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but excluded any p tags}
• Moved connected issue to Merged stable column in ZenHub
• Pushed merge commit to GitHub

Operator (after pushing the merge commit)

• Pushed merge commit to GitLab prod
• Build passes on GitLab prod
• Reviewed build logs for anomalies on GitLab prod
• Deleted PR branch from GitHub

Operator (reindex)

• Deindexed all unreferenced catalogs in prod _{or this PR is neither labeled reindex:partial nor reindex:prod}
• Deindexed specific sources in prod _{or this PR is neither labeled reindex:partial nor reindex:prod}
• Indexed specific sources in prod _{or this PR is neither labeled reindex:partial nor reindex:prod}
• Started reindex in prod _{or neither this PR nor a failed, prior promotion requires it}
• Checked for, triaged and possibly requeued messages in both fail queues in prod _{or neither this PR nor a failed, prior promotion requires it}
• Emptied fail queues in prod _{or neither this PR nor a failed, prior promotion requires it}
• Created backport PR and linked to it in a comment on this PR

Operator

• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[coveralls]

coverage: 85.399%. remained the same
when pulling 5a23ace on hotfixes/achave11-ucsc/6558-dcp42-catalog-prod
into c7475dc on prod.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.38%. Comparing base (c7475dc) to head (5a23ace).
Report is 3 commits behind head on prod.

Additional details and impacted files

@@           Coverage Diff           @@
##             prod    #6567   +/-   ##
=======================================
  Coverage   85.38%   85.38%           
=======================================
  Files         155      155           
  Lines       20735    20735           
=======================================
  Hits        17704    17704           
  Misses       3031     3031           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hannes-ucsc]

This is over-indented.

[hannes-ucsc]

One of the commits lacked the H tag. I added it and force-pushed.

[hannes-ucsc]

Security design review

• Security design review completed; this PR does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with Terra
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary Hotfix prod: Index dcp42 in prod and make dcp41 default (#6556, #6558) #6567 (comment)
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below

[hannes-ucsc]

Security design review: This change adds an managed-access snapshot for HCA production as requested and approved by HCA stakeholders. Only Google identities that are registered with Terra and that have been given access to that snapshot by the Terra team will be able to access Azul's copy of the metadata from that snapshot.

[achave11-ucsc]

Backport PR.