Bug 1655477 Support X-Source-Tags header

[jklukas]

Also see pipeline support in mozilla/gcp-ingestion#1328

Checklist for reviewer:

• Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
• Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
• If the PR comes from a fork, trigger the integration CI test by pushing this revision as discussed in the README and review the report posted in the comments.

For glean changes:

• Update include/glean/CHANGELOG.md

[dataops-ci-bot]

Integration report for "Bug 1655477 Support X-Source-Tags header"

Report for upstream
Report for latest commit

689d81b-a5ba26a.diff

No content detected.

bq_schema_689d81b-a5ba26a.diff

Click to expand!

diff integration/689d81b/metadata.decoded.1.bq integration/a5ba26a/metadata.decoded.1.bq
92a93,98
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.pioneer-decoded.1.bq integration/a5ba26a/metadata.pioneer-decoded.1.bq
92a93,98
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.pioneer-ingestion.1.bq integration/a5ba26a/metadata.pioneer-ingestion.1.bq
78a79,84
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.structured-ingestion.1.bq integration/a5ba26a/metadata.structured-ingestion.1.bq
78a79,84
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.telemetry-ingestion.1.bq integration/a5ba26a/metadata.telemetry-ingestion.1.bq
78a79,84
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"

[dataops-ci-bot]

Integration report for "Add line to Glean changelog"

Report for upstream
Report for latest commit

689d81b-500ccdd.diff

No content detected.

bq_schema_689d81b-500ccdd.diff

Click to expand!

diff integration/689d81b/metadata.decoded.1.bq integration/500ccdd/metadata.decoded.1.bq
92a93,98
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.pioneer-decoded.1.bq integration/500ccdd/metadata.pioneer-decoded.1.bq
92a93,98
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.pioneer-ingestion.1.bq integration/500ccdd/metadata.pioneer-ingestion.1.bq
78a79,84
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.structured-ingestion.1.bq integration/500ccdd/metadata.structured-ingestion.1.bq
78a79,84
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.telemetry-ingestion.1.bq integration/500ccdd/metadata.telemetry-ingestion.1.bq
78a79,84
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"

[whd]

Most likely we also want this in https://github.com/mozilla-services/mozilla-pipeline-schemas/blob/master/schemas/metadata/error/error.1.schema.json (and probably pioneer-error as well). I don't remember if any of this requires changes to ingestion-sink.

[dataops-ci-bot]

Integration report for "Add x_source_tags to raw schema include"

Report for upstream
Report for latest commit

689d81b-6a88b00.diff

No content detected.

bq_schema_689d81b-6a88b00.diff

Click to expand!

diff integration/689d81b/metadata.decoded.1.bq integration/6a88b00/metadata.decoded.1.bq
92a93,98
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.error.1.bq integration/6a88b00/metadata.error.1.bq
221a222,227
>   },
>   {
>     "description": "X-Source-Tags header",
>     "mode": "NULLABLE",
>     "name": "x_source_tags",
>     "type": "STRING"
diff integration/689d81b/metadata.pioneer-decoded.1.bq integration/6a88b00/metadata.pioneer-decoded.1.bq
92a93,98
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.pioneer-error.1.bq integration/6a88b00/metadata.pioneer-error.1.bq
233a234,239
>   },
>   {
>     "description": "X-Source-Tags header",
>     "mode": "NULLABLE",
>     "name": "x_source_tags",
>     "type": "STRING"
diff integration/689d81b/metadata.pioneer-ingestion.1.bq integration/6a88b00/metadata.pioneer-ingestion.1.bq
78a79,84
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.raw.1.bq integration/6a88b00/metadata.raw.1.bq
96a97,102
>   },
>   {
>     "description": "X-Source-Tags header",
>     "mode": "NULLABLE",
>     "name": "x_source_tags",
>     "type": "STRING"
diff integration/689d81b/metadata.structured-ingestion.1.bq integration/6a88b00/metadata.structured-ingestion.1.bq
78a79,84
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"
diff integration/689d81b/metadata.telemetry-ingestion.1.bq integration/6a88b00/metadata.telemetry-ingestion.1.bq
78a79,84
>           },
>           {
>             "description": "X-Source-Tags HTTP header",
>             "mode": "NULLABLE",
>             "name": "x_source_tags",
>             "type": "STRING"

[jklukas]

Most likely we also want this in https://github.com/mozilla-services/mozilla-pipeline-schemas/blob/master/schemas/metadata/error/error.1.schema.json (and probably pioneer-error as well).

Added for error, pioneer-error, and raw.

I don't remember if any of this requires changes to ingestion-sink.

No, for raw and error, we directly jsonify the message's AttributeMap and let the existence of appropriate destination fields in BQ determine what attributes are dropped.

[whd]

LGTM