Backfills & deprecating legacy tables

[max-ostapenko]

Closes HTTPArchive/data-pipeline#26
Closes HTTPArchive/data-pipeline#138

Deprecation plan

---

all.requests
2011-06 - 2015-12 definitions/output/all/backfill_summary_requests.js
2016-01 - 2022-02 definitions/output/all/backfill_requests.js

Sources used:

• requests
  1st day of month: 2016-01-01 till 2022-06-01
  15th day of month: 2016-01-15 till 2018-12-15

• response_bodies
  1st day of month: 2016-01-01 till 2020-11-01(mobile)
  15th day of month: 2016-01-15 till 2018-12-15

• summary_requests
  1st day of month: 2011-06-01 till 2015-12-01
  15th day of month: 2011-06-15 till 2015-12-15

all.pages

• 2016-01 - 2022-02 definitions/output/all/backfill_pages.js
• 2011-06 - 2015-12 definitions/output/all/backfill_summary_pages.js

Sources used:

• summary_pages
  1st day of month: 2011-06-01 till 2015-12-01
  15th day of month: 2011-06-15 till 2015-12-15

• pages
  1st day of month: 2016-01-01 till 2022-06-01
  15th day of month: 2016-01-15 till 2018-12-15

Summary schema evolution:

SELECT 
  REGEXP_REPLACE(ddl, 'CREATE TABLE .+\n', '') AS schema,
  MIN(table_name) AS min,
  MAX(table_name) AS max,
  ARRAY_AGG(DISTINCT REGEXP_EXTRACT(table_name, r'\d+_\d+_\d+') ORDER BY REGEXP_EXTRACT(table_name, r'\d+_\d+_\d+') ASC),
  COUNT(1)
FROM summary_requests.INFORMATION_SCHEMA.TABLES
GROUP BY schema
ORDER BY min ASC

Known issues

Missing partitions/clusters

• pages: 2016-03-15(mobile)
• summary_pages: 2013-07-15, 2013-12-01, 2014-06-15(mobile), 2015-06-01(mobile), 2015-09-01(mobile)
• summary_requests: 2013-12-01, 2013-07-15, 2014-06-15(mobile), 2015-06-01(mobile), 2015-09-01(mobile)
• response_bodies: from 2020-11-01(desktop) till 2022-06-01(desktop)

Wherever the individual tables are missing the queries will fail and I'll need to re-run manually without a JOIN.

More issues related to historical data:

• Missing recent summary_requests tables httparchive.org#122
• Desktop Summary Requests incomplete httparchive.org#135
• Bigquery: Add missing response bodies data for 2020 November-2021 June (Desktop and Mobile) httparchive.org#592

[max-ostapenko]

As we can't join requests and summary_requests tables I've recalculated the summary metrics using the UDF.

[max-ostapenko]

@tunetheweb thanks to a few hints in the comments I think I've resolved all the missing metrics for the backfill.

And added new dataset to the crawl pipeline as a new default. All the reports are dependents of crawl. The legacy tables + all dataset will be updated in parallel.

all.parsed_css will be a snapshot of crawl.parsed_css to avoid duplication.