Stable all.requests

[max-ostapenko]

Schema changes

• summary column trimmed of a number of metrics (Closes The new schema and cost concerns for users data-pipeline#149)
• payload column trimmed of headers
• rank data added and used for clustering (Resolves Add rank field to all.requests data-pipeline#189, Closes Consider clustering all.requests table by page or rank  data-pipeline#263)
• payload and summary columns are of JSON type

Intermediate table

all.requests_stable

Run reprocessing jobs with all_requests_stable tag.

After reprocessing

DROP TABLE `all.requests`;

CREATE TABLE `all.requests`
COPY `all_dev.requests_stable`;

[max-ostapenko]

Test run

Original and processed table

[max-ostapenko]

Would really appreciate review here, as I'm almost ready with backfill PR #10.

[tunetheweb]

Would really appreciate review here, as I'm almost ready with backfill PR #10.

Hey @max-ostapenko sorry for radio silence here. I mistakenly presumed this was discussed on the last HTTP Archive maintainers call (that I missed) but just found out that @rviscomi missed this too so think we both presumed the other was dealing with this.

I think it would be good to discuss this on a call (whether the next planned one on 15th October or before). Questions/concerns I have include:

• Is this definitely all the changes we plan?
• What happens while this is running? Is the table not accessible? Or will it continue to be accessible from the run as each data is back-populated?
• How long it will take to run?
• Any impact on the next crawl (assuming it's not complete by then)?
• Any impact on Web Almanac and if we're better to wait until majority of analysis is completed there?

[max-ostapenko]

I think it would be good to discuss this on a call (whether the next planned one on 15th October or before). Questions/concerns I have include:

I would definitely like to discuss it before the next crawl. And notify users of the upcoming changes beforehand.

• Is this definitely all the changes we plan?

Please drop here any issues or comments that I've missed.

• What happens while this is running? Is the table not accessible? Or will it continue to be accessible from the run as each data is back-populated?

The reprocessing will write into all_dev.requests_stable.

Once the reprocessing is done we can:

• run checks on the new table,
• test all the production queries changes,
• notify users with the exact date,
• update Almanac queries from this and last years,

and eventually replace the live table with CREATE TABLE COPY DDL. So no downtime.

• How long it will take to run?

We can only estimate by running the first month.

In September 2024 it took 6h for requests tables without any transformations.
Let's assume it would take 2 times longer for this one - 12h.
So ~14 days.

• Any impact on the next crawl (assuming it's not complete by then)?

Ideally we start this before the new crawl starts (so that we can already have it synced with new schema and cleaned objects to just run INSERT INTO SELECT *).

If anything not yet synced in WPT agent before the crawl, we can always cleanup the last crawl data within the pipeline.

• Any impact on Web Almanac and if we're better to wait until majority of analysis is completed there?

Queries reading from struct columns need to be adjusted to a new schema to be valid.
All the analysis PRs are still pending, I'll need to go through them, and push the review where the articles were written.
Then we merge adjustments to the published queries at the moment we replace the live table. It will give a smooth experience for readers.

[tunetheweb]

We could pull _font_details stuff into it's own column? However I think it's niche enough (and the payload will be small enough with above changes) that it's OK to leave in the payload column. WDYT @rviscomi ?

    "_font_details": {
        "table_sizes": {
            "GDEF": 100,
            "GPOS": 3264,
            "GSUB": 458,
            "OS/2": 96,
...
        "counts": {
            "num_cmap_codepoints": 215,
            "num_glyphs": 238
        }
    },

[tunetheweb]

We need to look at native JSON columns as there were some that couldn't be processed in that and so had to use JavaScript JSON columns. See: HTTPArchive/httparchive.org#923 (comment). Is this going to be a problem if we move to native JSON columns?

[tunetheweb]

Ignore. SAFE.PARSE_JSON(payload, wide_number_mode => 'round') is the answer as you already implemented.

[tunetheweb]

LGTM with one small request to leave the crawlids and pagesid in summary.

[rviscomi]

We could pull _font_details stuff into it's own column? However I think it's niche enough (and the payload will be small enough with above changes) that it's OK to leave in the payload column. WDYT @rviscomi ?

Agreed. It could work, but I'm happy to leave it as is.