cdn.dl.k8s.io doesn't seem to be cutting bandwidth to the backing GCS bucket

[BenTheElder]

I suspect the objects are too large for the current cache configuration? Not an expert with fastly.

cc @dims @ameukam

We don't have super good visibility on the bucket end of things just because I don't know anyone that currently has e.g. audit log access, but I can provide bandwidth graphs (there's a lot of noise) and we have graphs on the fastly side.

Best I can tell looking at the fastly UI and the GCS bandwidth graphs .... we're basically operating in pull-through mode w/o caching.

We'll need to fix this before we can rotate to a kubernetes.io GCS bucket (or S3 on AWS or ...).

[BenTheElder]

Concretely: I think we need to improve our fastly configuration to ensure binaries are actually cached. There was a slack thread suggesting the default is <20MB but that would exclude most or all of kubernetes core binaries ...

[BenTheElder]

For context: #5603 merged July 24th.

This may be working better than we thought, but we should still be confirming if we have suitable caching parameters etc.

requests:

sent bytes:

[BenTheElder]

I think we should replace this with new issues about revisiting the config, the bandwidth is definitely down.

@xmudrii mentioned a problem with slow downloads, but I don't think we have a tracking issue yet.

[xmudrii]

I created #5755 to track the problem with slow downloads.

[xmudrii]

I created #5757 to track increasing cache TTL, this might further reduce bandwidth to the origin bucket.