added redirect to pull non-existant local assets from live_endpoint url

[sjuerges]

This PR implements #69

It achieves that the nginx can serve local assets as before, but it will send HTTP 302 redirects to the standard live_endpoint url for files that are NOT found locally.

[antonym]

Seemed to work, but I notice it seems to have issues with the assets that contain a part2 file. It only downloads the first part if not found locally but fails on the second part.

[sjuerges]

Weird, wasnt an issue for me, specifically tested for it:
(local):

172.18.0.5 - - [03/Aug/2024:22:02:31 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f/vmlinuz HTTP/1.1" 200 12881792 "-" "iPXE/1.21.1+ (g5a9f4)"                                                                            
172.18.0.5 - - [03/Aug/2024:22:02:32 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f/initrd HTTP/1.1" 200 104039372 "-" "iPXE/1.21.1+ (g5a9f4)"                                                                            
172.18.0.5 - - [03/Aug/2024:22:02:34 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f//airootfs.sfs HTTP/1.1" 206 1 "-" "curl/8.4.0"                                                                                        
172.18.0.5 - - [03/Aug/2024:22:02:52 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f//airootfs.sfs HTTP/1.1" 200 2097151999 "-" "curl/8.4.0"                                                                               
172.18.0.5 - - [03/Aug/2024:22:02:53 +0000] "HEAD /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f//airootfs.sfs.part2 HTTP/1.1" 200 0 "-" "curl/8.4.0"                                                                                 
172.18.0.5 - - [03/Aug/2024:22:02:55 +0000] "GET /asset-mirror/releases/download/Galileo-11-2023-e0c48f1f//airootfs.sfs.part2 HTTP/1.1" 200 258531329 "-" "curl/8.4.0"  

(remote):

172.18.0.5 - - [03/Aug/2024:21:55:07 +0000] "GET /ubuntu-squash/releases/download/1.2-e7db33e3/vmlinuz HTTP/1.1" 302 145 "-" "iPXE/1.21.1+ (g5a9f4)"                                                                                            
172.18.0.5 - - [03/Aug/2024:21:55:09 +0000] "GET /ubuntu-squash/releases/download/1.2-e7db33e3/initrd HTTP/1.1" 302 145 "-" "iPXE/1.21.1+ (g5a9f4)"                                                                                             
172.18.0.5 - - [03/Aug/2024:21:55:19 +0000] "HEAD /ubuntu-squash/releases/download/1.2-e7db33e3/filesystem.squashfs HTTP/1.1" 302 0 "-" "curl/7.67.0"                                                                                           
172.18.0.5 - - [03/Aug/2024:21:55:19 +0000] "GET /ubuntu-squash/releases/download/1.2-e7db33e3/filesystem.squashfs HTTP/1.1" 302 145 "-" "curl/7.67.0"                                                                                          
172.18.0.5 - - [03/Aug/2024:21:57:05 +0000] "HEAD /ubuntu-squash/releases/download/1.2-e7db33e3/filesystem.squashfs.part2 HTTP/1.1" 302 0 "-" "curl/7.67.0"                                                                                     
172.18.0.5 - - [03/Aug/2024:21:57:05 +0000] "GET /ubuntu-squash/releases/download/1.2-e7db33e3/filesystem.squashfs.part2 HTTP/1.1" 302 145 "-" "curl/7.67.0" 

@antonym Which image did you test with, maybe I can replicate/debug?

[nrvale0]

Related but tangential. Hope that's ok...

If I wanted to test this using netboot.xyz running in Docker Compose would it be sufficient to just pull the branch and then use the docker-compose.yml.example? I suspect I might also have to modify the image name also? Or just do a docker build of the default Dockerfile + tags but while this branch is selected in order to pull the modified default file?

EDIT: Yes, changing to the PR branch and rebuilding the image with the same tag/name worked.

[jakelmg]

I tried this out and it worked fine. Any word on pulling this? @antonym

[antonym]

@sjuerges

Tested with the Ubuntu 22.04 Image (Gnome), with endpoint set to local machine and cache downloaded:

10.0.100.148 - - [30/Dec/2024:18:35:05 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/vmlinuz HTTP/1.1" 200 14928264 "-" "iPXE/1.21.1+ (g8fc11)"
10.0.100.148 - - [30/Dec/2024:18:35:08 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/initrd HTTP/1.1" 200 129283057 "-" "iPXE/1.21.1+ (g8fc11)"
10.0.100.148 - - [30/Dec/2024:18:35:11 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs HTTP/1.1" 200 0 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:35:13 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs HTTP/1.1" 200 2097151999 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:35:13 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part2 HTTP/1.1" 200 0 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:35:14 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part2 HTTP/1.1" 200 848650241 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:35:14 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part3 HTTP/1.1" 302 0 "-" "curl/8.4.0"

Removed cache, and attempted, dumps to busy box after the first image pull, doesn't seem to handle the redirect for part2 or part3 checks correctly:

10.0.100.148 - - [30/Dec/2024:18:41:19 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/vmlinuz HTTP/1.1" 302 145 "-" "iPXE/1.21.1+ (g8fc11)"
10.0.100.148 - - [30/Dec/2024:18:41:21 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/initrd HTTP/1.1" 302 145 "-" "iPXE/1.21.1+ (g8fc11)"
10.0.100.148 - - [30/Dec/2024:18:41:28 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs HTTP/1.1" 302 0 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:41:28 +0000] "GET /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs HTTP/1.1" 302 145 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:42:17 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part2 HTTP/1.1" 302 0 "-" "curl/8.4.0"
10.0.100.148 - - [30/Dec/2024:18:42:17 +0000] "HEAD /ubuntu-squash/releases/download/22.04.5-b0159fca/filesystem.squashfs.part3 HTTP/1.1" 302 0 "-" "curl/8.4.0"

After it pulls the first squashfs part, it dumps to busy box.

Also noticed that the generation of the config file was squashing the variables with envsubst which I'm going to fix as well.

[antonym]

Fix for the vars in this commit getting squashed during new deploy: #78