Git LFS Client Worker

This middleware allows Cloudflare Pages sites built from a Git repository to serve large files directly from the repository's Git LFS server, transparently overcoming the 25 MiB Pages file size limit.

Usage

We'll assume you already have a Git repository to publish on Cloudflare Pages, and your working directory is the root of your repository:

cd "$(git rev-parse --show-toplevel)"

Install Git LFS

If you haven't used Git LFS before, you may need to install it. Run the following command:

git lfs version

If your output includes git: 'lfs' is not a git command, then follow the Git LFS installation instructions.

Install smudge and clean filters

Even if the Git LFS binary was already installed, the smudge and clean filters Git LFS relies upon may not be. Ensure they are installed for your user account:

git lfs install

Install the LFS Client Worker

The LFS Client Worker works like the standard Git LFS client: when your Pages site is about to serve an LFS "pointer" file, it looks up the LFS object it points to and serves it instead. To do this, the LFS Client Worker needs to be run as "middleware" on your site, which means _middleware.js from this repo should end up as functions/_middleware.js in your repo. It also needs a symlink from functions/.lfsconfig.txt to .lfsconfig to load .lfsconfig correctly. If you don't have other functions on your Pages site (i.e. there is no functions directory in the root of your project), the simplest way to install LFS Client Worker this is to add this repo to your site as a submodule called functions:

git submodule add https://github.com/milkey-mouse/git-lfs-client-worker functions
git commit -m "Add Git LFS Client Worker"

Disable smudge and clean filters by default

The worker will replace LFS "pointers" with the underlying objects on the fly, so we don't want the standard Git LFS client to do so when the repo is cloned by Cloudflare Pages. (Otherwise, Cloudflare Pages builds will fail even when LFS stores all large files.) Environment variables don't seem to influence the "Cloning git repository" step in Pages builds, so smudge and clean filters must be disabled by default in .lfsconfig:

git config -f .lfsconfig lfs.fetchexclude '*'
git add .lfsconfig
git commit -m "Disable LFS fetching by default"

When a commit including this .lfsconfig change is checked out, Git LFS will not replace LFS pointers with the objects they point to. On your own copies of the repo, override lfs.fetchexclude to continue doing so:

git config lfs.fetchexclude ''

Change LFS servers

GitHub and GitLab both provide default LFS servers, but they are not ideal for this use case:

• Storage space is limited.
  • You can store up to 10 GiB of LFS objects across all repositories on GitHub's free tier. Additional storage costs $0.07/GiB-month.
  • You can store up to 5 GB on GitLab's free tier, but this includes all other GitLab services. Additional storage must be bought in $60/year "packs".
• Bandwidth is limited.
  • You can download up to 10 GiB/month of LFS objects across all repositories on GitHub's free tier. Additional bandwidth costs $0.0875/GiB.
  • You can download up to 10 GB/month on GitLab's free tier, but this includes all other GitLab services. Additional bandwidth must be bought in $60/year "packs".
• Latency is questionable.
  • GitHub's LFS server takes ~270ms to reply with an object's URL. The actual object store takes another 20-100ms to reply.
  • Neither GitHub nor GitLab built their LFS servers with serving web content in mind (which, to be fair, is slightly cursed).

Consider setting up LFS S3 Proxy with R2 as your LFS server instead. On Cloudflare's free tier, it can serve up to 10 GB of LFS objects with unlimited bandwidth and the lowest possible latency (your objects are in the same datacenters as the LFS Client Worker). If you have more than 10 GB of assets, additional storage is $0.015/GB-month, several times cheaper than GitHub or GitLab.

If you decide to use the default LFS server, you'll still want to explicitly specify its URL in .lfsconfig as the LFS Client Worker has no other way of knowing it (workers don't know what repo their Pages site was built from).

Add files to Git LFS

You're now ready to start using Git LFS. At the very least, you should add all files larger than the 25 MiB Cloudflare Pages limit to Git LFS:

find . -type f '!' -path './.*' -size +25M -exec git lfs track {} +
find . -type f '!' -path './.*' -size +25M -exec git add --renormalize {} +
git add .gitattributes
git commit -m "Add files over 25 MiB to Git LFS"

If you add more large files in the future, add them to Git LFS before you commit them with git lfs track:

git lfs track bigfile

If certain file types are consistently larger than 25 MiB, you can automatically track them with Git LFS:

git lfs track '*.mp4'

Create a Pages site

If you haven't already, create a Cloudflare Pages site from your repo. If you have, push your changes to trigger a rebuild of your site:

git push

Optional: Bind your R2 bucket to LFS Client Worker

Once your Pages site is set up, if you changed your LFS server to LFS S3 Proxy with R2, you should add a binding to your Pages site for slightly better performance:

• Open the Workers & Pages section of the Cloudflare dashboard.
• Select your Pages site.
• Set up LFS_BUCKET:
  • Navigate to Settings > Functions > R2 bucket bindings > Production.
  • Click Add binding.
  • Set Variable name to LFS_BUCKET.
  • For R2 bucket, select the bucket you created for LFS S3 Proxy.
  • Click Save.
• Set up LFS_BUCKET_URL:
  • Navigate to Settings > Environment variables > Production.
  • click Add variables.
  • Set Variable name to LFS_BUCKET_URL.
  • Set Value to your LFS server URL without the access key (just https://<INSTANCE>/<ENDPOINT>/<BUCKET>).
• Re-deploy your Pages site:
  • Navigate to Deployments > Production > View details.
  • Click Manage deployment > Retry deployment.

With these variables set, LFS Client Worker can skip asking LFS S3 Proxy for presigned URLs and fetch objects directly from the bucket.