Enable Better Caching for Static Resources #61

trieloff · 2019-04-18T13:52:02Z

provide two new resource types for static resources: .esi to replace a static URL with a cacheable static URL and .hlx_[0-9a-f]+ for resources that are cached forever in the browser

In order to increase the cache efficiency of static resources, we are implementing a multi-stage process which replaces the generic, short and uncacheable URLs of static resources like /scripts/site.js with long and version-specific URLs like /scripts/site.js.hlx_f7c3bc1d808e04732adf679965ccc34ca7ae3441. This is done by offering a virtual resource like /scripts/site.js.esi that can be included in an ESI tag like <script src="<esi:include src="/scripts/site.js.esi" />"> from within HTML, JS or CSS (the replacement of .js and .css file endings iwth .js.esi and .css.esi will be handled in helix-pipeline and helix-static at a later point in time). This virtual resource will then simply replace the short URL /scripts/site.js with the long-cachable URL /scripts/site.js.hlx_f7c3bc1d808e04732adf679965ccc34ca7ae3441 through the ESI resolution process. Any static resource requested with a long-cachable URL will be delivered with Cache-Control headers setting expiry 366 days into the future.

Partial implementation of adobe/helix-pipeline#224 (comment)
Requires adobe/helix-cli#797 and adobe/helix-pipeline#281

…si to replace a static URL with a cacheable static URL and .hlx_[0-9a-f]+ for resources that are cached forever in the browser In order to increase the cache efficiency of static resources, we are implementing a multi-stage process which replaces the generic, short and uncacheable URLs of static resources like `/scripts/site.js` with long and version-specific URLs like `/scripts/site.js.hlx_f7c3bc1d808e04732adf679965ccc34ca7ae3441`. This is done by offering a virtual resource like `/scripts/site.js.esi` that can be included in an ESI tag like `<script src="<esi:include src="/scripts/site.js.esi" />">` from within HTML, JS or CSS (the replacement of `.js` and `.css` file endings iwth `.js.esi` and `.css.esi` will be handled in helix-pipeline and helix-static at a later point in time). This virtual resource will then simply replace the short URL `/scripts/site.js` with the long-cachable URL `/scripts/site.js.hlx_f7c3bc1d808e04732adf679965ccc34ca7ae3441` through the ESI resolution process. Any static resource requested with a long-cachable URL will be delivered with Cache-Control headers setting expiry 366 days into the future. Partial implementation of adobe/helix-pipeline#224 (comment)

codecov-io · 2019-04-18T14:09:32Z

Codecov Report

Merging #61 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master    #61   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files           8      8           
  Lines         182    182           
  Branches       18     18           
=====================================
  Hits          182    182

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e3f85e...5d1f8c7. Read the comment docs.

…ing can be achieved When static is called with the `?esi=true` parameter, then all URL references in CSS files will be replaced with ESI tags that include the `.esi` version of the same resource. This `.esi` version will then resolve, with the help of adobe/helix-publish#61 to a long-cachable URL for the same resource. Partial implementation of adobe/helix-pipeline#267

Same as 396c55b but for JavaScript. For now, only `import` statements are affected depends on adobe/helix-publish#61 and contributes to adobe/helix-pipeline#224 (comment)

trieloff · 2019-04-26T12:08:52Z

You can see the entire thing in action here: https://www.primordialsoup.life/index.html

layouts/fastly/helix.vcl

# [1.1.0](v1.0.0...v1.1.0) (2019-04-29) ### Bug Fixes * **static:** normalize URLs in rewritten CSS and JS ([25964ba](25964ba)) * **static:** turn on ESI processing when ESI flag is set ([8b2436b](8b2436b)) ### Features * **static:** add ESI aliasing support for JavaScript modules ([1fa564d](1fa564d)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** Rewrite CSS URLs to Static ESI URLs so that better caching can be achieved ([396c55b](396c55b)), closes [adobe/helix-publish#61](adobe/helix-publish#61) [adobe/helix-pipeline#267](adobe/helix-pipeline#267)

# [1.1.0](v1.0.0...v1.1.0) (2019-04-29) ### Bug Fixes * **package:** update @adobe/helix-shared to version 0.10.5 ([8efda26](8efda26)) * **static:** normalize URLs in rewritten CSS and JS ([25964ba](25964ba)) * **static:** turn on ESI processing when ESI flag is set ([8b2436b](8b2436b)) ### Features * **static:** add ESI aliasing support for JavaScript modules ([1fa564d](1fa564d)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** Rewrite CSS URLs to Static ESI URLs so that better caching can be achieved ([396c55b](396c55b)), closes [adobe/helix-publish#61](adobe/helix-publish#61) [adobe/helix-pipeline#267](adobe/helix-pipeline#267)

ejthurgo · 2019-04-29T17:12:42Z

Appreciate this is just semantics: but why .esi?
Besides the fact it’s using ESI to achieve it, I’m not sure I associate ESI with what the feature is actually doing. Why not something like .static or .immuatable or similar to signify that you want the "versionless" entity.

trieloff · 2019-04-29T17:33:39Z

@ejthurgo I've been considering .url as the extension, as it contains an URL. Later on, I'd also create a variant of the resource that redirects to the immutable resource and would probably pick the extension .302.

@ejthurgo

…hich returns a redirect As suggested by @ejthurgo in #61 (comment) .esi is a misleading extension for a resource that is not using any ESI, therefore renaming it to .url, which describes the content of the resource better. In addition, introduce a .302 pseudo-resource that simply redirects to the immutable resource

…irects possible

# [2.0.0](v1.0.0...v2.0.0) (2019-05-06) ### Bug Fixes * **package:** update @adobe/helix-shared to version 0.10.5 ([8efda26](8efda26)) * **package:** update @adobe/helix-shared to version 0.11.0 ([0de812b](0de812b)) * **package:** update @adobe/helix-shared to version 1.1.0 ([fabb0ab](fabb0ab)) * **package:** update chokidar to version 3.0.0 ([4688a0f](4688a0f)) * **package:** update dotenv to version 8.0.0 ([0786a94](0786a94)) * **package:** update snyk to version 1.154.1 ([3038ec9](3038ec9)) * **package:** update snyk to version 1.155.0 ([a8bb6d6](a8bb6d6)) * **package:** update snyk to version 1.156.0 ([e321706](e321706)) * **package:** update snyk to version 1.159.0 ([adeb477](adeb477)) * **package:** update snyk to version 1.161.0 ([79be958](79be958)) * **package:** update snyk to version 1.161.1 ([72ad988](72ad988)) * **static:** always return entry path for 404 errors ([84fe876](84fe876)) * **static:** cache error responses for 5 minutes ([4247842](4247842)) * **static:** handle more 404 errors with path ([ed5a887](ed5a887)) * **static:** normalize URLs in rewritten CSS and JS ([25964ba](25964ba)) * **static:** prevent possible XSS by sanitizing output ([97d6387](97d6387)) * **static:** return the original URL in case a static resource cannot get retieved for an ESI include ([3f6b3ff](3f6b3ff)), closes [#813](#813) * **static:** turn on ESI processing when ESI flag is set ([8b2436b](8b2436b)) * **static:** use .url instead of .esi as extension for immutable resources ([70b9674](70b9674)) ### Documentation * **changelog:** explain switchover to version 2.0 ([920b39c](920b39c)) ### Features * **static:** add ESI aliasing support for JavaScript modules ([1fa564d](1fa564d)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** Rewrite CSS URLs to Static ESI URLs so that better caching can be achieved ([396c55b](396c55b)), closes [adobe/helix-publish#61](adobe/helix-publish#61) [adobe/helix-pipeline#267](adobe/helix-pipeline#267) ### BREAKING CHANGES * **changelog:** not breaking any functionality, bumping to version 2.0 as the v1.* name space for git tags has already been polluted

…ing can be achieved When static is called with the `?esi=true` parameter, then all URL references in CSS files will be replaced with ESI tags that include the `.esi` version of the same resource. This `.esi` version will then resolve, with the help of adobe/helix-publish#61 to a long-cachable URL for the same resource. Partial implementation of adobe/helix-pipeline#267

Same as 396c55bf462a2dadd1897a9f05c370b9a6dbfb18 but for JavaScript. For now, only `import` statements are affected depends on adobe/helix-publish#61 and contributes to adobe/helix-pipeline#224 (comment)

# 1.0.0 (2019-05-14) ### Bug Fixes * **static:** always return entry path for 404 errors ([552ef3d](552ef3d)) * **static:** cache error responses for 5 minutes ([3173cea](3173cea)) * **static:** handle more 404 errors with path ([2c1ac18](2c1ac18)) * **static:** normalize URLs in rewritten CSS and JS ([f238adb](f238adb)) * **static:** prevent possible XSS by sanitizing output ([0795581](0795581)) * **static:** return the original URL in case a static resource cannot get retieved for an ESI include ([453c724](453c724)), closes [#813](https://github.com/adobe/helix-static/issues/813) * **static:** turn on ESI processing when ESI flag is set ([a6d96b8](a6d96b8)) * **static:** use .url instead of .esi as extension for immutable resources ([d6c89f8](d6c89f8)) ### Features * **pipeline:** consistently use `context` instead of payload. ([ddea2cb](ddea2cb)), closes [#743](#743) * **static:** add ESI aliasing support for JavaScript modules ([6c0d0d5](6c0d0d5)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** Rewrite CSS URLs to Static ESI URLs so that better caching can be achieved ([75b47e5](75b47e5)), closes [adobe/helix-publish#61](adobe/helix-publish#61) [adobe/helix-pipeline#267](adobe/helix-pipeline#267)

# 1.0.0 (2019-06-24) ### Bug Fixes * **static:** process ESI for static resources, too ([8ea480d](8ea480d)) * .snyk & package.json to reduce vulnerabilities ([d68bf41](d68bf41)) * .snyk & package.json to reduce vulnerabilities ([c2314e5](c2314e5)) * **deploy:** use nodejs:10 container ([b64541e](b64541e)), closes [#55](#55) * **dictionaries:** set missing directory index ([ef56581](ef56581)), closes [#59](#59) * **linting:** linting again... ([b496576](b496576)) * **linting:** linting... ([275494f](275494f)) * **monitoring:** remove epsagon token from source code ([be35e31](be35e31)) * **package:** update @adobe/fastly-native-promises to version 1.10.0 ([258c5eb](258c5eb)) * **package:** update @adobe/fastly-native-promises to version 1.4.0 ([c29ed05](c29ed05)) * **package:** update @adobe/fastly-native-promises to version 1.6.0 ([dba6bd0](dba6bd0)), closes [#17](#17) * **package:** update @adobe/fastly-native-promises to version 1.6.1 ([8b755ae](8b755ae)) * **package:** update @adobe/fastly-native-promises to version 1.7.0 ([8955790](8955790)) * **package:** update @adobe/fastly-native-promises to version 1.8.0 ([23c37eb](23c37eb)) * **package:** update @adobe/fastly-native-promises to version 1.9.0 ([68c16f6](68c16f6)) * **package:** update @adobe/fastly-native-promises to version 1.9.1 ([de465f3](de465f3)) * **package:** update @adobe/helix-shared to version 0.10.3 ([7c9caef](7c9caef)) * **package:** update @adobe/helix-shared to version 0.10.4 ([d4b5e92](d4b5e92)) * **package:** update @adobe/helix-shared to version 0.10.5 ([fd67213](fd67213)) * **package:** update @adobe/helix-shared to version 0.11.0 ([d976bcb](d976bcb)) * **package:** update @adobe/helix-shared to version 0.6.2 ([20703f3](20703f3)) * **package:** update @adobe/helix-shared to version 0.7.0 ([d49ba13](d49ba13)) * **package:** update @adobe/helix-shared to version 0.8.0 ([4731732](4731732)) * **package:** update @adobe/helix-shared to version 0.8.1 ([024f137](024f137)) * **package:** update @adobe/helix-shared to version 0.8.3 ([6dd418b](6dd418b)) * **package:** update @adobe/helix-shared to version 0.8.4 ([feaf8eb](feaf8eb)) * **package:** update @adobe/helix-shared to version 1.0.1 ([ffad201](ffad201)), closes [#67](#67) * **package:** update @adobe/helix-shared to version 1.0.2 ([f3c27b6](f3c27b6)) * **package:** update @adobe/helix-shared to version 1.1.1 ([c1bf275](c1bf275)) * **package:** update @adobe/helix-shared to version 1.2.0 ([41faf50](41faf50)) * **package:** update @adobe/helix-shared to version 1.3.1 ([d002710](d002710)) * **package:** update @adobe/helix-shared to version 1.5.0 ([1a61355](1a61355)) * **package:** update fs-extra to version 8.0.1 ([cffa67c](cffa67c)) * **package:** update glob-to-regexp to version 0.4.1 ([5b0782f](5b0782f)) * **proxy:** Don't attempt to recover from 404 errors in proxy strains ([8b84690](8b84690)), closes [#75](#75) * **proxy:** prevent duplicate creation of backends for proxy strains ([c157e8b](c157e8b)) * **proxy:** use FastlyJSON when talking to Fastly API ([9d40b93](9d40b93)) * **static:** add missing slash to GitHub raw URL ([c07f401](c07f401)) * **static:** disable GZip for all backend requests that could contain ESI ([393ae0a](393ae0a)) * **static:** don't redeclare var ([3f23556](3f23556)) * **static:** drop accept-encoding to enforce esi ([2729f64](2729f64)) * **static:** enable compression for ESI-processed resources ([0eaf027](0eaf027)) * **static:** fix syntax error ([852969a](852969a)) * **static:** fix syntax error ([82949b0](82949b0)) * **static:** fix syntax error ([d22c465](d22c465)) * **static:** fix variable name ([cbb125e](cbb125e)) * **static:** fix VCL syntax error ([db4c524](db4c524)) * **static:** fix VCL syntax error ([76749d6](76749d6)) * **static:** handle immutable requests without restart ([79c0ba7](79c0ba7)) * **static:** move esi handling into fetch ([46bc081](46bc081)) * **static:** never use GZip when talking with Runtime, as there might be ESI ([21ce4a1](21ce4a1)) * **static:** no slash between ref and path in static retrieval ([209cd7f](209cd7f)) * **static:** protect against cache poisoning by verifying the etag ([1bc1e24](1bc1e24)) * **static:** remove brackets in URL ([4009ff5](4009ff5)) * **static:** restrict regex for immutable resources ([c71043f](c71043f)) * **static:** strip 302 from URL when requesting a redirect ([fd5a5db](fd5a5db)) * **static:** submit location headers for redirects, making actual redirects possible ([615bf33](615bf33)) * **static:** throw error for invalid resource paths ([92d4c82](92d4c82)) * **static:** turn on ESI processing when backend response requests it ([5459d30](5459d30)) * **static:** unset Accept-Encoding more aggressively ([df42f20](df42f20)) * **static:** use beresp instead of resp for ETag value ([112796d](112796d)) * **static:** use correct location header ([5d1f8c7](5d1f8c7)) * **static:** use correct path for entry for immutable resources ([c66f15f](c66f15f)) * **static:** use local var for new extension ([c995533](c995533)) * **static:** use re instead of req ([245b583](245b583)) * **static:** use req for tracing instead of resp ([806d48d](806d48d)) * **vcl:** bum version ([73d3f8e](73d3f8e)) * **vcl:** copyright & unrelated change ([e88ed93](e88ed93)) * **vcl:** include must be outside of a subroutine ([cef774f](cef774f)) * **vcl:** is line break required ? ([67080b2](67080b2)) * **vcl:** is linebreak required? ([9cc1ea2](9cc1ea2)) * **vcl:** remove unnecessary line breaks ([8502341](8502341)) * **vcl:** respect remapped url also in static ([f404c4d](f404c4d)), closes [#101](#101) ### Features * **monitoring:** add epsagon wrapper ([2d52627](2d52627)), closes [#104](#104) * **proxy:** make proxy strains respect the URL path and backend path ([ff77ba7](ff77ba7)) * **static:** enable asset statification for assets loaded from static URLs ([befc1e2](befc1e2)) * **static:** provide two new resource types for static resources: .esi to replace a static URL with a cacheable static URL and .hlx_[0-9a-f]+ for resources that are cached forever in the browser ([9f8e52c](9f8e52c)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** rename static-esi to static-url, introduce static-302 which returns a redirect ([403ae1b](403ae1b)), closes [/github.com//pull/61#issuecomment-487664612](https://github.com//github.com/adobe/helix-publish/pull/61/issues/issuecomment-487664612) * **vcl:** Add extensions.vcl ([ae1b52c](ae1b52c)) * **vcl:** add more extension points ([979cfbe](979cfbe)) * **vcl:** Add more extension points ([3dee07e](3dee07e)) * **vcl:** Documentation ([902f6f3](902f6f3)) * **vcl:** Support Custom VCL ([b2ca735](b2ca735)) * **vcl:** testing extension logic ([717c192](717c192)) * **vcl:** Tests for synthetize method ([71bcd2d](71bcd2d)) ### Performance Improvements * **static:** declare static files to be immutable ([f0369b7](f0369b7)) ### BREAKING CHANGES * **proxy:** changes root paths of all proxy strains - fixes #64

adobe-bot · 2019-06-24T11:57:43Z

🎉 This PR is included in version 1.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

trieloff added 5 commits April 18, 2019 13:47

fix(static): fix syntax error

852969a

fix(static): fix VCL syntax error

76749d6

fix(static): fix variable name

cbb125e

fix(static): use local var for new extension

c995533

This was referenced Apr 20, 2019

Rewrite Static Asset References? adobe/helix-pipeline#224

Closed

Rewrite Static Asset References to ESI adobe/helix-pipeline#281

Merged

Use Static References in Static CSS and JS code adobe/helix-cli#797

Merged

perf(static): declare static files to be immutable

f0369b7

trieloff marked this pull request as ready for review April 24, 2019 12:23

trieloff added 17 commits April 25, 2019 09:49

refactor(static): improve tracing for static esi

91de16d

fix(static): no slash between ref and path in static retrieval

209cd7f

fix(static): add missing slash to GitHub raw URL

c07f401

refactor(static): more tracing

d7d4916

refactor(static): more tracing

34c5efe

refactor(static): handle ESI changes first

7846b80

fix(static): remove brackets in URL

4009ff5

fix(static): move esi handling into fetch

46bc081

fix(static): use beresp instead of resp for ETag value

112796d

refactor(static): flip static and header delivery

611cd06

refactor(static): more verbose tracing in deliver static

ffd6403

fix(static): fix syntax error

82949b0

refactor(static): more tracing for static esi content

14649d1

refactor(static): using an error handler for synthetic response

ad45256

refactor(static): move esi logic into fetch

3d24575

fix(static): use req for tracing instead of resp

806d48d

fix(static): use correct path for entry for immutable resources

c66f15f

trieloff added 7 commits April 26, 2019 07:39

refactor(static): trace accept encoding after esi processing

2453022

fix(static): unset Accept-Encoding more aggressively

df42f20

refactor(static): trace more, esi more

09fecb0

fix(static): fix syntax error

d22c465

fix(static): handle immutable requests without restart

79c0ba7

fix(static): restrict regex for immutable resources

c71043f

fix(static): drop accept-encoding to enforce esi

2729f64

trieloff requested review from stefan-guggisberg and ejthurgo April 26, 2019 12:08

trieloff commented Apr 26, 2019

View reviewed changes

layouts/fastly/helix.vcl Show resolved Hide resolved

trieloff and others added 2 commits April 29, 2019 09:29

fix(static): enable compression for ESI-processed resources

0eaf027

fix(static): fix VCL syntax error

db4c524

trieloff added 4 commits April 30, 2019 09:03

fix(static): strip 302 from URL when requesting a redirect

fd5a5db

fix(static): submit location headers for redirects, making actual red…

615bf33

…irects possible

fix(static): use correct location header

5d1f8c7

trieloff merged commit c6892b8 into master Apr 30, 2019

adobe-bot added the released label Jun 24, 2019

tripodsan deleted the static-esi branch July 7, 2019 05:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Better Caching for Static Resources #61

Enable Better Caching for Static Resources #61

trieloff commented Apr 18, 2019 •

edited

Loading

codecov-io commented Apr 18, 2019 •

edited

Loading

trieloff commented Apr 26, 2019

ejthurgo commented Apr 29, 2019

trieloff commented Apr 29, 2019

adobe-bot commented Jun 24, 2019

Enable Better Caching for Static Resources #61

Enable Better Caching for Static Resources #61

Conversation

trieloff commented Apr 18, 2019 • edited Loading

codecov-io commented Apr 18, 2019 • edited Loading

Codecov Report

trieloff commented Apr 26, 2019

ejthurgo commented Apr 29, 2019

trieloff commented Apr 29, 2019

adobe-bot commented Jun 24, 2019

trieloff commented Apr 18, 2019 •

edited

Loading

codecov-io commented Apr 18, 2019 •

edited

Loading