use cache busting for KP bundles #64414

mshustov · 2020-04-24T10:56:56Z

Summary

This PR changes the cache invalidation policy for the platform + plugins bundles in distributable version of Kibana.
Instead of etag usage which leads to a round-trip per an asset, we use cache busting pattern, with a build number included in URL. (it looks like http://localhost:5601/bundles/8467/plugin/dashboard/dashboard.plugin.js)
Kibana server always uses the build number in an asset path to have similar URLs in development and distributable version. The difference exists in cache-control headers:

Cache-control: max-age=31536000 for distributable version
etag: ... & Cache-control: must-revalidate for non-distributable version

Some numbers:

for http://localhost:5601/app/kibana#/home loaded locally
Loading time for a page with assets cached via etag: 8.61 sec.
Loading time for a page with assets cached via cache-busting: 8.08 sec. (-8%)
NP app http://localhost:5601/app/logs/stream loaded locally
Loading time for a page with assets cached via etag: 5.77 sec.
Loading time for a page with assets cached via cache-busting: 5.25 sec. (-10%)

In the case of real-world client-server communication with the growing number of migrated plugins, that difference would be more significant.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

Dev Docs

Kibana static assets from now on served under a release-specific URL with long-term caching headers Cache-Control: max-age=31536000.
Before:
http://localhost:5601/bundles/plugin/dashboard/dashboard.plugin.js
After:
http://localhost:5601/bundles/8467/plugin/dashboard/dashboard.plugin.js

elasticmachine · 2020-04-24T10:56:58Z

Pinging @elastic/kibana-platform (Team:Platform)

mshustov · 2020-04-27T13:56:57Z

src/legacy/ui/ui_render/ui_render_mixin.js


-        const uiPluginIds = [...kbnServer.newPlatform.__internals.uiPlugins.public.keys()];
+          ...['kibanaUtils', 'esUiShared', 'kibanaReact', ...uiPluginIds].map(


I moved plugins loading logic to consolidate it in the one place.

src/legacy/ui/ui_render/ui_render_mixin.js

jbudz · 2020-04-27T21:27:35Z

in case anyone has issues testing - i've seen a few cases of strange behaviour recently more at the browser level than the configuration.

in the context of firefox with default settings, there's a memory cache and a disk cache.

disk caches have a max file size of ~50mb or 1/8 of the disk cache size. the disk cache size is dynamic based on free space, not sure on the math but probably maxes around 1gb. we're at a decent fraction of this for some files, but not there.

memory cache defaults to 5mb. most of our bundles won't fit there. you'll see this in private browsing

afharo

Great piece of work here!
I just added one NIT in case we want to use (in addition) the stale-while-revalidate Cache-Control policy. The tests and reasoning is explained in the PR #64518 (superseded by the way more detailed piece of work done in here) :)

src/optimize/bundles_route/dynamic_asset_response.ts

spalger · 2020-04-28T10:41:25Z

@elasticmachine merge upstream

jbudz

LGTM

I had a mini thought in between viewing the paths. There's an assumption that we always have a build hash or else routes //. And then I was wondering if there's any utility to adding overrides. It was definitely a stretch - "i want to manually rewrite all the paths bottom up to double make sure there's no vulnerabilities", "i want to turn off caching because we have our own cdn". But noting just in case it springboards some other ideas.

# Conflicts: # src/core/public/plugins/plugin_reader.ts # src/legacy/ui/ui_render/ui_render_mixin.js

# Conflicts: # src/optimize/bundles_route/bundles_route.ts # src/optimize/bundles_route/dynamic_asset_response.ts # src/optimize/bundles_route/proxy_bundles_route.ts # src/optimize/index.ts # src/optimize/np_ui_plugin_public_dirs.ts

mshustov · 2020-04-29T06:49:25Z

@elasticmachine merge upstream

* convert into TS * load plugin scripts in html body * use buildNum as a unique Id for cache busting * add tests for caching * fix tests * remove the last TODO. url should be inlined with assetss server * this logic handled by publicPathMap on the client * cache kbn-shared-deps as well * attempt to fix karma tests * always run file through replace stream * place buildHash at begining of path, include all static files * update bundles_route tests to inject buildNum everywhere * fix karma config to point to right prefix * use isDist naming throughout * explain magic number with variables * restore replacePublicPath option from elastic#64226 * replace one more instance of replacePublicPath * use promisify instead of bluebird + non-null assertions * remove one more magic number Co-authored-by: spalger <spalger@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

kibanamachine · 2020-06-04T16:30:35Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 2943a61
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Test Failures

Kibana Pipeline / kibana-intake-agent / Jest Integration Tests.src/core/public/application/integration_tests.AppContainer can navigate between two applications with custom appRoutes

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 11 times on tracked branches: https://github.com/elastic/kibana/issues/64381

Stack Trace

Error: Caught error after test environment was torn down

Http server is not setup up yet
    at HttpServer.start (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/http/http_server.ts:117:13)
    at HttpService.start (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/server/http/http_service.ts:110:29)

Kibana Pipeline / kibana-intake-agent / Jest Integration Tests.src/core/public/application/integration_tests.AppContainer should not mount when partial route path has higher specificity

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 11 times on tracked branches: https://github.com/elastic/kibana/issues/64382

Stack Trace

Error: expect(jest.fn()).toHaveBeenCalled()

Expected number of calls: >= 1
Received number of calls:    0
    at Object.it (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/src/core/public/application/integration_tests/router.test.tsx:174:35)

Kibana Pipeline / kibana-intake-agent / Jest Integration Tests.src/core/server/http/integration_tests.http service elasticsearch passes request authorization header to Elasticsearch if registerAuth was not set

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 33 times on tracked branches: https://github.com/elastic/kibana/issues/64310

Stack Trace

Error: unknown error
    at respond (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/elasticsearch/src/lib/transport.js:351:15)
    at checkRespForFailure (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at ClientRequest.wrapper (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at ClientRequest.emit (events.js:198:13)
    at Socket.socketErrorListener (_http_client.js:401:9)
    at Socket.emit (events.js:198:13)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)

History

💚 Build #44000 succeeded 2943a61
💛 Build #43694 was flaky 5c7c3f1
💚 Build #43606 succeeded b46fcd5
💚 Build #43469 succeeded 3de245b
💔 Build #43372 failed 5e5c145

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

mshustov added 2 commits April 24, 2020 11:11

convert into TS

0f88c86

load plugin scripts in html body

5c2bc55

mshustov added chore Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:New Platform performance v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.8.0 labels Apr 24, 2020

use buildNum as a unique Id for cache busting

3ab5f29

mshustov changed the title ~~Js cachable assets~~ use cache busting for KP bundles Apr 24, 2020

mshustov added 3 commits April 27, 2020 12:15

add tests for caching

f2c8d51

fix tests

9f46398

remove the last TODO. url should be inlined with assetss server

5e5c145

mshustov commented Apr 27, 2020

View reviewed changes

mshustov added 2 commits April 27, 2020 16:07

this logic handled by publicPathMap on the client

03b1584

cache kbn-shared-deps as well

5082c6e

pgayvallet reviewed Apr 27, 2020

View reviewed changes

src/legacy/ui/ui_render/ui_render_mixin.js Outdated Show resolved Hide resolved

mshustov and others added 3 commits April 27, 2020 19:26

attempt to fix karma tests

c0332e1

Merge branch 'master' into js-cachable-assets

3de245b

Merge branch 'master' of github.com:elastic/kibana into pr/64414

9f45612

spalger marked this pull request as ready for review April 27, 2020 19:34

spalger requested review from a team as code owners April 27, 2020 19:34

always run file through replace stream

c66b362

jbudz mentioned this pull request Apr 27, 2020

[SDH] Cache-Control optimizations #64518

Closed

2 tasks

spalger added 2 commits April 27, 2020 14:53

place buildHash at begining of path, include all static files

b1d7133

Merge branch 'master' of github.com:elastic/kibana into pr/64414

db23f0b

spalger added 6 commits April 28, 2020 01:39

explain magic number with variables

240553f

restore replacePublicPath option from elastic#64226

7c35408

replace one more instance of replacePublicPath

0beb6ab

use promisify instead of bluebird + non-null assertions

604ab26

Merge branch 'master' of github.com:elastic/kibana into pr/64414

50dc863

remove one more magic number

94690da

afharo reviewed Apr 28, 2020

View reviewed changes

src/optimize/bundles_route/dynamic_asset_response.ts Show resolved Hide resolved

Merge branch 'master' into js-cachable-assets

5c7c3f1

mshustov requested a review from pgayvallet April 28, 2020 16:11

jbudz approved these changes Apr 28, 2020

View reviewed changes

This was referenced Apr 28, 2020

Consolidate downloading plugin bundles to bootstrap script #64685

Merged

typescript-ify portions of src/optimize #64688

Merged

wylieconlon mentioned this pull request Apr 28, 2020

[Lens] Fix flaky test by closing options after use #64714

Merged

1 task

spalger added 2 commits April 28, 2020 17:55

Merge branch 'master' of github.com:elastic/kibana into pr/64414

6770a4a

# Conflicts: # src/core/public/plugins/plugin_reader.ts # src/legacy/ui/ui_render/ui_render_mixin.js

Merge branch 'master' into js-cachable-assets

2943a61

pgayvallet approved these changes Apr 29, 2020

View reviewed changes

spalger merged commit 5082ff3 into elastic:master Apr 29, 2020

spalger mentioned this pull request Apr 29, 2020

[7.x] use cache busting for KP bundles (#64414) #64816

Merged

spalger pushed a commit that referenced this pull request Apr 29, 2020

[7.x] use cache busting for KP bundles (#64414) (#64816)

73ae576

mshustov mentioned this pull request May 4, 2020

[meta] Kibana loading performances optimizations #62263

Closed

17 tasks

mshustov mentioned this pull request May 11, 2020

Improve plugin bundle load time #55241

Closed

mshustov added release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. and removed release_note:skip Skip the PR/issue when compiling release notes labels May 12, 2020

tylersmalley mentioned this pull request Jul 22, 2020

Support for serving static assets over a CDN #72880

Closed

tylersmalley mentioned this pull request Sep 14, 2022

Update cache policy for plugins public assets and UI shared dependencies #140761

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use cache busting for KP bundles #64414

use cache busting for KP bundles #64414

mshustov commented Apr 24, 2020 •

edited

Loading

elasticmachine commented Apr 24, 2020

mshustov Apr 27, 2020

jbudz commented Apr 27, 2020

afharo left a comment

spalger commented Apr 28, 2020

jbudz left a comment

mshustov commented Apr 29, 2020

kibanamachine commented Jun 4, 2020

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace


		const uiPluginIds = [...kbnServer.newPlatform.__internals.uiPlugins.public.keys()];
		...['kibanaUtils', 'esUiShared', 'kibanaReact', ...uiPluginIds].map(

use cache busting for KP bundles #64414

use cache busting for KP bundles #64414

Conversation

mshustov commented Apr 24, 2020 • edited Loading

Summary

Checklist

For maintainers

Dev Docs

elasticmachine commented Apr 24, 2020

mshustov Apr 27, 2020

Choose a reason for hiding this comment

jbudz commented Apr 27, 2020

afharo left a comment

Choose a reason for hiding this comment

spalger commented Apr 28, 2020

jbudz left a comment

Choose a reason for hiding this comment

mshustov commented Apr 29, 2020

kibanamachine commented Jun 4, 2020

💔 Build Failed

Failed CI Steps

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace

History

mshustov commented Apr 24, 2020 •

edited

Loading