Skip to content

Commit

Permalink
Rename kbn-optimizer to osd-optimizer (#37) (#59)
Browse files Browse the repository at this point in the history
Signed-off-by: Bishoy Boktor <boktorbb@amazon.com>
  • Loading branch information
boktorbb authored and mihirsoni committed Mar 20, 2021
1 parent 6913462 commit 7b8a09f
Show file tree
Hide file tree
Showing 102 changed files with 196 additions and 198 deletions.
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# @kbn/optimizer
# @osd/optimizer

`@kbn/optimizer` is a package for building Kibana platform UI plugins (and hopefully more soon).
`@osd/optimizer` is a package for building OpenSearch Dashboards platform UI plugins (and hopefully more soon).

Kibana Platform plugins with `"ui": true` in their `kibana.json` file will have their `public/index.ts` file (and all of its dependencies) bundled into the `target/public` directory of the plugin. The build output does not need to be updated when other plugins are updated and is included in the distributable without requiring that we ship `@kbn/optimizer` 🎉.
OpenSearch Dashboards Platform plugins with `"ui": true` in their `opensearch_dashboards.json` file will have their `public/index.ts` file (and all of its dependencies) bundled into the `target/public` directory of the plugin. The build output does not need to be updated when other plugins are updated and is included in the distributable without requiring that we ship `@osd/optimizer` 🎉.

## Webpack config

Expand All @@ -16,25 +16,25 @@ To make front-end code easier to debug the optimizer uses the `BROWSERSLIST_ENV=

## Running the optimizer

The `@kbn/optimizer` is automatically executed from the dev cli, the Kibana build scripts, and in CI. If you're running Kibana locally in some other way you might need to build the plugins manually, which you can do by running `node scripts/build_kibana_platform_plugins` (pass `--help` for options).
The `@osd/optimizer` is automatically executed from the dev cli, the OpenSearch Dashboards build scripts, and in CI. If you're running OpenSearch Dashboards locally in some other way you might need to build the plugins manually, which you can do by running `node scripts/build_opensearch_dashboards_platform_plugins` (pass `--help` for options).

### Worker count

You can limit the number of workers the optimizer uses by setting the `KBN_OPTIMIZER_MAX_WORKERS` environment variable. You might want to do this if your system struggles to keep up while the optimizer is getting started and building all plugins as fast as possible. Setting `KBN_OPTIMIZER_MAX_WORKERS=1` will cause the optimizer to take the longest amount of time but will have the smallest impact on other components of your system.
You can limit the number of workers the optimizer uses by setting the `OSD_OPTIMIZER_MAX_WORKERS` environment variable. You might want to do this if your system struggles to keep up while the optimizer is getting started and building all plugins as fast as possible. Setting `OSD_OPTIMIZER_MAX_WORKERS=1` will cause the optimizer to take the longest amount of time but will have the smallest impact on other components of your system.

We only limit the number of workers we will start at any given time. If we start more workers later we will limit the number of workers we start at that time by the maximum, but we don't take into account the number of workers already started because it is assumed that those workers are doing very little work. This greatly simplifies the logic as we don't ever have to reallocate workers and provides the best performance in most cases.

### Caching

Bundles built by the the optimizer include a cache file which describes the information needed to determine if the bundle needs to be rebuilt when the optimizer is restarted. Caching is enabled by default and is very aggressive about invalidating the cache output, but if you need to disable caching you can pass `--no-cache` to `node scripts/build_kibana_platform_plugins`, or set the `KBN_OPTIMIZER_NO_CACHE` environment variable to anything (env overrides everything).
Bundles built by the the optimizer include a cache file which describes the information needed to determine if the bundle needs to be rebuilt when the optimizer is restarted. Caching is enabled by default and is very aggressive about invalidating the cache output, but if you need to disable caching you can pass `--no-cache` to `node scripts/build_opensearch_dashboards_platform_plugins`, or set the `OSD_OPTIMIZER_NO_CACHE` environment variable to anything (env overrides everything).

When a bundle is determined to be up-to-date a worker is not started for the bundle. If running the optimizer with the `--dev/--watch` flag, then all the files referenced by cached bundles are watched for changes. Once a change is detected in any of the files referenced by the built bundle a worker is started. If a file is changed that is referenced by several bundles then workers will be started for each bundle, combining workers together to respect the worker limit.

## Bundle Refs

In order to dramatically reduce the size of our bundles, and the time it takes to build them, bundles will "ref" other bundles being built at the same time. When the optimizer starts it creates a list of "refs" that could be had from the list of bundles being built. Each worker uses that list to determine which import statements in a bundle should be replaced with a runtime reference to the output of another bundle.

At runtime the bundles share a set of entry points via the `__kbnBundles__` global. By default a plugin shares `public` so that other code can use relative imports to access that directory. To expose additional directories they must be listed in the plugin's kibana.json "extraPublicDirs" field. The directories listed there will **also** be exported from the plugins bundle so that any other plugin can import that directory. "common" is commonly in the list of "extraPublicDirs".
At runtime the bundles share a set of entry points via the `__osdBundles__` global. By default a plugin shares `public` so that other code can use relative imports to access that directory. To expose additional directories they must be listed in the plugin's opensearch_dashboards.json "extraPublicDirs" field. The directories listed there will **also** be exported from the plugins bundle so that any other plugin can import that directory. "common" is commonly in the list of "extraPublicDirs".

> NOTE: We plan to replace the `extraPublicDirs` functionality soon with better mechanisms for statically sharing code between bundles.
Expand All @@ -44,22 +44,22 @@ Any import in a bundle which resolves into another bundles "context" directory,

## Themes

SASS imports in bundles are automatically converted to CSS for one or more themes. In development we build the `v7light` and `v7dark` themes by default to improve build performance. When producing distributable bundles the default shifts to `*` so that the distributable bundles will include all themes, preventing the bundles from needing to be rebuilt when users change the active theme in Kibana's advanced settings.
SASS imports in bundles are automatically converted to CSS for one or more themes. In development we build the `v7light` and `v7dark` themes by default to improve build performance. When producing distributable bundles the default shifts to `*` so that the distributable bundles will include all themes, preventing the bundles from needing to be rebuilt when users change the active theme in OpenSearch Dashboards's advanced settings.

To customize the themes that are built for development you can specify the `KBN_OPTIMIZER_THEMES` environment variable to one or more theme tags, or use `*` to build styles for all themes. Unfortunately building more than one theme significantly impacts build performance, so try to be strategic about which themes you build.
To customize the themes that are built for development you can specify the `OSD_OPTIMIZER_THEMES` environment variable to one or more theme tags, or use `*` to build styles for all themes. Unfortunately building more than one theme significantly impacts build performance, so try to be strategic about which themes you build.

Currently supported theme tags: `v7light`, `v7dark`, `v8light`, `v8dark`

Examples:
```sh
# start Kibana with only a single theme
KBN_OPTIMIZER_THEMES=v7light yarn start
# start OpenSearch Dashboards with only a single theme
OSD_OPTIMIZER_THEMES=v7light yarn start

# start Kibana with dark themes for version 7 and 8
KBN_OPTIMIZER_THEMES=v7dark,v8dark yarn start
# start OpenSearch Dashboards with dark themes for version 7 and 8
OSD_OPTIMIZER_THEMES=v7dark,v8dark yarn start

# start Kibana with all the themes
KBN_OPTIMIZER_THEMES=* yarn start
# start OpenSearch Dashboards with all the themes
OSD_OPTIMIZER_THEMES=* yarn start
```

## API
Expand All @@ -68,9 +68,9 @@ To run the optimizer from code, you can import the [`OptimizerConfig`][Optimizer

Example:
```ts
import { runOptimizer, OptimizerConfig, logOptimizerState } from '@kbn/optimizer';
import { REPO_ROOT } from '@kbn/utils';
import { ToolingLog } from '@kbn/dev-utils';
import { runOptimizer, OptimizerConfig, logOptimizerState } from '@osd/optimizer';
import { REPO_ROOT } from '@osd/utils';
import { ToolingLog } from '@osd/dev-utils';

const log = new ToolingLog({
level: 'verbose',
Expand All @@ -89,7 +89,7 @@ await lastValueFrom(
);
```

This is essentially what we're doing in [`script/build_kibana_platform_plugins`][Cli] and the new [build system task][BuildTask].
This is essentially what we're doing in [`script/build_opensearch_dashboards_platform_plugins`][Cli] and the new [build system task][BuildTask].

## Internals

Expand Down Expand Up @@ -140,4 +140,4 @@ For an example of how to handle these states checkout the [`logOptimizerState()`
[OptimizerConfig]: src/optimizer_config.ts
[LogOptimizerState]: src/log_optimizer_state.ts
[AssignBundlesToWorkers]: src/assign_bundles_to_workers.ts
[BuildTask]: ../../src/dev/build/tasks/build_kibana_platform_plugins.js
[BuildTask]: ../../src/dev/build/tasks/build_opensearch_dashboards_platform_plugins.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@
*/

module.exports = {
presets: ['@kbn/babel-preset/node_preset'],
presets: ['@osd/babel-preset/node_preset'],
ignore: ['**/*.test.js'],
};
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ pageLoadAssetSize:
embeddable: 242671
embeddableEnhanced: 41145
enterpriseSearch: 35741
esUiShared: 326798
opensearchUiShared: 326798
expressions: 224120
features: 31211
fileUpload: 24717
Expand All @@ -40,10 +40,10 @@ pageLoadAssetSize:
ingestPipelines: 58003
inputControlVis: 172819
inspector: 148999
kibanaLegacy: 107855
kibanaOverview: 56426
kibanaReact: 162353
kibanaUtils: 198829
opensearchDashboardsLegacy: 107855
opensearchDashboardsOverview: 56426
opensearchDashboardsReact: 162353
opensearchDashboardsUtils: 198829
lens: 96624
licenseManagement: 41961
licensing: 39008
Expand Down
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
{
"name": "@kbn/optimizer",
"name": "@osd/optimizer",
"version": "1.0.0",
"private": true,
"license": "Apache-2.0",
"main": "./target/index.js",
"scripts": {
"build": "babel src --out-dir target --copy-files --delete-dir-on-start --extensions .ts --ignore *.test.ts --source-maps=inline",
"kbn:bootstrap": "yarn build",
"kbn:watch": "yarn build --watch"
"osd:bootstrap": "yarn build",
"osd:watch": "yarn build --watch"
},
"dependencies": {
"@babel/cli": "^7.10.5",
"@babel/core": "^7.11.6",
"@kbn/babel-preset": "1.0.0",
"@kbn/dev-utils": "1.0.0",
"@kbn/std": "1.0.0",
"@kbn/ui-shared-deps": "1.0.0",
"@osd/babel-preset": "1.0.0",
"@osd/dev-utils": "1.0.0",
"@osd/std": "1.0.0",
"@osd/ui-shared-deps": "1.0.0",
"autoprefixer": "^9.7.4",
"babel-loader": "^8.0.6",
"clean-webpack-plugin": "^3.0.0",
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ import 'source-map-support/register';

import Path from 'path';

import { REPO_ROOT } from '@kbn/utils';
import { lastValueFrom } from '@kbn/std';
import { run, createFlagError, CiStatsReporter } from '@kbn/dev-utils';
import { REPO_ROOT } from '@osd/utils';
import { lastValueFrom } from '@osd/std';
import { run, createFlagError, CiStatsReporter } from '@osd/dev-utils';

import { logOptimizerState } from './log_optimizer_state';
import { OptimizerConfig } from './optimizer';
Expand Down Expand Up @@ -175,10 +175,10 @@ run(
--no-cache disable the cache
--filter comma-separated list of bundle id filters, results from multiple flags are merged, * and ! are supported
--no-examples don't build the example plugins
--dist create bundles that are suitable for inclusion in the Kibana distributable, enabled when running with --update-limits
--dist create bundles that are suitable for inclusion in the OpenSearch Dashboards distributable, enabled when running with --update-limits
--scan-dir add a directory to the list of directories scanned for plugins (specify as many times as necessary)
--no-inspect-workers when inspecting the parent process, don't inspect the workers
--report-stats attempt to report stats about this execution of the build to the kibana-ci-stats service using this name
--report-stats attempt to report stats about this execution of the build to the opensearch-dashboards-ci-stats service using this name
--validate-limits validate the limits.yml config to ensure that there are limits defined for every bundle
--update-limits run a build and rewrite the limits file to include the current bundle sizes +5kb
`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ it('parses bundles from JSON specs', () => {
Bundle {
"banner": undefined,
"cache": BundleCache {
"path": "/foo/bar/target/.kbn-optimizer-cache",
"path": "/foo/bar/target/.osd-optimizer-cache",
"state": undefined,
},
"contextDir": "/foo/bar",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ export interface BundleSpec {
readonly outputDir: string;
/** Banner that should be written to all bundle JS files */
readonly banner?: string;
/** Absolute path to a kibana.json manifest file, if omitted we assume there are not dependenices */
/** Absolute path to a opensearch_dashboards.json manifest file, if omitted we assume there are not dependenices */
readonly manifestPath?: string;
}

Expand Down Expand Up @@ -87,7 +87,7 @@ export class Bundle {
this.manifestPath = spec.manifestPath;
this.banner = spec.banner;

this.cache = new BundleCache(Path.resolve(this.outputDir, '.kbn-optimizer-cache'));
this.cache = new BundleCache(Path.resolve(this.outputDir, '.osd-optimizer-cache'));
}

/**
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

import * as Rx from 'rxjs';
import { toArray, map } from 'rxjs/operators';
import { lastValueFrom } from '@kbn/std';
import { lastValueFrom } from '@osd/std';

import { pipeClosure, debounceTimeBuffer, maybeMap, maybe } from './rxjs_helpers';

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

import * as Rx from 'rxjs';
import { mergeMap, tap, debounceTime, map, toArray } from 'rxjs/operators';
import { firstValueFrom } from '@kbn/std';
import { firstValueFrom } from '@osd/std';

type Operator<T1, T2> = (source: Rx.Observable<T1>) => Rx.Observable<T2>;
type MapFn<T1, T2> = (item: T1, index: number) => T2;
Expand Down
File renamed without changes.
Loading

0 comments on commit 7b8a09f

Please sign in to comment.