Skip to content

Commit

Permalink
Merge branch 'master' into data/allow-custom-formatting
Browse files Browse the repository at this point in the history
  • Loading branch information
tsullivan committed Jul 13, 2020
2 parents 54fa2fe + 17dc043 commit 34df331
Show file tree
Hide file tree
Showing 186 changed files with 5,692 additions and 3,278 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ disabledPlugins
webpackstats.json
/config/*
!/config/kibana.yml
!/config/node.options
coverage
selenium
.babel_register_cache.json
Expand Down
738 changes: 2 additions & 736 deletions CONTRIBUTING.md

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions config/node.options
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
## Node command line options
## See `node --help` and `node --v8-options` for available options
## Please note you should specify one option per line

## max size of old space in megabytes
#--max-old-space-size=4096
38 changes: 0 additions & 38 deletions docs/developer/add-data-guide.asciidoc

This file was deleted.

18 changes: 18 additions & 0 deletions docs/developer/advanced/development-basepath.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[[development-basepath]]
=== Considerations for basepath

In dev mode, {kib} by default runs behind a proxy which adds a random path component to its URL.

You can set this explicitly using `server.basePath` in <<settings>>.

Use the server.rewriteBasePath setting to tell {kib} if it should remove the basePath from requests it receives, and to prevent a deprecation warning at startup. This setting cannot end in a slash (/).

If you want to turn off the basepath when in development mode, start {kib} with the `--no-basepath` flag

[source,bash]
----
yarn start --no-basepath
----



Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
[[development-es-snapshots]]
=== Daily Elasticsearch Snapshots

For local development and CI, Kibana, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests).
For local development and CI, {kib}, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests).

A snapshot is just a group of tarballs, one for each supported distribution/architecture/os of Elasticsearch, and a JSON-based manifest file containing metadata about the distributions.

https://ci.kibana.dev/es-snapshots[A dashboard] is available that shows the current status and compatibility of the latest Elasticsearch snapshots.

==== Process Overview

1. Elasticsearch snapshots are built for each current tracked branch of Kibana.
1. Elasticsearch snapshots are built for each current tracked branch of {kib}.
2. Each snapshot is uploaded to a public Google Cloud Storage bucket, `kibana-ci-es-snapshots-daily`.
** At this point, the snapshot is not automatically used in CI or local development. It needs to be tested/verified first.
3. Each snapshot is tested with the latest commit of the corresponding Kibana branch, using the full CI suite.
3. Each snapshot is tested with the latest commit of the corresponding {kib} branch, using the full CI suite.
4. After CI
** If the snapshot passes, it is promoted and automatically used in CI and local development.
** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and Kibana.
** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and {kib}.

==== Using the latest snapshot

Expand All @@ -39,7 +39,7 @@ KBN_ES_SNAPSHOT_USE_UNVERIFIED=true node scripts/functional_tests_server

Currently, there is not a way to run your pull request with the latest unverified snapshot without a code change. You can, however, do it with a small code change.

1. Edit `Jenkinsfile` in the root of the Kibana repo
1. Edit `Jenkinsfile` in the root of the {kib} repo
2. Add `env.KBN_ES_SNAPSHOT_USE_UNVERIFIED = 'true'` at the top of the file.
3. Commit the change

Expand Down Expand Up @@ -75,13 +75,13 @@ The file structure for this bucket looks like this:

==== How snapshots are built, tested, and promoted

Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of Kibana.
Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of {kib}.

===== Build

https://kibana-ci.elastic.co/job/elasticsearch+snapshots+build/[This Jenkins job] builds the Elasticsearch snapshots and uploads them to GCS.

The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_build_es[in the kibana repo].
The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_build_es[in the {kib} repo].

1. Checkout Elasticsearch repo for the given branch/version.
2. Run `./gradlew -p distribution/archives assemble --parallel` to create all of the Elasticsearch distributions.
Expand All @@ -91,15 +91,15 @@ The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/ma
** e.g. `<version>/archives/<unique id>`
6. Replace `<version>/manifest-latest.json` in GCS with this newest manifest.
** This allows the `KBN_ES_SNAPSHOT_USE_UNVERIFIED` flag to work.
7. Trigger the verification job, to run the full Kibana CI test suite with this snapshot.
7. Trigger the verification job, to run the full {kib} CI test suite with this snapshot.

===== Verification and Promotion

https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest Elasticsearch snapshot with the full Kibana CI pipeline, and promotes if it there are no test failures.
https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest Elasticsearch snapshot with the full {kib} CI pipeline, and promotes if it there are no test failures.

The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_verify_es[in the kibana repo].
The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_verify_es[in the {kib} repo].

1. Checkout Kibana and set up CI environment as normal.
1. Checkout {kib} and set up CI environment as normal.
2. Set the `ES_SNAPSHOT_MANIFEST` env var to point to the latest snapshot manifest.
3. Run CI (functional tests, integration tests, etc).
4. After CI
Expand Down
12 changes: 12 additions & 0 deletions docs/developer/advanced/index.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
[[advanced]]
== Advanced

* <<running-elasticsearch>>
* <<development-es-snapshots>>
* <<development-basepath>>

include::development-es-snapshots.asciidoc[]

include::running-elasticsearch.asciidoc[]

include::development-basepath.asciidoc[]
118 changes: 118 additions & 0 deletions docs/developer/advanced/running-elasticsearch.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
[[running-elasticsearch]]
=== Running elasticsearch during development

There are many ways to run Elasticsearch while you are developing.

[float]

==== By snapshot

This will run a snapshot of elasticsearch that is usually built nightly. Read more about <<development-es-snapshots>>.

[source,bash]
----
yarn es snapshot
----

See all available options, like how to specify a specific license, with the `--help` flag.

[source,bash]
----
yarn es snapshot --help
----

`trial` will give you access to all capabilities.

**Keeping data between snapshots**

If you want to keep the data inside your Elasticsearch between usages of this command, you should use the following command, to keep your data folder outside the downloaded snapshot folder:

[source,bash]
----
yarn es snapshot -E path.data=../data
----

==== By source

If you have the Elasticsearch repo checked out locally and wish to run against that, use `source`. By default, it will reference an elasticsearch checkout which is a sibling to the {kib} directory named elasticsearch. If you wish to use a checkout in another location you can provide that by supplying --source-path

[source,bash]
----
yarn es source
----

==== From an archive

Use this if you already have a distributable. For released versions, one can be obtained on the Elasticsearch downloads page.

[source,bash]
----
yarn es archive <full_path_to_archive>
----

Each of these will run Elasticsearch with a basic license. Additional options are available, pass --help for more information.

==== From a remote host

You can save some system resources, and the effort of generating sample data, if you have a remote Elasticsearch cluster to connect to. (Elasticians: you do! Check with your team about where to find credentials)

You'll need to create a kibana.dev.yml (<<customize-kibana-yml>>) and add the following to it:

[source,bash]
----
elasticsearch.hosts:
- {{ url }}
elasticsearch.username: {{ username }}
elasticsearch.password: {{ password }}
elasticsearch.ssl.verificationMode: none
----

If many other users will be interacting with your remote cluster, you'll want to add the following to avoid causing conflicts:

[source,bash]
----
kibana.index: '.{YourGitHubHandle}-kibana'
xpack.task_manager.index: '.{YourGitHubHandle}-task-manager-kibana'
----

===== Running remote clusters

Setup remote clusters for cross cluster search (CCS) and cross cluster replication (CCR).

Start your primary cluster by running:

[source,bash]
----
yarn es snapshot -E path.data=../data_prod1
----

Start your remote cluster by running:

[source,bash]
----
yarn es snapshot -E transport.port=9500 -E http.port=9201 -E path.data=../data_prod2
----

Once both clusters are running, start {kib}. {kib} will connect to the primary cluster.

Setup the remote cluster in {kib} from either Management -> Elasticsearch -> Remote Clusters UI or by running the following script in Console.

[source,bash]
----
PUT _cluster/settings
{
"persistent": {
"cluster": {
"remote": {
"cluster_one": {
"seeds": [
"localhost:9500"
]
}
}
}
}
}
----

Follow the cross-cluster search instructions for setting up index patterns to search across clusters (<<management-cross-cluster-search>>).
38 changes: 38 additions & 0 deletions docs/developer/architecture/add-data-tutorials.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
[[add-data-tutorials]]
=== Add data tutorials

`Add Data` in the {kib} Home application contains tutorials for setting up data flows in the Elastic stack.

Each tutorial contains three sets of instructions:

* `On Premise.` Set up a data flow when both {kib} and Elasticsearch are running on premise.
* `On Premise Elastic Cloud.` Set up a data flow when {kib} is running on premise and Elasticsearch is running on Elastic Cloud.
* `Elastic Cloud.` Set up a data flow when both {kib} and Elasticsearch are running on Elastic Cloud.

[float]
==== Creating a new tutorial
1. Create a new directory in the link:https://github.com/elastic/kibana/tree/master/src/plugins/home/server/tutorials[tutorials directory].
2. In the new directory, create a file called `index.ts` that exports a function.
The function must return a function object that conforms to the `TutorialSchema` interface link:{kib-repo}tree/{branch}/src/plugins/home/server/services/tutorials/lib/tutorial_schema.ts[tutorial schema].
3. Register the tutorial in link:{kib-repo}tree/{branch}/src/plugins/home/server/tutorials/register.ts[register.ts] by adding it to the `builtInTutorials`.
// TODO update path once assets are migrated
4. Add image assets to the link:{kib-repo}tree/{branch}/src/legacy/core_plugins/kibana/public/home/tutorial_resources[tutorial_resources directory].
5. Run {kib} locally to preview the tutorial.
6. Create a PR and go through the review process to get the changes approved.

If you are creating a new plugin and the tutorial is only related to that plugin, you can also place the `TutorialSchema` object into your plugin folder. Add `home` to the `requiredPlugins` list in your `kibana.json` file.
Then register the tutorial object by calling `home.tutorials.registerTutorial(tutorialObject)` in the `setup` lifecycle of your server plugin.

[float]
===== Variables
String values can contain variables that are substituted when rendered. Variables are specified by `{}`.
For example: `{config.docs.version}` is rendered as `6.2` when running the tutorial in {kib} 6.2.

link:{kib-repo}tree/{branch}/src/legacy/core_plugins/kibana/public/home/np_ready/components/tutorial/replace_template_strings.js#L23[Provided variables]

[float]
===== Markdown
String values can contain limited Markdown syntax.

link:{kib-repo}tree/{branch}/src/legacy/core_plugins/kibana/public/home/components/tutorial/content.js#L8[Enabled Markdown grammars]

Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
[[development-visualize-index]]
== Developing Visualizations
=== Developing Visualizations

[IMPORTANT]
==============================================
These pages document internal APIs and are not guaranteed to be supported across future versions of Kibana.
These pages document internal APIs and are not guaranteed to be supported across future versions of {kib}.
==============================================

The internal APIs for creating custom visualizations are in a state of heavy churn as
they are being migrated to the new Kibana platform, and large refactorings have been
they are being migrated to the new {kib} platform, and large refactorings have been
happening across minor releases in the `7.x` series. In particular, in `7.5` and later
we have made significant changes to the legacy APIs as we work to gradually replace them.

Expand All @@ -20,7 +20,7 @@ If you would like to keep up with progress on the visualizations plugin in the m
here are a few resources:

* The <<breaking-changes,breaking changes>> documentation, where we try to capture any changes to the APIs as they occur across minors.
* link:https://github.com/elastic/kibana/issues/44121[Meta issue] which is tracking the move of the plugin to the new Kibana platform
* link:https://github.com/elastic/kibana/issues/44121[Meta issue] which is tracking the move of the plugin to the new {kib} platform
* Our link:https://www.elastic.co/blog/join-our-elastic-stack-workspace-on-slack[Elastic Stack workspace on Slack].
* The {kib-repo}blob/{branch}/src/plugins/visualizations[source code], which will continue to be
the most accurate source of information.
25 changes: 25 additions & 0 deletions docs/developer/architecture/index.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
[[kibana-architecture]]
== Architecture

[IMPORTANT]
==============================================
{kib} developer services and apis are in a state of constant development. We cannot provide backwards compatibility at this time due to the high rate of change.
==============================================

Our developer services are changing all the time. One of the best ways to discover and learn about them is to read the available
READMEs from all the plugins inside our {kib-repo}tree/{branch}/src/plugins[open source plugins folder] and our
{kib-repo}/tree/{branch}/x-pack/plugins[commercial plugins folder].

A few services also automatically generate api documentation which can be browsed inside the {kib-repo}tree/{branch}/docs/development[docs/development section of our repo]

A few notable services are called out below.

* <<development-security>>
* <<add-data-tutorials>>
* <<development-visualize-index>>

include::add-data-tutorials.asciidoc[]

include::development-visualize-index.asciidoc[]

include::security/index.asciidoc[]
Loading

0 comments on commit 34df331

Please sign in to comment.