Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added a section of how to deal with flakey tests to the developer guide. #4868

Merged
merged 1 commit into from
Oct 21, 2022
Merged

Added a section of how to deal with flakey tests to the developer guide. #4868

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Apply reproducible builds configuration for OpenSearch plugins through gradle plugin ([#4746](https://github.com/opensearch-project/OpenSearch/pull/4746))
- Add groupId value propagation tests for ZIP publication task ([#4772](https://github.com/opensearch-project/OpenSearch/pull/4772))
- Add support for GeoJson Point type in GeoPoint field ([#4597](https://github.com/opensearch-project/OpenSearch/pull/4597))
- Add dev guide for dealing with flakey tests ([4868](https://github.com/opensearch-project/OpenSearch/pull/4868))

### Dependencies
- Bumps `log4j-core` from 2.18.0 to 2.19.0
Expand Down
108 changes: 65 additions & 43 deletions DEVELOPER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,21 +38,22 @@
- [Gradle Plugins](#gradle-plugins)
- [Distribution Download Plugin](#distribution-download-plugin)
- [Creating fat-JAR of a Module](#creating-fat-jar-of-a-module)
- [Misc](#misc)
- [git-secrets](#git-secrets)
- [Installation](#installation)
- [Configuration](#configuration)
- [Components](#components)
- [Build libraries & interfaces](#build-libraries--interfaces)
- [Clients & Libraries](#clients--libraries)
- [Plugins](#plugins-1)
- [Indexing & search](#indexing--search)
- [Indexing & Search](#indexing--search)
- [Aggregations](#aggregations)
- [Distributed Framework](#distributed-framework)
- [Submitting Changes](#submitting-changes)
- [Backports](#backports)
- [LineLint](#linelint)
- [Lucene Snapshots](#lucene-snapshots)
- [Misc](#misc)
- [Git Secrets](#git-secrets)
- [Installation](#installation)
- [Configuration](#configuration)
- [Submitting Changes](#submitting-changes)
- [Backports](#backports)
- [LineLint](#linelint)
- [Lucene Snapshots](#lucene-snapshots)
- [Flakey Tests](#flakey-tests)

# Developer Guide

Expand Down Expand Up @@ -414,37 +415,12 @@ Refer the installed JAR as any other maven artifact, e.g.
</dependency>
```

## Misc

### git-secrets

Security is our top priority. Avoid checking in credentials.

#### Installation
Install [awslabs/git-secrets](https://github.com/awslabs/git-secrets) by running the following commands.
```
git clone https://github.com/awslabs/git-secrets.git
cd git-secrets
make install
```

#### Configuration
You can configure git secrets per repository, you need to change the directory to the root of the repository and run the following command.
```
git secrets --install
✓ Installed commit-msg hook to .git/hooks/commit-msg
✓ Installed pre-commit hook to .git/hooks/pre-commit
✓ Installed prepare-commit-msg hook to .git/hooks/prepare-commit-msg
```
Then, you need to apply patterns for git-secrets, you can install the AWS standard patterns by running the following command.
```
git secrets --register-aws
```

## Components

As you work in the OpenSearch repo you may notice issues getting labeled with component labels. It's a housekeeping task to help group together similar pieces of work. You can pretty much ignore it, but if you're curious, here's what the different labels mean:

### Build libraries & interfaces

Tasks to make sure the build tasks are useful and packaging and distribution are easy.

Includes:
Expand All @@ -458,6 +434,7 @@ Includes:


### Clients & Libraries

APIs and communication mechanisms for external connections to OpenSearch. This includes the “library” directory in OpenSearch (a set of common functions).

Includes:
Expand All @@ -467,6 +444,7 @@ Includes:
- CLI

### Plugins

Anything touching the plugin infrastructure within core OpenSearch.

Includes:
Expand All @@ -476,7 +454,8 @@ Includes:
- Plugin interfaces


### Indexing & search
### Indexing & Search

The critical path of indexing and search, including: Measure index and search, performance, Improving the performance of indexing and search, ensure synchronization OpenSearch APIs with upstream Lucene change (e.g. new field types, changing doc values and codex).

Includes:
Expand All @@ -487,6 +466,7 @@ Includes:
- DocValues

### Aggregations

Making sure OpenSearch can be used as a compute engine.

Includes:
Expand All @@ -495,6 +475,7 @@ Includes:
- Framework

### Distributed Framework

Work to make sure that OpenSearch can scale in a distributed manner.

Includes:
Expand All @@ -506,15 +487,43 @@ Includes:
- Shard Strategies
- Circuit Breakers

## Submitting Changes
## Misc

### Git Secrets

Security is our top priority. Avoid checking in credentials.

#### Installation
Install [awslabs/git-secrets](https://github.com/awslabs/git-secrets) by running the following commands.
```
git clone https://github.com/awslabs/git-secrets.git
cd git-secrets
make install
```

#### Configuration
You can configure git secrets per repository, you need to change the directory to the root of the repository and run the following command.
```
git secrets --install
✓ Installed commit-msg hook to .git/hooks/commit-msg
✓ Installed pre-commit hook to .git/hooks/pre-commit
✓ Installed prepare-commit-msg hook to .git/hooks/prepare-commit-msg
```
Then, you need to apply patterns for git-secrets, you can install the AWS standard patterns by running the following command.
```
git secrets --register-aws
```

### Submitting Changes

See [CONTRIBUTING](CONTRIBUTING.md).

## Backports
### Backports

The Github workflow in [`backport.yml`](.github/workflows/backport.yml) creates backport PRs automatically when the original PR with an appropriate label `backport <backport-branch-name>` is merged to main with the backport workflow run successfully on the PR. For example, if a PR on main needs to be backported to `1.x` branch, add a label `backport 1.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is merged to main, the workflow will create a backport PR to the `1.x` branch.

## LineLint
### LineLint

A linter in [`code-hygiene.yml`](.github/workflows/code-hygiene.yml) that validates simple newline and whitespace rules in all sorts of files. It can:
- Recursively check a directory tree for files that do not end in a newline
- Automatically fix these files by adding a newline or trimming extra newlines.
Expand All @@ -529,7 +538,20 @@ Pass a list of files or directories to limit your search.

linelint README.md LICENSE

# Lucene Snapshots
### Lucene Snapshots

The Github workflow in [lucene-snapshots.yml](.github/workflows/lucene-snapshots.yml) is a Github worfklow executable by maintainers to build a top-down snapshot build of lucene.
These snapshots are available to test compatibility with upcoming changes to Lucene by updating the version at [version.properties](buildsrc/version.properties) with the `version-snapshot-sha` version.
Example: `lucene = 10.0.0-snapshot-2e941fc`.
These snapshots are available to test compatibility with upcoming changes to Lucene by updating the version at [version.properties](buildsrc/version.properties) with the `version-snapshot-sha` version. Example: `lucene = 10.0.0-snapshot-2e941fc`.

### Flakey Tests

OpenSearch has a very large test suite with long running, often failing (flakey), integration tests. Such individual tests are labelled as [Flakey Random Test Failure](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aopen+is%3Aissue+label%3A%22Flakey+Random+Test+Failure%22). Your help is wanted fixing these!

If you encounter a build/test failure in CI that is unrelated to the change in your pull request, it may be a known flakey test, or a new test failure.

1. Follow failed CI links, and locate the failing test(s).
2. Copy-paste the failure into a comment of your PR.
3. Search through [issues](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aopen+is%3Aissue+label%3A%22Flakey+Random+Test+Failure%22) using the name of the failed test for whether this is a known flakey test.
5. If an existing issue is found, paste a link to the known issue in a comment to your PR.
6. If no existing issue is found, open one.
7. Retry CI via the GitHub UX or by pushing an update to your PR.