Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: add integration tests #77

Merged
merged 2 commits into from
Sep 20, 2022
Merged

chore: add integration tests #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7a00497
chore: add integration tests
toddbaert Sep 16, 2022
ca0594f
improve POM spacing
toddbaert Sep 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/workflows/pullrequest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ permissions:
jobs:
build:
runs-on: ubuntu-latest
services:
flagd:
image: ghcr.io/open-feature/flagd-testbed:latest
ports:
- 8013:8013

Comment on lines +12 to +17
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Run flagd in the CI.

steps:
- name: Check out the code
uses: actions/checkout@v3
Expand All @@ -28,7 +34,7 @@ jobs:
${{ runner.os }}-maven-
- name: Build with Maven
run: mvn --batch-mode --update-snapshots verify
run: mvn --batch-mode --update-snapshots verify -P integration-test

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
Expand Down
3 changes: 3 additions & 0 deletions .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "test-harness"]
path = test-harness
url = https://github.com/open-feature/test-harness
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,12 @@ We hold regular meetings which you can see [here](https://github.com/open-featur

We are also present on the `#openfeature` channel in the [CNCF slack](https://slack.cncf.io/).

## Developing

### Integration tests

The continuous integration runs a set of [gherkin integration tests](https://github.com/open-feature/test-harness/blob/main/features/evaluation.feature) using [`flagd`](https://github.com/open-feature/flagd). These tests do not run with the default maven profile. If you'd like to run them locally, you can start the flagd testbed with `docker run -p 8013:8013 ghcr.io/open-feature/flagd-testbed:latest` and then run `mvn test -P integration-test`.

## Releasing

See [releasing](./docs/release.md).
Expand Down
124 changes: 121 additions & 3 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>${maven.compiler.source}</maven.compiler.target>
<junit.jupiter.version>5.8.1</junit.jupiter.version>
<!-- exclusion expression for integration tests -->
<testExclusions>**/integration/*.java</testExclusions>
</properties>

<name>OpenFeature Java SDK</name>
Expand Down Expand Up @@ -46,7 +48,6 @@
<scope>provided</scope>
</dependency>


<dependency>
<!-- used so that lombok can generate suppressions for spotbugs. It needs to find it on the relevant classpath -->
<groupId>com.github.spotbugs</groupId>
Expand Down Expand Up @@ -96,29 +97,81 @@
<version>${junit.jupiter.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>${junit.jupiter.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>${junit.jupiter.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-suite</artifactId>
<version>1.8.1</version>
<version>1.9.0</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit-platform-engine</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>31.1-jre</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>dev.openfeature.contrib.providers</groupId>
<artifactId>flagd</artifactId>
<version>0.3.2</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the plan to keep this up to date?

Copy link
Member Author

@toddbaert toddbaert Sep 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it's necessary to automate updates.

The purpose is to test the SDK, not flagd, so as long as the tests are working, we can have confidence the SDK is functioning at least in terms of it's provider interfaces. If there was some breaking change in the SDK itself that impacted flagd, this version of the flagd provider might throw, then we'd have to absorb the changes in there.

There will be similar tests in flagd itself that will run with the latest version of flagd.

I'm sure we can use a plugin or some github action to automate this update, but honestly I think that would just be adding an unhelpful variable.

<scope>test</scope>
</dependency>

</dependencies>

<dependencyManagement>
<dependencies>

<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-bom</artifactId>
<version>7.5.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<dependency>
<groupId>org.junit</groupId>
<artifactId>junit-bom</artifactId>
<version>5.9.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>

</dependencies>
</dependencyManagement>

<build>
<plugins>

<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.3.0</version>
Expand All @@ -137,16 +190,21 @@
<ignoredUnusedDeclaredDependency>org.junit*</ignoredUnusedDeclaredDependency>
</ignoredUnusedDeclaredDependencies>
<ignoredDependencies>
<ignoredDependency>com.google.guava*</ignoredDependency>
<ignoredDependency>io.cucumber*</ignoredDependency>
<ignoredDependency>org.junit*</ignoredDependency>
<ignoredDependency>com.google.code.findbugs*</ignoredDependency>
<ignoredDependency>com.github.spotbugs*</ignoredDependency>
<ignoredDependency>uk.org.lidalia:lidalia-slf4j-ext:*</ignoredDependency>
</ignoredDependencies>
</configuration>
</plugin>

<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
Expand All @@ -155,6 +213,10 @@
<argLine>
${surefireArgLine}
</argLine>
<excludes>
<!-- tests to exclude -->
<exclude>${testExclusions}</exclude>
</excludes>
</configuration>
</plugin>

Expand Down Expand Up @@ -224,7 +286,7 @@
</rule>
</rules>
</configuration>
</execution>
</execution>

</executions>
</plugin>
Expand Down Expand Up @@ -257,6 +319,7 @@
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
Expand Down Expand Up @@ -372,6 +435,61 @@
</plugins>
</build>

<profiles>
<profile>
<!-- this profile handles running the flagd integration tests -->
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pulls the test-harness gherkin repo, and copies it into place.

<id>integration-test</id>
<properties>
<!-- run the integration tests by clearing the exclusions -->
<testExclusions></testExclusions>
</properties>
<build>
<plugins>
<!-- pull the gherkin tests as a git submodule -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<id>update-test-harness-submodule</id>
<phase>validate</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<!-- run: git submodule update \-\-init \-\-recursive -->
<executable>git</executable>
<arguments>
<argument>submodule</argument>
<argument>update</argument>
<argument>--init</argument>
<argument>--recursive</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>copy-gherkin-tests</id>
<phase>validate</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<!-- run: cp test-harness/features/evaluation.feature src/test/resources/features/ -->
<executable>cp</executable>
<arguments>
<argument>test-harness/features/evaluation.feature</argument>
<argument>src/test/resources/features/</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>

<distributionManagement>
<snapshotRepository>
<id>ossrh</id>
Expand Down
16 changes: 16 additions & 0 deletions src/test/java/dev/openfeature/javasdk/integration/RunCucumberTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package dev.openfeature.javasdk.integration;

import org.junit.platform.suite.api.ConfigurationParameter;
import org.junit.platform.suite.api.IncludeEngines;
import org.junit.platform.suite.api.SelectClasspathResource;
import org.junit.platform.suite.api.Suite;

import static io.cucumber.junit.platform.engine.Constants.PLUGIN_PROPERTY_NAME;

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("features")
@ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "pretty")
public class RunCucumberTest {

}
Loading