Skip to content

Commit

Permalink
feat: support grouped test suites and waiting for info.connection (#…
Browse files Browse the repository at this point in the history 
…498)
  • Loading branch information
@AlCalzone
AlCalzone authored May 9, 2022
1 parent 365c4af commit ad70649
Show file tree
Hide file tree
Showing 8 changed files with 401 additions and 261 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@
PLACEHOLDER for next version:
## __WORK IN PROGRESS__
-->
## __WORK IN PROGRESS__
* BREAKING: The function signature of `defineAdditionalTests` in integration tests has changed. All user-defined integration tests must now be grouped in one or more `suite` blocks. The adapter will now only be started at the beginning of each suite. See the documentation for details.
* BREAKING: The function signature of `harness.startAdapterAndWait` has changed. It now accepts a boolean as the first parameter which controls whether to wait for the `alive` state (`false`) or the `info.connection` state (`true`).

## 2.6.0 (2022-04-18)
* The loglevel for the adapter and DB instances is now configurable and defaults to `"debug"` in both cases

Expand Down
58 changes: 30 additions & 28 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,29 +26,6 @@ tests.packageFiles(path.join(__dirname, ".."));
// This should be the adapter's root directory
```

### Adapter startup (Unit test)

**Unit tests for adapter startup were removed and are essentially a no-op now.**
If you defined your own tests, they should still work.

```ts
const path = require("path");
const { tests } = require("@iobroker/testing");

tests.unit(path.join(__dirname, ".."), {
// ~~~~~~~~~~~~~~~~~~~~~~~~~
// This should be the adapter's root directory

// Define your own tests inside defineAdditionalTests.
// If you need predefined objects etc. here, you need to take care of it yourself
defineAdditionalTests() {
it("works", () => {
// see below how these could look like
});
},
});
```

### Adapter startup (Integration test)

Run the following snippet in a `mocha` test file to test the adapter startup process against a real JS-Controller instance:
Expand All @@ -67,13 +44,15 @@ tests.integration(path.join(__dirname, ".."), {
allowedExitCodes: [11],

// Define your own tests inside defineAdditionalTests
// Since the tests are heavily instrumented, you need to create and use a so called "harness" to control the tests.
defineAdditionalTests(getHarness) {
describe("Test sendTo()", () => {
defineAdditionalTests({ suite }) {
// All tests (it, describe) must be grouped in one or more suites. Each suite sets up a fresh environment for the adapter tests.
// At the beginning of each suite, the databases will be reset and the adapter will be started.
// The adapter will run until the end of each suite.

// Since the tests are heavily instrumented, each suite gives access to a so called "harness" to control the tests.
suite("Test sendTo()", (harness) => {
it("Should work", () => {
return new Promise(async (resolve) => {
// Create a fresh harness instance each test!
const harness = getHarness();
// Start the adapter and wait until it has started
await harness.startAdapterAndWait();

Expand All @@ -89,6 +68,29 @@ tests.integration(path.join(__dirname, ".."), {
});
```

### Adapter startup (Unit test)

**Unit tests for adapter startup were removed and are essentially a no-op now.**
If you defined your own tests, they should still work.

```ts
const path = require("path");
const { tests } = require("@iobroker/testing");

tests.unit(path.join(__dirname, ".."), {
// ~~~~~~~~~~~~~~~~~~~~~~~~~
// This should be the adapter's root directory

// Define your own tests inside defineAdditionalTests.
// If you need predefined objects etc. here, you need to take care of it yourself
defineAdditionalTests() {
it("works", () => {
// see below how these could look like
});
},
});
```

### Helper functions for your own tests

Under `utils`, several functions are exposed to use in your own tests:
Expand Down
14 changes: 13 additions & 1 deletion build/tests/integration/index.d.ts
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
/// <reference types="iobroker" />
/// <reference types="mocha" />
import { TestHarness } from "./lib/harness";
export interface TestAdapterOptions {
allowedExitCodes?: (number | string)[];
Expand All @@ -7,6 +8,17 @@ export interface TestAdapterOptions {
/** How long to wait before the adapter startup is considered successful */
waitBeforeStartupSuccess?: number;
/** Allows you to define additional tests */
defineAdditionalTests?: (getHarness: () => TestHarness) => void;
defineAdditionalTests?: (args: TestContext) => void;
}
export interface TestContext {
/** Gives access to the current test harness */
getHarness: () => TestHarness;
/**
* Defines a test suite. At the start of each suite, the adapter will be started with a fresh environment.
* To define tests in each suite, use describe and it as usual.
*/
suite: (name: string, fn: () => void) => void;
describe: Mocha.SuiteFunction;
it: Mocha.TestFunction;
}
export declare function testAdapter(adapterDir: string, options?: TestAdapterOptions): void;
243 changes: 144 additions & 99 deletions build/tests/integration/index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,113 +41,158 @@ function testAdapter(adapterDir, options = {}) {
let dbConnection;
let harness;
const controllerSetup = new controllerSetup_1.ControllerSetup(adapterDir, testDir);
let objectsBackup;
let statesBackup;
let isInSuite = false;
console.log();
console.log(`Running tests in ${testDir}`);
console.log();
describe(`Test the adapter (in a live environment)`, () => {
let objectsBackup;
let statesBackup;
before(async function () {
var _a;
// Installation may take a while - especially if rsa-compat needs to be installed
const oneMinute = 60000;
this.timeout(30 * oneMinute);
if (await controllerSetup.isJsControllerRunning()) {
throw new Error("JS-Controller is already running! Stop it for the first test run and try again!");
}
const adapterSetup = new adapterSetup_1.AdapterSetup(adapterDir, testDir);
// Installation happens in two steps:
// First we need to set up JS Controller, so the databases etc. can be created
// First we need to copy all files and execute an npm install
await controllerSetup.prepareTestDir();
// Only then we can install the adapter, because some (including VIS) try to access
// the databases if JS Controller is installed
await adapterSetup.installAdapterInTestDir();
const dbConnection = new dbConnection_1.DBConnection(appName, testDir, (0, logger_1.createLogger)((_a = options.loglevel) !== null && _a !== void 0 ? _a : "debug"));
await dbConnection.start();
controllerSetup.setupSystemConfig(dbConnection);
await controllerSetup.disableAdminInstances(dbConnection);
await adapterSetup.deleteOldInstances(dbConnection);
await adapterSetup.addAdapterInstance();
await dbConnection.stop();
// Create a copy of the databases that we can restore later
({ objects: objectsBackup, states: statesBackup } =
await dbConnection.backup());
async function prepareTests() {
var _a;
// Installation may take a while - especially if rsa-compat needs to be installed
const oneMinute = 60000;
this.timeout(30 * oneMinute);
if (await controllerSetup.isJsControllerRunning()) {
throw new Error("JS-Controller is already running! Stop it for the first test run and try again!");
}
const adapterSetup = new adapterSetup_1.AdapterSetup(adapterDir, testDir);
// Installation happens in two steps:
// First we need to set up JS Controller, so the databases etc. can be created
// First we need to copy all files and execute an npm install
await controllerSetup.prepareTestDir();
// Only then we can install the adapter, because some (including VIS) try to access
// the databases if JS Controller is installed
await adapterSetup.installAdapterInTestDir();
const dbConnection = new dbConnection_1.DBConnection(appName, testDir, (0, logger_1.createLogger)((_a = options.loglevel) !== null && _a !== void 0 ? _a : "debug"));
await dbConnection.start();
controllerSetup.setupSystemConfig(dbConnection);
await controllerSetup.disableAdminInstances(dbConnection);
await adapterSetup.deleteOldInstances(dbConnection);
await adapterSetup.addAdapterInstance();
await dbConnection.stop();
// Create a copy of the databases that we can restore later
({ objects: objectsBackup, states: statesBackup } =
await dbConnection.backup());
}
async function shutdownTests() {
// Stopping the processes may take a while
this.timeout(30000);
// Stop the controller again
await harness.stopController();
harness.removeAllListeners();
}
async function resetDbAndStartHarness() {
var _a, _b;
this.timeout(30000);
dbConnection = new dbConnection_1.DBConnection(appName, testDir, (0, logger_1.createLogger)((_a = options.loglevel) !== null && _a !== void 0 ? _a : "debug"));
// Clean up before every single test
await Promise.all([
controllerSetup.clearDBDir(),
controllerSetup.clearLogDir(),
dbConnection.restore(objectsBackup, statesBackup),
]);
// Create a new test harness
await dbConnection.start();
harness = new harness_1.TestHarness(adapterDir, testDir, dbConnection);
// Enable the adapter and set its loglevel to the selected one
await harness.changeAdapterConfig(adapterName, {
common: {
enabled: true,
loglevel: (_b = options.loglevel) !== null && _b !== void 0 ? _b : "debug",
},
});
beforeEach(async function () {
var _a, _b;
this.timeout(30000);
dbConnection = new dbConnection_1.DBConnection(appName, testDir, (0, logger_1.createLogger)((_a = options.loglevel) !== null && _a !== void 0 ? _a : "debug"));
// Clean up before every single test
await Promise.all([
controllerSetup.clearDBDir(),
controllerSetup.clearLogDir(),
dbConnection.restore(objectsBackup, statesBackup),
]);
// Create a new test harness
await dbConnection.start();
harness = new harness_1.TestHarness(adapterDir, testDir, dbConnection);
// Enable the adapter and set its loglevel to the selected one
await harness.changeAdapterConfig(adapterName, {
common: {
enabled: true,
loglevel: (_b = options.loglevel) !== null && _b !== void 0 ? _b : "debug",
},
// And enable the sendTo emulation
await harness.enableSendTo();
}
describe(`Adapter integration tests`, () => {
before(prepareTests);
describe("Adapter startup", () => {
beforeEach(resetDbAndStartHarness);
afterEach(shutdownTests);
it("The adapter starts", function () {
var _a;
this.timeout(60000);
const allowedExitCodes = new Set((_a = options.allowedExitCodes) !== null && _a !== void 0 ? _a : []);
// Adapters with these modes are allowed to "immediately" exit with code 0
switch (harness.getAdapterExecutionMode()) {
case "schedule":
case "once":
case "subscribe":
allowedExitCodes.add(0);
}
return new Promise((resolve, reject) => {
// Register a handler to check the alive state and exit codes
harness
.on("stateChange", async (id, state) => {
if (id ===
`system.adapter.${adapterName}.0.alive` &&
state &&
state.val === true) {
// Wait a bit so we can catch errors that do not happen immediately
await (0, async_1.wait)(options.waitBeforeStartupSuccess !=
undefined
? options.waitBeforeStartupSuccess
: 5000);
resolve(`The adapter started successfully.`);
}
})
.on("failed", (code) => {
if (!allowedExitCodes.has(code)) {
reject(new Error(`The adapter startup was interrupted unexpectedly with ${typeof code === "number"
? "code"
: "signal"} ${code}`));
}
else {
// This was a valid exit code
resolve(`The expected ${typeof code === "number"
? "exit code"
: "signal"} ${code} was received.`);
}
});
harness.startAdapter();
}).then((msg) => console.log(msg));
});
// And enable the sendTo emulation
await harness.enableSendTo();
});
afterEach(async function () {
// Stopping the processes may take a while
this.timeout(30000);
// Stop the controller again
await harness.stopController();
harness.removeAllListeners();
});
it("The adapter starts", function () {
var _a;
this.timeout(60000);
const allowedExitCodes = new Set((_a = options.allowedExitCodes) !== null && _a !== void 0 ? _a : []);
// Adapters with these modes are allowed to "immediately" exit with code 0
switch (harness.getAdapterExecutionMode()) {
case "schedule":
case "once":
case "subscribe":
allowedExitCodes.add(0);
}
return new Promise((resolve, reject) => {
// Register a handler to check the alive state and exit codes
harness
.on("stateChange", async (id, state) => {
if (id === `system.adapter.${adapterName}.0.alive` &&
state &&
state.val === true) {
// Wait a bit so we can catch errors that do not happen immediately
await (0, async_1.wait)(options.waitBeforeStartupSuccess != undefined
? options.waitBeforeStartupSuccess
: 5000);
resolve(`The adapter started successfully.`);
}
})
.on("failed", (code) => {
if (!allowedExitCodes.has(code)) {
reject(new Error(`The adapter startup was interrupted unexpectedly with ${typeof code === "number"
? "code"
: "signal"} ${code}`));
}
else {
// This was a valid exit code
resolve(`The expected ${typeof code === "number"
? "exit code"
: "signal"} ${code} was received.`);
}
});
harness.startAdapter();
}).then((msg) => console.log(msg));
});
// Call the user's tests
if (typeof options.defineAdditionalTests === "function") {
options.defineAdditionalTests(() => harness);
const originalIt = global.it;
// Ensure no it() gets called outside of a suite()
function assertSuite() {
if (!isInSuite) {
throw new Error("In user-defined adapter tests, it() must NOT be called outside of a suite()");
}
}
const patchedIt = new Proxy(originalIt, {
apply(target, thisArg, args) {
assertSuite();
return target.apply(thisArg, args);
},
get(target, propKey) {
assertSuite();
return target[propKey];
},
});
describe("User-defined tests", () => {
// patch the global it() function so nobody can bypass the checks
global.it = patchedIt;
const args = {
getHarness: () => harness,
// a test suite is a special describe which sets up and tears down the test environment before and after ALL tests
suite: (name, fn) => {
describe(name, () => {
isInSuite = true;
before(resetDbAndStartHarness);
after(shutdownTests);
fn();
isInSuite = false;
});
},
describe,
it: patchedIt,
};
options.defineAdditionalTests(args);
global.it = originalIt;
});
}
});
}
Expand Down
5 changes: 3 additions & 2 deletions build/tests/integration/lib/harness.d.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ export declare class TestHarness extends EventEmitter {
* @param testDir The directory the integration tests are executed in
*/
constructor(adapterDir: string, testDir: string, dbConnection: DBConnection);
private adapterName;
readonly adapterName: string;
private appName;
private testControllerDir;
private testAdapterDir;
Expand All @@ -48,9 +48,10 @@ export declare class TestHarness extends EventEmitter {
startAdapter(env?: NodeJS.ProcessEnv): Promise<void>;
/**
* Starts the adapter in a separate process and resolves after it has started
* @param waitForConnection By default, the test will wait for the adapter's `alive` state to become true. Set this to `true` to wait for the `info.connection` state instead.
* @param env Additional environment variables to set
*/
startAdapterAndWait(env?: NodeJS.ProcessEnv): Promise<void>;
startAdapterAndWait(waitForConnection?: boolean, env?: NodeJS.ProcessEnv): Promise<void>;
/** Tests if the adapter process is still running */
isAdapterRunning(): boolean;
/** Tests if the adapter process has already exited */
Expand Down
Loading

0 comments on commit ad70649

Please sign in to comment.