Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support testing with Postgres and MySQL for user management backend #2886

Merged
merged 64 commits into from
Mar 4, 2022
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
14c7efe
:card_file_box: Fix Postgres migrations
ivov Feb 24, 2022
46ed29b
:zap: Add DB-specific scripts
ivov Feb 24, 2022
26e7dd5
:sparkles: Set up test connections
ivov Feb 24, 2022
f7c1ec7
:zap: Add Postgres UUID check
ivov Feb 24, 2022
f815599
:test_tube: Make test adjustments for Postgres
ivov Feb 24, 2022
bb20508
:twisted_rightwards_arrows: Merge parent branch
ivov Feb 25, 2022
e8b9f67
:zap: Refactor connection logic
ivov Feb 25, 2022
47f6e65
:sparkles: Set up double init for Postgres
ivov Feb 25, 2022
bbdc9ff
:twisted_rightwards_arrows: Merge parent branch
ivov Feb 28, 2022
8a32a2a
:pencil2: Add TODOs
ivov Feb 28, 2022
dff5dd4
:zap: Refactor DB dropping logic
ivov Feb 28, 2022
ea3baec
:sparkles: Implement global teardown
ivov Feb 28, 2022
d36efac
:sparkles: Create TypeORM wrappers
ivov Feb 28, 2022
c12f42a
:sparkles: Initial MySQL setup
ivov Feb 28, 2022
9e47322
:zap: Clean up Postgres connection options
ivov Feb 28, 2022
daf0320
:zap: Simplify by sharing bootstrap connection name
ivov Feb 28, 2022
6447612
:card_file_box: Fix MySQL migrations
ivov Feb 28, 2022
fb53cb4
:fire: Remove comments
ivov Feb 28, 2022
1acbadd
:zap: Use ES6 imports
ivov Mar 1, 2022
6ace92c
:fire: Remove outdated comments
ivov Mar 1, 2022
727146a
:zap: Centralize bootstrap connection name handles
ivov Mar 1, 2022
e67fbc0
:zap: Centralize database types
ivov Mar 1, 2022
529ec60
:pencil2: Update comment
ivov Mar 1, 2022
9ffebcb
:truck: Rename `findRepository`
ivov Mar 1, 2022
510284b
:construction: Attempt to truncate MySQL
ivov Mar 1, 2022
dec005b
:sparkles: Implement creds router
ivov Mar 1, 2022
0531ccc
:twisted_rightwards_arrows: Merge parent branch
ivov Mar 1, 2022
ce3f50f
:bug: Fix duplicated MySQL bootstrap
ivov Mar 1, 2022
fbeaef8
:bug: Fix misresolved merge conflict
ivov Mar 1, 2022
3ede58f
:card_file_box: Fix tags migration
ivov Mar 1, 2022
e4c7185
:card_file_box: Fix MySQL UM migration
ivov Mar 1, 2022
8967982
:bug: Fix MySQL parallelization issues
ivov Mar 1, 2022
c5eaeb6
:blue_book: Augment TypeORM to prevent error
ivov Mar 1, 2022
0e32cc5
:fire: Remove comments
ivov Mar 1, 2022
1f48709
:sparkles: Support one sqlite DB per suite run
ivov Mar 2, 2022
6871d81
:truck: Move `testDb` to own module
ivov Mar 2, 2022
7f8c3bf
:fire: Deduplicate bootstrap Postgres logic
ivov Mar 2, 2022
80a3dcd
:fire: Remove unneeded comment
ivov Mar 2, 2022
64408df
:zap: Make logger init calls consistent
ivov Mar 2, 2022
67a5376
:pencil2: Improve comment
ivov Mar 2, 2022
ae3bdea
:pencil2: Add dividers
ivov Mar 2, 2022
ca69bdf
:art: Improve formatting
ivov Mar 2, 2022
3a7fb21
:fire: Remove duplicate MySQL global setting
ivov Mar 2, 2022
b89a604
:truck: Move comment
ivov Mar 2, 2022
086d3e8
:zap: Update default test script
ivov Mar 2, 2022
ff1799d
:fire: Remove unneeded helper
ivov Mar 2, 2022
df084ff
:zap: Unmarshal answers from Postgres
ivov Mar 2, 2022
9e49aa8
:bug: Phase out `isTestRun`
ivov Mar 2, 2022
e89fdb0
:zap: Refactor `isEmailSetup`
ivov Mar 2, 2022
729cc1b
:fire: Remove unneeded imports
ivov Mar 3, 2022
1b2a494
:zap: Handle bootstrap connection errors
ivov Mar 3, 2022
66b1807
:twisted_rightwards_arrows: Merge parent branch
ivov Mar 3, 2022
760d52c
:fire: Remove unneeded imports
ivov Mar 3, 2022
ca26f0f
:twisted_rightwards_arrows: Merge parent branch
ivov Mar 3, 2022
cf830ed
:fire: Remove outdated comments
ivov Mar 3, 2022
1a6a563
:pencil2: Fix typos
ivov Mar 4, 2022
7e352c8
:truck: Relocate `answersFormatter`
ivov Mar 4, 2022
93f31b7
:rewind: Undo package.json miscommit
ivov Mar 4, 2022
d115531
:fire: Remove unneeded import
ivov Mar 4, 2022
59e65e7
:zap: Refactor test DB prefixing
ivov Mar 4, 2022
3201249
:zap: Add no-leftover check to MySQL
ivov Mar 4, 2022
1007364
:package: Update package.json
ivov Mar 4, 2022
d1faf14
:zap: Autoincrement on simulated MySQL truncation
ivov Mar 4, 2022
32085e8
:fire: Remove debugging queries
ivov Mar 4, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@
"start:default": "cd bin && ./n8n",
"start:windows": "cd bin && n8n",
"test": "jest",
"test:sqlite": "export DB_TYPE=sqlite && jest",
"test:postgres": "export DB_TYPE=postgresdb && jest",
"watch": "tsc --watch",
"typeorm": "ts-node ../../node_modules/typeorm/cli.js"
},
Expand Down
188 changes: 100 additions & 88 deletions packages/cli/src/Db.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,13 @@
/* eslint-disable no-case-declarations */
/* eslint-disable @typescript-eslint/naming-convention */
import { UserSettings } from 'n8n-core';
import { ConnectionOptions, createConnection, getRepository, LoggerOptions } from 'typeorm';
import {
Connection,
ConnectionOptions,
createConnection,
getRepository,
LoggerOptions,
} from 'typeorm';
import { TlsOptions } from 'tls';
import * as path from 'path';
// eslint-disable-next-line import/no-cycle
Expand All @@ -18,8 +24,6 @@ import { entities } from './databases/entities';
import { postgresMigrations } from './databases/postgresdb/migrations';
import { mysqlMigrations } from './databases/mysqldb/migrations';
import { sqliteMigrations } from './databases/sqlite/migrations';
import { TEST_CONNECTION_OPTIONS } from '../test/integration/shared/constants';
import { isTestRun } from '../test/integration/shared/utils';

export const collections: IDatabaseCollections = {
Credentials: null,
Expand All @@ -34,82 +38,91 @@ export const collections: IDatabaseCollections = {
Settings: null,
};

export async function init(): Promise<IDatabaseCollections> {
export async function init(
testConnectionOptions?: ConnectionOptions,
): Promise<IDatabaseCollections> {
const dbType = (await GenericHelpers.getConfigValue('database.type')) as DatabaseType;
const n8nFolder = UserSettings.getUserN8nFolderPath();

let connectionOptions: ConnectionOptions;
let connection: Connection;

const entityPrefix = config.get('database.tablePrefix');

switch (dbType) {
case 'postgresdb':
const sslCa = (await GenericHelpers.getConfigValue('database.postgresdb.ssl.ca')) as string;
const sslCert = (await GenericHelpers.getConfigValue(
'database.postgresdb.ssl.cert',
)) as string;
const sslKey = (await GenericHelpers.getConfigValue('database.postgresdb.ssl.key')) as string;
const sslRejectUnauthorized = (await GenericHelpers.getConfigValue(
'database.postgresdb.ssl.rejectUnauthorized',
)) as boolean;

let ssl: TlsOptions | undefined;
if (sslCa !== '' || sslCert !== '' || sslKey !== '' || !sslRejectUnauthorized) {
ssl = {
ca: sslCa || undefined,
cert: sslCert || undefined,
key: sslKey || undefined,
rejectUnauthorized: sslRejectUnauthorized,
if (testConnectionOptions) {
connectionOptions = testConnectionOptions;
} else {
switch (dbType) {
case 'postgresdb':
const sslCa = (await GenericHelpers.getConfigValue('database.postgresdb.ssl.ca')) as string;
const sslCert = (await GenericHelpers.getConfigValue(
'database.postgresdb.ssl.cert',
)) as string;
const sslKey = (await GenericHelpers.getConfigValue(
'database.postgresdb.ssl.key',
)) as string;
const sslRejectUnauthorized = (await GenericHelpers.getConfigValue(
'database.postgresdb.ssl.rejectUnauthorized',
)) as boolean;

let ssl: TlsOptions | undefined;
if (sslCa !== '' || sslCert !== '' || sslKey !== '' || !sslRejectUnauthorized) {
ssl = {
ca: sslCa || undefined,
cert: sslCert || undefined,
key: sslKey || undefined,
rejectUnauthorized: sslRejectUnauthorized,
};
}

connectionOptions = {
type: 'postgres',
entityPrefix,
database: (await GenericHelpers.getConfigValue('database.postgresdb.database')) as string,
host: (await GenericHelpers.getConfigValue('database.postgresdb.host')) as string,
password: (await GenericHelpers.getConfigValue('database.postgresdb.password')) as string,
port: (await GenericHelpers.getConfigValue('database.postgresdb.port')) as number,
username: (await GenericHelpers.getConfigValue('database.postgresdb.user')) as string,
schema: config.get('database.postgresdb.schema'),
migrations: postgresMigrations,
migrationsRun: true,
migrationsTableName: `${entityPrefix}migrations`,
ssl,
};
}

connectionOptions = {
type: 'postgres',
entityPrefix,
database: (await GenericHelpers.getConfigValue('database.postgresdb.database')) as string,
host: (await GenericHelpers.getConfigValue('database.postgresdb.host')) as string,
password: (await GenericHelpers.getConfigValue('database.postgresdb.password')) as string,
port: (await GenericHelpers.getConfigValue('database.postgresdb.port')) as number,
username: (await GenericHelpers.getConfigValue('database.postgresdb.user')) as string,
schema: config.get('database.postgresdb.schema'),
migrations: postgresMigrations,
migrationsRun: true,
migrationsTableName: `${entityPrefix}migrations`,
ssl,
};

break;

case 'mariadb':
case 'mysqldb':
connectionOptions = {
type: dbType === 'mysqldb' ? 'mysql' : 'mariadb',
database: (await GenericHelpers.getConfigValue('database.mysqldb.database')) as string,
entityPrefix,
host: (await GenericHelpers.getConfigValue('database.mysqldb.host')) as string,
password: (await GenericHelpers.getConfigValue('database.mysqldb.password')) as string,
port: (await GenericHelpers.getConfigValue('database.mysqldb.port')) as number,
username: (await GenericHelpers.getConfigValue('database.mysqldb.user')) as string,
migrations: mysqlMigrations,
migrationsRun: true,
migrationsTableName: `${entityPrefix}migrations`,
timezone: 'Z', // set UTC as default
};
break;

case 'sqlite':
connectionOptions = {
type: 'sqlite',
database: path.join(n8nFolder, 'database.sqlite'),
entityPrefix,
migrations: sqliteMigrations,
migrationsRun: false, // migrations for sqlite will be ran manually for now; see below
migrationsTableName: `${entityPrefix}migrations`,
};
break;

default:
throw new Error(`The database "${dbType}" is currently not supported!`);

break;

case 'mariadb':
case 'mysqldb':
connectionOptions = {
type: dbType === 'mysqldb' ? 'mysql' : 'mariadb',
database: (await GenericHelpers.getConfigValue('database.mysqldb.database')) as string,
entityPrefix,
host: (await GenericHelpers.getConfigValue('database.mysqldb.host')) as string,
password: (await GenericHelpers.getConfigValue('database.mysqldb.password')) as string,
port: (await GenericHelpers.getConfigValue('database.mysqldb.port')) as number,
username: (await GenericHelpers.getConfigValue('database.mysqldb.user')) as string,
migrations: mysqlMigrations,
migrationsRun: true,
migrationsTableName: `${entityPrefix}migrations`,
timezone: 'Z', // set UTC as default
};
break;

case 'sqlite':
connectionOptions = {
type: 'sqlite',
database: path.join(n8nFolder, 'database.sqlite'),
entityPrefix,
migrations: sqliteMigrations,
migrationsRun: false, // migrations for sqlite will be ran manually for now; see below
migrationsTableName: `${entityPrefix}migrations`,
};
break;

default:
throw new Error(`The database "${dbType}" is currently not supported!`);
}
}

let loggingOption: LoggerOptions = (await GenericHelpers.getConfigValue(
Expand All @@ -128,10 +141,6 @@ export async function init(): Promise<IDatabaseCollections> {
}
}

if (isTestRun) {
connectionOptions = TEST_CONNECTION_OPTIONS;
}

Object.assign(connectionOptions, {
entities: Object.values(entities),
synchronize: false,
Expand All @@ -141,9 +150,9 @@ export async function init(): Promise<IDatabaseCollections> {
)) as string,
});

let connection = await createConnection(connectionOptions);
connection = await createConnection(connectionOptions);

if (dbType === 'sqlite') {
if (!testConnectionOptions && dbType === 'sqlite') {
// This specific migration changes database metadata.
// A field is now nullable. We need to reconnect so that
// n8n knows it has changed. Happens only on sqlite.
Expand All @@ -169,17 +178,20 @@ export async function init(): Promise<IDatabaseCollections> {
}
}

collections.Credentials = getRepository(entities.CredentialsEntity);
collections.Execution = getRepository(entities.ExecutionEntity);
collections.Workflow = getRepository(entities.WorkflowEntity);
collections.Webhook = getRepository(entities.WebhookEntity);
collections.Tag = getRepository(entities.TagEntity);

collections.Role = getRepository(entities.Role);
collections.User = getRepository(entities.User);
collections.SharedCredentials = getRepository(entities.SharedCredentials);
collections.SharedWorkflow = getRepository(entities.SharedWorkflow);
collections.Settings = getRepository(entities.Settings);
collections.Credentials = getRepository(entities.CredentialsEntity, testConnectionOptions?.name);
collections.Execution = getRepository(entities.ExecutionEntity, testConnectionOptions?.name);
collections.Workflow = getRepository(entities.WorkflowEntity, testConnectionOptions?.name);
collections.Webhook = getRepository(entities.WebhookEntity, testConnectionOptions?.name);
collections.Tag = getRepository(entities.TagEntity, testConnectionOptions?.name);

collections.Role = getRepository(entities.Role, testConnectionOptions?.name);
collections.User = getRepository(entities.User, testConnectionOptions?.name);
collections.SharedCredentials = getRepository(
entities.SharedCredentials,
testConnectionOptions?.name,
);
collections.SharedWorkflow = getRepository(entities.SharedWorkflow, testConnectionOptions?.name);
collections.Settings = getRepository(entities.Settings, testConnectionOptions?.name);

return collections;
}
11 changes: 11 additions & 0 deletions packages/cli/src/UserManagement/routes/users.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
/* eslint-disable no-restricted-syntax */
/* eslint-disable import/no-cycle */
/* eslint-disable @typescript-eslint/no-non-null-assertion */
import { Response } from 'express';
Expand Down Expand Up @@ -202,6 +203,16 @@ export function usersNamespace(this: N8nApp): void {
throw new ResponseHelper.ResponseError('Invalid payload', undefined, 400);
}

// Postgres validates UUID format
for (const userId of [inviterId, inviteeId]) {
if (!validator.isUUID(userId)) {
Logger.debug('Request to resolve signup token failed because of invalid user ID', {
userId,
});
throw new ResponseHelper.ResponseError('Invalid userId', undefined, 400);
}
}

const users = await Db.collections.User!.find({ where: { id: In([inviterId, inviteeId]) } });

if (users.length !== 2) {
Expand Down
Loading