fix(EMS-3524): no pdf - data migration - account, application relatio…

…nships (#2652)

* fix(EMS-3524): no pdf - data migration - account relationships

* fix(EMS-3524): fix typos

* fix(EMS-3524): no pdf - data migration - application relationships

* fix(EMS-3524): no pdf - data migraiton - createCuid function

* fix(EMS-3524): no pdf - data migration export contract relationships

* chore(EMS-3524): remove unused function

* fix(EMS-3524): no pdf - data migration - loss payee relationships

* chore(EMS-3524): no pdf - data migration - clean up

* chore(schema): update creditPeriodWithBuyer keystone varchar

* fix(EMS-3524): documentation improvements

* fix(EMS-3524): fix typos

* fix(EMS-3524): address todo comments

* fix(EMS-3524): no pdf - data migration - missing fields/relationships

* fix(EMS-3524): update README.md

* fix(EMS-3524): no pdf - data migration - default currency codes

* chore(ui): application mapping try/catch handlers

* fix(EMS-3524): fix typos

* fix(EMS-3524): fix loss payee relationship issues

src/api/.env.sample

@@ -11,6 +11,7 @@ DATABASE_URL=
 
 # DATABASE - data migration
 DATABASE_USER=
+CUID_FINGERPRINT=
 
 # API
 API_URL=

src/api/data-migration/version-1-to-version-2/README.md

@@ -40,6 +40,11 @@ This directory contains source code for migrating version 1 of EXIP data into th
 - NominatedLossPayee
 - PrivateMarket
 
+Note - there are 2 new fields that are created with a default currency code (GBP). Otherwise, any applications that are half complete will error when a user tries to continue the application:
+
+1. BuyerTradingHistory table - `currencyCode` column.
+2. Business table - `turnoverCurrencyCode` column.
+
 ## Prerequisites :gear:
 
 To set up and run the API locally, you'll need the following prerequisites:
@@ -48,9 +53,9 @@ To set up and run the API locally, you'll need the following prerequisites:
 - A MySQL database with the version 1 (MVP) data structure.
 - An operational API (parent directory - see the API's README).
 - The `DATABASE_URL` environment variable should be configured to point to your local MySQL database, for example: `mysql://root:@localhost:1234/db-name`.
-- The local `NODE_ENV` environment variable set to `migration`.
-- The local `DATABASE_USER` environment variable set to the database's user.
-- The local `DATABASE_PASSWORD` environment variable set to the database's password.
+- The local `DATABASE_USER` environment variable
+- The local `DATABASE_PASSWORD` environment variable
+- The local `CUID_FINGERPRINT` environment variable
 - `mysql2` NPM package installed as an API dependency.
 - `ts-node` NPM package installed locally.
 
@@ -72,19 +77,30 @@ The migration should successfully do the following:
 9. Create new application relationships.
 10. Exit the process.
 
-## SQL and KeystoneJS queries
+## How to ensure that data migration was successful
+
+1. All user accounts should have an AccountStatus table.
+2. All applications should be aligned with the version 2 data model (listed above).
+3. In the UI, all existing accounts work as expected (sign in, suspension etc)
+4. In the UI, all existing applications with a status of "in progress" can be progressed and successfully submitted.
+
+:warning: After running the migration script, `npm run dev` in the API will fail. `npm run start` should be used instead.
 
-The data migration uses a combination of raw SQL queries and KeystoneJS context queries.
+This is because, during `npm run dev`, KeystoneJS/prisma checks the schema against the database. It will then attempt to automatically build the database, with the latest schema. After running the migration script, this will fail because KeystoneJS/prisma attempts to create foreign key constraints that already exist.
+
+We manage our own data migration, so we do not need these checks to run.
+
+To run `npm run dev` after running the migration script, it can be achieved by adding a `--no-db-push` to the command. However, this should not be necessary since this is for development environments only. In a data migration scenario, `npm run start` should be used.
+
+## SQL and KeystoneJS queries
 
 In many instances, we need to obtain certain pieces of data that are currently stored in the database, and move these to another place.
 
 If the database and the KeystoneJS schema are out of sync (as it would be prior to running the migration script), KeystoneJS will not return all the data. For example, if field X is in the database, but it's been moved in the KeystoneJS/GraphQL schema, the GraphQL query will simply not return field X, because it is no longer in the schema.
 
 Therefore, it is not possible to use KeystoneJS context queries to obtain version 1 data in the database, whilst executing data migration.
 
-When KeystoneJS context queries cannot be used, we use the `mysql2` NPM package to execute raw database queries and create new tables, fields etc.
-
-KeystoneJS context queries can however be used to obtain data that has _not_ changed, but most importantly, the KeystoneJS context queries can be used to very easily create new relationships. This is used extensively in various migration functions.
+Therefore, we use raw SQL queries via the `mysql2` NPM package to obtain data, move data, create new tables and fields etc.
 
 ## What happens to applications that are in progress :microscope:
 

src/api/data-migration/version-1-to-version-2/create-cuid/index.ts

@@ -0,0 +1,17 @@
+import dotenv from 'dotenv';
+import { init } from '@paralleldrive/cuid2';
+
+dotenv.config();
+
+/**
+ * createCuid
+ * Create a CUID.
+ * This is only used for data migration
+ */
+const createCuid = init({
+  random: Math.random,
+  length: 25,
+  fingerprint: process.env.CUID_FINGERPRINT,
+});
+
+export default createCuid;

src/api/data-migration/version-1-to-version-2/create-new-account-status-relationships/account-status-columns.ts

@@ -0,0 +1,35 @@
+import { Connection } from 'mysql2/promise';
+import executeSqlQuery from '../execute-sql-query';
+import { AccountStatus } from '../../../types';
+
+/**
+ * updateAccountStatusColumns
+ * Update "account status" columns for all existing accounts.
+ * 1) Map over each account status entry.
+ * 2) Update the Account table to have a status ID/relationship.
+ * @param {Connection} connection: SQL database connection
+ * @param {Array<AccountStatus>} statusRows: Account status entries
+ * @param {Array<AccountStatus>} accountStatuses: Account statuses
+ * @returns {Promise<Array<object>>} executeSqlQuery response
+ */
+const updateAccountStatusColumns = async (connection: Connection, statusRows: Array<AccountStatus>, accountStatuses: Array<AccountStatus>) => {
+  console.info('✅ Updating account status columns for all accounts (account table)');
+
+  const accountPromises = statusRows.map(async (accountStatus: AccountStatus, index: number) => {
+    const status = accountStatuses[index];
+
+    const loggingMessage = `Updating status column in account table for account status ${accountStatus.id}`;
+
+    const query = `
+      UPDATE Account SET status='${status.id}' WHERE id='${accountStatus.id}'
+    `;
+
+    const updated = await executeSqlQuery({ connection, query, loggingMessage });
+
+    return updated;
+  });
+
+  return Promise.all(accountPromises);
+};
+
+export default updateAccountStatusColumns;

src/api/data-migration/version-1-to-version-2/create-new-account-status-relationships/account-status-rows.ts

@@ -0,0 +1,33 @@
+import { format } from 'date-fns';
+import { Connection } from 'mysql2/promise';
+import executeSqlQuery from '../execute-sql-query';
+import { AccountMvp } from '../../../types';
+
+/**
+ * createAccountStatusRows
+ * Create new "account status" rows for all existing accounts.
+ * 1) Map over each account entry.
+ * 2) Insert status data into the AccountStatus table - using isVerified, isBlocked and updatedAt from the original accounts data.
+ * @param {Connection} connection: SQL database connection
+ * @param {Array<AccountMvp>} accounts: MVP accounts
+ * @returns {Promise<Array<object>>} executeSqlQuery responses
+ */
+const createAccountStatusRows = async (connection: Connection, accounts: Array<AccountMvp>) => {
+  console.info('✅ Creating account status rows for all accounts (account status table)');
+
+  const accountStatusData = accounts.map(
+    (account: AccountMvp) => `('${account.id}', ${account.isBlocked}, ${account.isVerified}, '${format(account.updatedAt, 'yyyy-MM-dd HH:mm')}')`,
+  );
+
+  const loggingMessage = 'Creating new status rows for all accounts';
+
+  const query = `
+    INSERT INTO AccountStatus (id, isBlocked, isVerified, updatedAt) VALUES ${accountStatusData};
+  `;
+
+  await executeSqlQuery({ connection, query, loggingMessage });
+
+  return accounts;
+};
+
+export default createAccountStatusRows;

src/api/data-migration/version-1-to-version-2/create-new-account-status-relationships/index.ts

@@ -1,15 +1,16 @@
-import { format } from 'date-fns';
 import { Connection } from 'mysql2/promise';
+import createAccountStatusRows from './account-status-rows';
+import getAllAccountStatuses from '../get-all-account-statuses';
+import updateAccountStatusColumns from './account-status-columns';
 import getAllAccounts from '../get-all-accounts';
-import executeSqlQuery from '../execute-sql-query';
-import { AccountMvp } from '../../../types';
 
 /**
  * createNewAccountStatusRelationships
  * Create new "account status" relationships for all existing accounts.
  * 1) Get all accounts
- * 2) Create an array of "account status" data - using isVerified and isBlocked from the original accounts data.
- * 3) Create new "account status" entries.
+ * 2) Create account status rows/entries in the "AccountStatus" table
+ * 3) Get all account statuses
+ * 4) Update status ID columns in the "Account" table
  * @param {Connection} connection: SQL database connection
  * @returns {Promise<Boolean>}
  */
@@ -21,15 +22,11 @@ const createNewAccountStatusRelationships = async (connection: Connection): Prom
   try {
     const accounts = await getAllAccounts(connection);
 
-    const accountStatusData = accounts.map(
-      (account: AccountMvp) => `('${account.id}', ${account.isBlocked}, ${account.isVerified}, '${format(account.updatedAt, 'yyyy-MM-dd HH:mm')}')`,
-    );
+    const statusRows = await createAccountStatusRows(connection, accounts);
 
-    const query = `
-      INSERT INTO AccountStatus (id, isBlocked, isVerified, updatedAt) VALUES ${accountStatusData};
-    `;
+    const accountStatuses = await getAllAccountStatuses(connection);
 
-    await executeSqlQuery({ connection, query, loggingMessage });
+    await updateAccountStatusColumns(connection, statusRows, accountStatuses);
 
     return true;
   } catch (err) {

src/api/data-migration/version-1-to-version-2/create-new-application-relationships/create-company-different-trading-address.ts

@@ -0,0 +1,46 @@
+import { Connection } from 'mysql2/promise';
+import createCuid from '../create-cuid';
+import executeSqlQuery from '../execute-sql-query';
+import { Application } from '../../../types';
+
+/**
+ * createCompanyDifferentTradingAddress
+ * Create new "company different trading address" entries with company relationships.
+ * 1) Map over each application.
+ * 2) Generate "different trading address" values.
+ * 3) Insert the values into the CompanyDifferentTradingAddress table.
+ * @param {Connection} connection: SQL database connection
+ * @param {Array<Application>} applications: Applications
+ * @returns {Promise<Array<object>>} executeSqlQuery responses
+ */
+const createCompanyDifferentTradingAddress = async (connection: Connection, applications: Array<Application>) => {
+  const loggingMessage = 'Creating companyDifferentTradingAddresses entries with company relationships';
+
+  console.info(`✅ ${loggingMessage}`);
+
+  try {
+    const promises = applications.map(async (application: Application) => {
+      const theValues = `('${createCuid()}', '${application.company}')`;
+
+      const query = `
+        INSERT INTO CompanyDifferentTradingAddress (id, company) VALUES ${theValues};
+      `;
+
+      const updated = await executeSqlQuery({
+        connection,
+        query,
+        loggingMessage: `Creating CompanyDifferentTradingAddress entry for application ${application.id}`,
+      });
+
+      return updated;
+    });
+
+    return Promise.all(promises);
+  } catch (err) {
+    console.error(`🚨 error ${loggingMessage} %O`, err);
+
+    throw new Error(`🚨 error ${loggingMessage} ${err}`);
+  }
+};
+
+export default createCompanyDifferentTradingAddress;

src/api/data-migration/version-1-to-version-2/create-new-application-relationships/export-contract-agent/create-agent-service-charges.ts

@@ -1,27 +1,46 @@
-import { Context } from '.keystone/types'; // eslint-disable-line
+import { Connection } from 'mysql2/promise';
+import createCuid from '../../create-cuid';
+import executeSqlQuery from '../../execute-sql-query';
+import { Application } from '../../../../types';
 
 /**
  * createAgentServiceCharges
- * Create new "export contract agent service charge" entires
- * @param {Context} context: KeystoneJS context API
- * @param {Array<object>} agentServiceIdsConnectArray: Array of agent service IDs "connect" objects
- * @returns {Promise<Array<ApplicationExportContractAgentServiceCharge>>} Export contract agent service charge entries
+ * Create new "export contract agent service charges" entries
+ * 1) Map over each application.
+ * 2) Generate "agent service charge" values (CUID)
+ * 3) Insert the values into the ExportContractAgentServiceCharge table.
+ * @param {Connection} connection: SQL database connection
+ * @param {Array<Application>} applications: Applications
+ * @returns {Promise<Array<object>>} executeSqlQuery responses
  */
-const createAgentServiceCharges = async (context: Context, agentServiceIdsConnectArray: Array<object>) => {
-  const loggingMessage = 'Creating exportContractAgentServiceCharges with agent service relationships';
+const createAgentServiceCharges = async (connection: Connection, applications: Array<Application>) => {
+  const loggingMessage = 'Creating exportContract agent service charges';
 
   console.info(`✅ ${loggingMessage}`);
 
   try {
-    const created = await context.db.ExportContractAgentServiceCharge.createMany({
-      data: agentServiceIdsConnectArray,
+    const servicesPromises = applications.map(async (application: Application) => {
+      const theValues = `('${createCuid()}')`;
+
+      const query = `
+        INSERT INTO ExportContractAgentServiceCharge (id) VALUES ${theValues};
+      `;
+
+      const updated = await executeSqlQuery({
+        connection,
+        query,
+        loggingMessage: `Creating ExportContractAgentServiceCharge entry for application ${application.id}`,
+      });
+
+      return updated;
     });
 
-    return created;
+    return Promise.all(servicesPromises);
   } catch (err) {
     console.error(`🚨 error ${loggingMessage} %O`, err);
 
     throw new Error(`🚨 error ${loggingMessage} ${err}`);
   }
 };
+
 export default createAgentServiceCharges;