Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V8 data migration improvements #3545

Merged
merged 2 commits into from
Dec 19, 2024
Merged

V8 data migration improvements #3545

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5d36dbb
Read rpc endpoints from default config if not in noderc
Dec 19, 2024
0dda627
Update constants for local testing
Dec 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions v8-data-migration/constants.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,15 @@ export const BATCH_SIZE = 50;
// export const NODERC_CONFIG_PATH =
// '/Users/zvonimir/projects/ot-node/v8-data-migration/.origintrail_noderc';
// export const ENV_PATH = '/Users/zvonimir/projects/ot-node/data-migration/.env';
// export const DEFAULT_CONFIG_PATH = '/Users/zvonimir/projects/ot-node/config/config.json';
// export const DATA_MIGRATION_DIR =
// '/Users/zvonimir/projects/ot-node/v8-data-migration/data/data-migration';
// export const MIGRATION_PROGRESS_FILE =
// '/Users/zvonimir/projects/ot-node/v8-data-migration/data/migrations/v8DataMigration';
// // LOCAL TESTING END

// REMOTE
export const DEFAULT_CONFIG_PATH = '/root/ot-node/current/config/config.json';
export const NODERC_CONFIG_PATH = '/root/ot-node/.origintrail_noderc';
export const DATA_MIGRATION_DIR = '/root/ot-node/data/data-migration';
export const ENV_PATH = '/root/ot-node/current/.env';
Expand Down
9 changes: 8 additions & 1 deletion v8-data-migration/v8-data-migration-utils.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import path from 'path';
import { stringify } from 'csv-stringify';
import { parse as csvParse } from 'csv-parse';
import readLastLines from 'read-last-lines';
import { NODERC_CONFIG_PATH, MIGRATION_PROGRESS_FILE } from './constants.js';
import { NODERC_CONFIG_PATH, MIGRATION_PROGRESS_FILE, DEFAULT_CONFIG_PATH } from './constants.js';
import { validateSuccessfulInserts, validateConfig } from './validation.js';
import logger from './logger.js';

Expand All @@ -14,6 +14,13 @@ export function initializeConfig() {
return config;
}

export function initializeDefaultConfig() {
const configPath = path.resolve(DEFAULT_CONFIG_PATH);
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
validateConfig(config);
return config;
}

export async function getHighestTokenId(csvFilePath) {
try {
const lastLine = await readLastLines.read(csvFilePath, 1); // Read the last line
Expand Down
11 changes: 9 additions & 2 deletions v8-data-migration/v8-data-migration.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import {
updateCsvFile,
initializeConfig,
initializeDefaultConfig,
getCsvDataStream,
getHighestTokenId,
ensureDirectoryExists,
Expand Down Expand Up @@ -239,9 +240,12 @@
ensureDirectoryExists(DATA_MIGRATION_DIR);
// REMOTE END

// initialize node config
// initialize noderc config
const config = initializeConfig();

// initialize default config
const defaultConfig = initializeDefaultConfig();

// Initialize blockchain config
const blockchainConfig = config.modules.blockchain;
if (!blockchainConfig || !blockchainConfig.implementation) {
Expand Down Expand Up @@ -291,7 +295,10 @@
logger.info(`Blockchain ${blockchain} is not enabled. Skipping...`);
continue;
}
const rpcEndpoints = blockchainImplementation?.config?.rpcEndpoints;
const rpcEndpoints = blockchainImplementation?.config?.rpcEndpoints
? blockchainImplementation.config.rpcEndpoints
: defaultConfig[process.env.NODE_ENV].modules.blockchain.implementation[blockchain]
.config.rpcEndpoints;
if (!Array.isArray(rpcEndpoints) || rpcEndpoints.length === 0) {
throw new Error(`RPC endpoints are not defined for blockchain ${blockchain}.`);
}
Expand Down Expand Up @@ -384,7 +391,7 @@
logger.info(`Amount of tokenIds left to process: ${tokenIdsToProcessCount}`);

// Iterate through the csv data and push to triple store until all data is processed
while (true) {

Check warning on line 394 in v8-data-migration/v8-data-migration.js

View workflow job for this annotation

GitHub Actions / lint 

Unexpected constant condition
logger.time('GETTING THE NEW CSV DATA BATCH');
const next = await csvDataStream.next();
logger.timeEnd('GETTING THE NEW CSV DATA BATCH');
Expand Down
Loading