Skip to content
This repository has been archived by the owner on Dec 26, 2022. It is now read-only.

feat(db): Implement importer and listener for permanode #631

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

YingHan-Chen
Copy link
Contributor

The listener subscribes newly confirmed transactions from IRI and adds inserting tasks into the task queue of thread pool.

The importer reads confirmed transactions from historical transaction files dumped from IRI and adds inserting tasks into the task queue of thread pool.

Each worker in the thread pool establishes a session connected to ScyllaDB cluster and takes inserting tasks.

storage/scylladb_permanode_thpool.c Outdated Show resolved Hide resolved
docs/permanode.md Outdated Show resolved Hide resolved
docs/permanode.md Outdated Show resolved Hide resolved
The listener subscribes newly confirmed transactions from IRI and adds inserting tasks into the task queue of thread pool.

The importer reads confirmed transactions from historical transaction files dumped from IRI and adds inserting tasks into the task queue of thread pool.

Each worker in the thread pool establishes a session connected to ScyllaDB cluster and takes inserting tasks.
@YingHan-Chen YingHan-Chen added A-storage Area - Storage C-feature Category - feature C-doc Category - Document labels May 21, 2020

static status_t init_importer_data(db_importer_thread_t* thread_data, db_permanode_pool_t* pool, char* file_list) {
status_t ret = SC_OK;
pthread_mutex_init(&thread_data->thread_mutex, NULL);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need error checking here?

status_t ret = SC_OK;
pthread_mutex_init(&thread_data->thread_mutex, NULL);
thread_data->pool = pool;
thread_data->file_path = strdup(file_list);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto


status_t ret = SC_OK;
db_importer_thread_t* thread_data = (db_importer_thread_t*)data;
pthread_mutex_lock(&thread_data->thread_mutex);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

ta_log_error("Failed to open file %s\n", file_name_buffer);
goto exit;
}
ta_log_info("%s %s\n", "starting to import file : ", file_name_buffer);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will get an extra space after the colon


exit:
if (ret == SC_OK) {
ta_log_info("%s %s\n", "Successfully import file : ", thread_data->file_path);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

worker_threads = malloc(thread_num * sizeof(pthread_t));
worker_data = malloc(thread_num * sizeof(db_worker_thread_t));

db_permanode_thpool_init(&pool);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

db_permanode_thpool_init_worker(worker_data + i, &pool, db_host);
pthread_create(&worker_threads[i], NULL, db_permanode_worker_handler, (void*)&worker_data[i]);
}
init_importer_data(&importer_data, &pool, file_path);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

init_importer_data(&importer_data, &pool, file_path);
pthread_create(&importer_thread, NULL, (void*)importer_handler, (void*)&importer_data);

pthread_join(importer_thread, NULL);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

*
* @param[in] data pointer to db_permanode_pool_t
*
* @return NULL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no returning NULL here

#include "scylladb_identity.h"
#include "scylladb_permanode.h"

#include "scylladb_permanode_thpool.h"
/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a new line between line 15 and 16

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A-storage Area - Storage C-doc Category - Document C-feature Category - feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants