Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat : job handler tests #67

Merged
merged 52 commits into from
Aug 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
726f1ed
feat : added tests for increasing coverage
ocdbytes Jul 23, 2024
b3a80b0
feat : added mongo db tests and fixtures and updated the ci for tests
ocdbytes Jul 24, 2024
5826d44
update : removed unwanted fixtures
ocdbytes Jul 24, 2024
0e70fff
update : removed unwanted fixtures
ocdbytes Jul 24, 2024
97f1f21
update : added mongo db runner in ci
ocdbytes Jul 24, 2024
a64613c
update : added mongo db runner in ci
ocdbytes Jul 24, 2024
db472d5
update : added mongo db runner in ci
ocdbytes Jul 24, 2024
15541ef
update : updated with new changes and ci
ocdbytes Jul 25, 2024
fc7d493
update : updated test cases for s3 client
ocdbytes Jul 26, 2024
9a101ba
update : added .env.test file in the commit
ocdbytes Jul 26, 2024
906a1eb
feat : added database necessary tests
ocdbytes Jul 26, 2024
65d66e6
feat : added database necessary tests
ocdbytes Jul 26, 2024
20d82c3
Revert "feat : added database necessary tests"
ocdbytes Jul 26, 2024
63bc846
Revert "feat : added database necessary tests"
ocdbytes Jul 26, 2024
f36b483
update: Replaced Build_Config Fixture with TestConfigBuilder
heemankv Jul 26, 2024
8b7e68f
update : config update
ocdbytes Jul 26, 2024
74fea52
update : test_put_and_get_data_s3 test
ocdbytes Jul 26, 2024
385974d
update: moved testconfigbuilder to tests/config.rs & added docs , dro…
heemankv Jul 29, 2024
eb43d49
feat : create job test case error returning
ocdbytes Jul 30, 2024
a79b633
mock job handler
apoorvsadana Jul 30, 2024
2209405
merge main
ocdbytes Jul 30, 2024
c01c990
Merge branch 'main' into feat/increasing-test-coverage
heemankv Jul 31, 2024
e026d8d
feat : added jobs test and modified worker tests
ocdbytes Jul 31, 2024
661adaf
feat : added queue checks to tests and revamped some tests
ocdbytes Aug 1, 2024
dc05623
Merge branch 'feat/increasing-test-coverage' into feat/job_tests_global
ocdbytes Aug 1, 2024
6ecef60
feat : updated tests and resolved comments
ocdbytes Aug 2, 2024
b31212c
feat : updated test config and added config type to aws s3 config
ocdbytes Aug 2, 2024
ad585f1
feat : updated tests and test names
ocdbytes Aug 2, 2024
57b21af
Merge branch 'feat/increasing-test-coverage' into feat/job_tests_global
ocdbytes Aug 2, 2024
b3de1a2
feat : lint fixes
ocdbytes Aug 3, 2024
cd5e9e8
Merge branch 'main' into feat/job_tests_global
ocdbytes Aug 3, 2024
763379e
feat : lint fixes
ocdbytes Aug 3, 2024
ec61d72
feat : lint fixes
ocdbytes Aug 3, 2024
ae2d4d9
chore: resolved pr comments
ocdbytes Aug 3, 2024
67141ba
Update crates/orchestrator/src/tests/database/mod.rs
ocdbytes Aug 3, 2024
6c9b68e
Merge branch 'main' into feat/increasing-test-coverage
ocdbytes Aug 3, 2024
26df776
chore : lint fixes
ocdbytes Aug 3, 2024
20a480e
feat : lint fix
ocdbytes Aug 3, 2024
7f01895
fix : coverage tests fix
ocdbytes Aug 5, 2024
44aecd7
fix : test fix
ocdbytes Aug 5, 2024
2de874d
fix : updated region in localstack .env.test
ocdbytes Aug 5, 2024
ea867ae
feat : updated region
ocdbytes Aug 5, 2024
0887de5
debug : added debug log to github ci
ocdbytes Aug 5, 2024
d6e31cb
feat : updated queue code for test fixes
ocdbytes Aug 5, 2024
a6d9255
fix : sqs region fix
ocdbytes Aug 5, 2024
fa91ccb
debug : added debug logs for ci debugging
ocdbytes Aug 5, 2024
2f6283e
feat : added override endpoint to queue url in producer and consumer
ocdbytes Aug 5, 2024
b1b9c9f
feat : added override endpoint to queue url in producer and consumer
ocdbytes Aug 5, 2024
4ab388f
fix : removed logs and refactored the code
ocdbytes Aug 5, 2024
c4aef5d
chore : refactor code
ocdbytes Aug 6, 2024
e27de07
Merge branch 'feat/increasing-test-coverage' into feat/job_tests_global
ocdbytes Aug 6, 2024
53e9213
Merge branch 'main' into feat/job_tests_global
ocdbytes Aug 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,14 @@ STARKNET_CAIRO_CORE_CONTRACT_ADDRESS=
# MongoDB connection string
MONGODB_CONNECTION_STRING=

# SQS
# AWS
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=

# SQS
SQS_JOB_PROCESSING_QUEUE_URL=
SQS_JOB_VERIFICATION_QUEUE_URL=

# S3
AWS_S3_BUCKET_NAME=
Expand Down
3 changes: 3 additions & 0 deletions .env.test
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ AWS_SECRET_ACCESS_KEY="AWS_SECRET_ACCESS_KEY"
AWS_S3_BUCKET_NAME="madara-orchestrator-test-bucket"
AWS_S3_BUCKET_REGION="us-east-1"
AWS_ENDPOINT_URL="http://localhost.localstack.cloud:4566"
SQS_JOB_PROCESSING_QUEUE_URL="http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/madara_orchestrator_job_processing_queue"
SQS_JOB_VERIFICATION_QUEUE_URL="http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/madara_orchestrator_job_verification_queue"
AWS_DEFAULT_REGION="localhost"

##### On chain config #####

Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- Function to calculate the kzg proof of x_0.
- Tests for updating the state.
- Function to update the state and publish blob on ethereum in state update job.
- Tests for job handlers in orchestrator/src/jobs/mod.rs.
- Fixtures for testing.
- Basic rust-toolchain support.
- `AWS_DEFAULT_REGION="localhost"` var. in .env.test for omniqueue queue testing.
- Added basic rust-toolchain support.

## Changed

- GitHub's coverage CI yml file for localstack and db testing.
- Orchestrator :Moved TestConfigBuilder to `config.rs` in tests folder.
- `.env` file requires two more variables which are queue urls for processing
and verification.

## Removed

Expand Down
85 changes: 63 additions & 22 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion crates/orchestrator/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,12 @@ path = "src/main.rs"
[dependencies]
alloy = { workspace = true }
arc-swap = { workspace = true }
assert_matches = "1.5.0"
async-std = "1.12.0"
async-trait = { workspace = true }
aws-config = { version = "1.1.7", features = ["behavior-version-latest"] }
aws-sdk-s3 = { version = "1.38.0", features = ["behavior-version-latest"] }
aws-sdk-sqs = "1.36.0"
axum = { workspace = true, features = ["macros"] }
axum-macros = { workspace = true }
bincode = { workspace = true }
Expand All @@ -35,7 +37,8 @@ lazy_static = { workspace = true }
log = "0.4.21"
majin-blob-core = { git = "https://github.com/AbdelStark/majin-blob", branch = "main" }
majin-blob-types = { git = "https://github.com/AbdelStark/majin-blob", branch = "main" }
mockall = "0.12.1"
mockall = { version = "0.13.0" }
mockall_double = "0.3.1"
mongodb = { workspace = true, features = ["bson-uuid-1"], optional = true }
num = { workspace = true }
num-bigint = { workspace = true }
Expand Down
2 changes: 1 addition & 1 deletion crates/orchestrator/src/database/mod.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use ::mongodb::bson::doc;
use std::collections::HashMap;

use ::mongodb::bson::doc;
use async_trait::async_trait;
use color_eyre::Result;
use mockall::automock;
Expand Down
1 change: 1 addition & 0 deletions crates/orchestrator/src/jobs/da_job/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,7 @@ mod tests {
use serde_json::json;

use super::*;
// use majin_blob_types::serde;
use crate::tests::common::init_config;

#[rstest]
Expand Down
65 changes: 65 additions & 0 deletions crates/orchestrator/src/jobs/job_handler_factory.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
use mockall::automock;

#[automock]
pub mod factory {
use std::sync::Arc;

#[allow(unused_imports)]
use mockall::automock;

use crate::jobs::types::JobType;
use crate::jobs::{da_job, proving_job, snos_job, state_update_job, Job};

/// To get the job handler
// +-------------------+
// | |
// | Arc<Box<dyn Job>>|
// | |
// +--------+----------+
// |
// | +----------------+
// | | |
// +--->| Box<dyn Job> |
// | | |
// | +----------------+
// | |
// | |
// +-------v-------+ |
// | | |
// | Closure 1 | |
// | | |
// +---------------+ |
// |
// +---------------+ |
// | | |
// | Closure x | |
// | | |
// +---------------+ |
// |
// |
// v
// +--------------+
// | |
// | dyn Job |
// | (job_handler)|
// | |
// +--------------+
/// We are using Arc so that we can call the Arc::clone while testing that will point
/// to the same Box<dyn Job>. So when we are mocking the behaviour :
///
/// - We create the MockJob
/// - We return this mocked job whenever a function calls `get_job_handler`
/// - Making it an Arc allows us to return the same MockJob in multiple calls to `get_job_handler`. This is needed because `MockJob` doesn't implement Clone
pub async fn get_job_handler(job_type: &JobType) -> Arc<Box<dyn Job>> {
// Original implementation
let job: Box<dyn Job> = match job_type {
JobType::DataSubmission => Box::new(da_job::DaJob),
JobType::SnosRun => Box::new(snos_job::SnosJob),
JobType::ProofCreation => Box::new(proving_job::ProvingJob),
JobType::StateTransition => Box::new(state_update_job::StateUpdateJob),
_ => unimplemented!("Job type not implemented yet."),
};

Arc::new(job)
}
}
Loading
Loading