Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(store): Database integration #374

Merged
merged 38 commits into from
Jan 14, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
9b90d05
feat(repo): creating fuel-streams-store
pedronauck Jan 6, 2025
6b418d5
refactor(repo): change to Db inside store
pedronauck Jan 6, 2025
bb152a5
refactor(repo): add final fuel-streams-store
pedronauck Jan 6, 2025
368a334
refactor(repo): add serialize json on store
pedronauck Jan 6, 2025
6795991
refactor(repo): use store inside core
pedronauck Jan 6, 2025
72f7f51
refactor(repo): use store across the repo
pedronauck Jan 7, 2025
7afc487
fix(store): a lot of issues
pedronauck Jan 7, 2025
a0fa9aa
test(store): add tests for live data
pedronauck Jan 7, 2025
a2dde3b
refactor(repo): create fuel-streams-type
pedronauck Jan 7, 2025
dd82823
feat(core): add subscribe_historical
pedronauck Jan 8, 2025
49918bb
refactor(repo): adjust errors
pedronauck Jan 8, 2025
26f5948
refactor(webserver): add subscriber
pedronauck Jan 8, 2025
7833c32
refactor(webserver): split socket.rs into modules
pedronauck Jan 8, 2025
005b49d
refactor(repo): adjust env vars
pedronauck Jan 8, 2025
8cd48c3
refactor(repo): remove localstack
pedronauck Jan 8, 2025
dc296ae
refactor(repo): let just one single nats instance
pedronauck Jan 8, 2025
e9ff134
build(repo): change from nats-core to nats
pedronauck Jan 8, 2025
ad25b30
feat(macros): add json and sql features for subject
pedronauck Jan 9, 2025
0a1c600
feat(repo): create separated tables
pedronauck Jan 9, 2025
88abd65
feat(repo): add some tests
pedronauck Jan 9, 2025
caf3144
feat(macros): add json convertion for subjects
pedronauck Jan 10, 2025
29b0575
feat(repo): finish improvements
pedronauck Jan 10, 2025
e7e4d77
build(repo): fix warnings
pedronauck Jan 10, 2025
9e5c83b
test(store): testing stream_by_subject
pedronauck Jan 10, 2025
6e366be
refactor(repo): improve streams
pedronauck Jan 10, 2025
36b285e
feat(repo): add delivery policy from block height
pedronauck Jan 10, 2025
3e04820
refactor(repo): adjust namespace on store and stream
pedronauck Jan 10, 2025
4c0ce04
refactor(repo): final adjustments
pedronauck Jan 10, 2025
fee31a7
refactor(repo): final adjustments 2
pedronauck Jan 10, 2025
893e309
feat(store): add few improvements
pedronauck Jan 10, 2025
cf9cad3
feat(repo): add message broker crate
pedronauck Jan 11, 2025
96a0087
build(repo): adjust chart
pedronauck Jan 11, 2025
8adab68
build(repo): adjust comments
pedronauck Jan 13, 2025
df1d0e8
Merge branch 'main' of github.com:FuelLabs/data-systems into pn/store
pedronauck Jan 13, 2025
17faa5d
refactor(repo): webserver adjustments
pedronauck Jan 13, 2025
7cf15a4
fix(webserver): fix erros handlers
pedronauck Jan 13, 2025
51c4064
fix(webserver): gracefully shutdown
pedronauck Jan 14, 2025
04c674a
test(repo): fix tests
pedronauck Jan 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions .env.sample
Original file line number Diff line number Diff line change
@@ -1,19 +1,20 @@
# General Configuration
DB_POOL_SIZE=5
STORE_PAGINATION_LIMIT=100
STORE_MAX_RETRIES=3
STORE_INITIAL_BACKOFF_MS=100
STREAM_THROTTLE_HISTORICAL=100
STREAM_THROTTLE_LIVE=100

# Authentication & Security
KEYPAIR=generated-p2p-secret
JWT_AUTH_SECRET=generated-secret

# AWS S3 Configuration
AWS_ACCESS_KEY_ID=test
AWS_SECRET_ACCESS_KEY=test
AWS_ENDPOINT_URL=http://localhost:4566
AWS_REGION=us-east-1
AWS_S3_ENABLED=false
AWS_S3_BUCKET_NAME=fuel-streams-local
STORAGE_MAX_RETRIES=5
# Database Configuration
DATABASE_URL=postgresql://root@localhost:26257/defaultdb?sslmode=disable

# NATS Configuration
NATS_URL=nats://localhost:4222
NATS_PUBLISHER_URL=nats://localhost:4333
NATS_SYSTEM_USER=sys
NATS_SYSTEM_PASS=sys
NATS_ADMIN_USER=admin
Expand Down
15 changes: 5 additions & 10 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,17 +38,18 @@ jobs:
refactor
test
scopes: |
benches
repo
deps
release
data-parser
message-broker
fuel-streams
core
domains
executors
macros
nats
storage
store
types
consumer
publisher
webserver
Expand Down Expand Up @@ -211,26 +212,20 @@ jobs:
runs-on: ubuntu-latest
env:
NATS_URL: nats://127.0.0.1:4222
NATS_PUBLISHER_URL: nats://127.0.0.1:4333
NATS_SYSTEM_USER: sys
NATS_SYSTEM_PASSWORD: sys
NATS_ADMIN_USER: admin
NATS_ADMIN_PASS: admin
NATS_PUBLIC_USER: default_user
NATS_PUBLIC_PASS: ""
AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: test
AWS_REGION: us-east-1
AWS_ENDPOINT_URL: http://localhost:4566
AWS_S3_BUCKET_NAME: fuel-streams-local
strategy:
fail-fast: false
matrix:
package:
- fuel-data-parser
- fuel-streams
- fuel-streams-core
- fuel-streams-macros
- fuel-streams-store
- sv-webserver
- sv-publisher
steps:
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,6 @@ docs/
**/**/charts/**.tgz
values-secrets.yaml
values-publisher-env.yaml
localstack-data
.vscode
**/Cargo.lock
.sqlx
4 changes: 0 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ You can check the [./scripts/setup.sh](./scripts/setup.sh) file to see what is b
Here's an overview of the project's directory structure:

- `crates/`: Contains the main Rust crates for the project
- `benches/`: Benchmarking code and performance tests
- `tests/`: Integration and end-to-end tests
- `examples/`: Example code and usage demonstrations
- `cluster/`: Kubernetes cluster configuration and deployment files
Expand Down Expand Up @@ -91,9 +90,7 @@ This is a general rule used for commits. When you are creating a PR, ensure that
- `core`: Changes that affect the core package.
- `publisher`: Changes that affect the publisher package.
- `fuel-streams`: Changes that affect the fuel-streams package.
- `benches`: Changes related to benchmarks.
- `deps`: Changes related to dependencies.
- `data-parser`: Changes that affect the data-parser package.
- `macros`: Changes that affect the macros package.

## 📜 Useful Commands
Expand All @@ -111,7 +108,6 @@ To make your life easier, here are some commands to run common tasks in this pro
| `make test-watch` | Run tests in watch mode |
| `make clean` | Clean the build artifacts |
| `make dev-watch` | Run the project in development mode with auto-reload |
| `make bench` | Run benchmarks for the project |
| `make audit` | Run security audit on dependencies |
| `make audit-fix` | Fix security vulnerabilities in dependencies |
| `make version` | Show current version |
Expand Down
Loading
Loading