Skip to content

Commit

Permalink
chore: merge main into prod for v0.12.0 (#730)
Browse files Browse the repository at this point in the history
* feat: shuttle-serenity initial commit poc (#429)

* feat: shuttle-serenity initial commit poc

* remove shuttle-service

* refactor: drop more shuttle_service stuff

* refactor: drop default serenity framework

* misc: add wasm32-wasi to nix shell

* refactor: cargo sort

* refactor: cargo fmt

Co-authored-by: chesedo <pieter@chesedo.me>

* shuttle next wrapper POC (#431)

* feat: runtime skeleton

* feat: cleanup

* feat: update cargo.lock, sort deps

* fix: typo

* refactor: create runtimes workspace (#432)

* refactor: rename plugins to runtimes

* refactor: drop serenity folder

* refactor: rename runtime to shuttle-next

* refactor: prepare shuttle-legacy runtime

* feat: add   --provisioner-address arg to both runtimes (#433)

* refactor: fix up dependencies

* feat: add --provisioner-address to both runtimes

* feat: create a control plane interface (part 1) (#436)

* feat: add proto for runtimes

* refactor: legacy move main to lib

* feat: impl Runtime server for legacy

* feat: impl Runtime server for next

* refactor: combine runtimes into one (#438)

* feat: add --legacy flag

* refactor: merge next and legacy runtimes

* refactor: merge proto crates

* refactor: rename to runtime

* refactor: promote runtime

* refactor: update README.md

* refactor: cargo sort

* feat(deployer): add support for building wasm projects (#437)

* feat: move factory to runtime (#444)

* feat: initial commit of moving factory to runtime

* feat: replace deployer factory with dummy,
clean commented code

* feat: update legacy runtime readme

* WIP feat: start runtime from deployer (#450)

* WIP feat: start runtime from deployer

* feat: attempt to start runtime as a child process

* fix: add current dir

* feat: clean up, update readme

* feat: use cargo manifest dir for runtime path

* refactor: clean up, fmt

* refactor: clippy warnings for Eq

Co-authored-by: chesedo <pieter@chesedo.me>

* feat: get logs from runtime (#459)

* feat: get logs from runtime

* refactor: trim down deployer

* refactor: start runtime earlier

* refactor: connect to client earlier

* refactor: hook runtime logs to persistence

* bug: associate deployment id with logs

* refactor: cleanup

* feat: make sure grpc connection stays open

* feat(shuttle-next): first edition of axum-wasm router (#472)

* feat(shuttle-next): POC of axum wasm router

* feat: use std mutex

* feat: serialize/deserialize http requests to rmp

this serializes/deserializes http requests across the ffi to the axum-wasm router, using the rust messagepack dataformat. sending the body is still a wip

* feat: serialize the full HTTP req/res to rmp

* fix: comment typo

* feat: start hyper with tonic and serve wasm router

* feat: clone inner router arc

* feat: send request body without serialization

* docs: todo comment

* feat: write response body

* fix: serialize response parts

* feat: deserialize parts directly from reader

this also adds a new fd to separate streaming of body and parts

* feat: add axum-wasm to runtime cli

* refactor: remove unused method

* refactor: typo

Co-authored-by: Pieter <pieter@chesedo.me>

* refactor: comments, clean up wrappers

* refactor: move axum-wasm utils to shuttle-common

* refactor: fmt

* refactor: clippy

Co-authored-by: Pieter <pieter@chesedo.me>

* Feat(shuttle-next): stop runtime services (#481)

* feat: add tonic endpoint to stop runtime services

* refactor: use tracing

* feat: implement stop method for serenity bot

* feat: install protoc 21.9 in dockerfile

* refactor: comment

* feat: convert uuid bytes to string then uuid

* docs: update readme

* docs: update readme for serenity bot

* feat: implement stop for axum-wasm and document it

* ci: install newer protoc in CI

* feat: remove axum-wasm feature in runtime for now

we will come back to this later and likely split the runtime by features

* docs: remove feature flags from readme

* fix: clippy

* fix: clippy

* fix: clippy

* feat: send kill signal without spawning task

* feat: return error response on kill_tx err

* feat(next): expand macro into axum routes (#488)

* feat: app codegen model

* refactor: qualify all namespaces

* feat: low-level wasi export fn

* refactor: restrict to supported axum methods

* Feat: parse shuttle::endpoint macro (#490)

* feat: parse params

* feat: parse app

* feat: add test for missing endpoint attribute

* test: invalid args, params and syntax

* feat: check for and test no params

* feat: don't emit error for missing annotation

this way users can create separate functions for their endpoint logic

* fix: remove missing attribute error, not param

* refactor: reword comment

Co-authored-by: Pieter <pieter@chesedo.me>

* feat: duplicate param test, address review

* feat: only strip endpoint attributes

* feat: check for extra endpoint attributes

* refactor: clearer error span for extra endpoints

* feat: has_err variable to group param errors

* refactor: remove optional hint

Co-authored-by: Pieter <pieter@chesedo.me>

* docs: add TODO for multiple endpoint attributes

Co-authored-by: Pieter <pieter@chesedo.me>

* feat: merge main into shuttle-next (#543)

* feat(blog): add missing sqlx migration code to auth blog post (#408)

* feat(blog): add missing sqlx migration code

* fix: nit

* misc: 0.7.0 (#407)

* misc: drop patches

* infra: pin postgres to version 14

* refactor: fix thruster example app name

* infra: db_fqdn fix

* tests: warp hello world

* fix(cargo-shuttle): prevent crash when config owned by root (#409)

* fix: use correct timeout start point (#410)

* fix: use correct timeout start point

* tweak health check frequency

* fmt

* feat(deployer): implement container memory limits (#411)

* feat(deployer): implement container memory limits

* test: fix warp integration test

* bug: `transport error` when trying to connect to provisioner (#416)

* bug: keep provisioner connections alive

* refactor: reduce scope provisioner client is running

* feat: gateway admin revive (#412)

* feat: gateway admin command (revive)

* fmt

* clippy

* refactor: revive deployers using GatewayService

* tests: add ContextArgs

* refactor: simplify passing around of fqdn

* tests: update test archive

* refactor: remove stray exec.rs file

* refactor: unused is_error()

Co-authored-by: chesedo <pieter@chesedo.me>

* bug: timeout curl health check on deployer (#415)

* Article/beta article (#420)

* updated docs url

* beta article w/ damiens corrections

* Feat(www): shuttle beta signup (#421)

* feat(www): add beta signup form

* feat: add formspree endpoint

* feat: remove socials footer from beta page

* feat: add dummy text above sign up form

* feat: placeholder gif

* feat: align dummy text left

* feat: remove gif, add text

* feat: update beta page text

* feat: update the rest of the site

* bug: Fix thruster postgres example (#414)

* feat: shell completions (#343)

* bug: package Secrets.toml (#422)

* bug: package Secrets.toml

* refactor: clippy suggestion

* bug: big archives being cut off at 32 768 bytes (#423)

* refactor: set chunked header

* refactor: don't send stream across threads

Doing this is causing us to loose everything after 32 768 bytes. Don't
know the reason why 🤷

* refactor: fix health check

* refactor: remove unused use

* chore: v0.7.1 (#424)

* chore: v0.7.1

* chore: resources v0.7.1

* docs: v0.7.1

* tests: v0.7.1

* tests: fix warp test

* tests: fix warp test

* tests: fix secrets e2e test

* ci: remove hard coded DD env

* misc: unintended shell changes

* refactor: cargo check suggestion

* refactor: provide better context for errors (#430)

* refactor: do our own health checks on deployer containers (#427)

* refactor: do our own health checks on deployer containers

We are doing this because the docker health checks eats up 25% of a
single CPU core and we are spinning up many of these containers.

https://www.reddit.com/r/docker/comments/b68r53/healtchecks_add_high_cpu_load/
moby/moby#39102

* refactor: clippy suggestion

* bug: clear build folder before extracting (#428)

* refactor: make sure extract directory is created

* tests: test build folder is cleared

* bug: clear the build directory

* added some images (#435)

* feat(www): beta blog updates (#434)

* feat: add captioned image component (#440)

* feat: add captioned image component

* feat: update images

* feat: add header caption

* fix: quotes in header

* www: post small tweaks (#439)

* a few tweaks to the article

* feat: update gif

Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>

* feat: replace cursed gif (#441)

* Feat/update contributing (#426)

* feat(docs): update contributing.md

* feat(e2e): make e2e admin user unique

* feat(e2e): clean up e2e test projects on test complete

* feat(e2e): don't fail on superuser conflict

* feat: add section about contributing to docs

* feat: remove unwrap, add comment to drop impl

* tests: update e2e test key

Co-authored-by: chesedo <pieter@chesedo.me>

* chore: v0.7.2 (#442)

* chore: v0.7.2

* chore: resources v0.7.2

* docs: v0.7.2

* misc: v0.7.2

* misc: script to deploy all examples

* misc: targets for publishing crates

* misc: remove old scripts

* misc: instructions for next steps

* misc: get Cargo.lock changes

* misc: update .PHONY

* misc: show usage

* feat: link the tracing spans between services (#445)

* ci: DD environments

* refactor: try lowering broadcast channel for quicker feedback

* refactor: propagate tracing across threads in deployer

* refactor: propagate tracing between gateway and deployer

* refactor: trace account name

* refactor: associate project with each container

* feat: trace and propagate proxy in gateway

* feat: gateway record project

* refactor: tracing use 'error' for errors

* refactor: fix comment

* refactor: drop recording unused field

* Add docker-compose extra flags param in Makefile (#446)

* Add docker-compose extra flags param

* rename to avoid confusion

* misc: restructure repo (#453)

* misc: only run e2e on production

* misc: move www to shuttle-hq/www repo

* misc: move examples to shuttle-hq/examples repo

* refactor: update links to examples repo

* ci: remove checking fmt of examples

* ci: checkout submodules

* refactor: tf files have been moved to shuttle-hq/terraform-aws-shuttle

* fix: wrap around common::ProjectName for parsing (#451)

* fix: gateway state drifts, health checks and project recreation (#447)

* misc: add more helpful flags to Makefile

* misc: remove old migrator (#463)

* feat: add account_tier column (#458)

* feat: prefetch shuttle-service crates (#461)

* feat: prefetch shuttle-service crates

* refactor: add comment to prepare.sh files

* Feat: revive via gateway endpoint (#460)

* feat: revive via gateway api initial commit

* feat: cleanup, document gateway testing

* refactor: use post for revive endpoint

Co-authored-by: Damien <damienbroka@mailbox.org>

* feat: send task to global queue

Co-authored-by: Damien <damienbroka@mailbox.org>

* feat: create a new admin cli binary crate (#462)

* feat: init shuttle-admin

* feat: add args

* feat: simple client

* feat: read api_key

* refactor: hook it all together

* refactor: switch to post

* fix: broken link (#467)

* fix: add timeout to health checks (#468)

* fix: add timeout to health checks

* Parameterise the timeout

* rollback on inadvertent circleci change

* misc: log out a warning when a task has been running for a long time

* longer timeouts

* Address review

* Fmt

* feat(gateway): add custom domains table and routing (#465)

* WIP feat: count recent start events before restart (#469)

* feat: count recent start events before restart

* feat: try_collect and handle error in start counter

* fix: clippy

* refactor: gateway clippy

* feat: make deployer only answer its own project (#466)

* feat: make deployer only answer its own project

* bug: use correct project name

* refactor: make backwards compatible

* Fixed Links in Readme (#477)

* refactor: base client error off response status code (#470)

* refactor: base client error off response status code

* Update common/src/models/error.rs

Co-authored-by: Oddbjørn Grødem <29732646+oddgrd@users.noreply.github.com>

* refactor: use tracing

Co-authored-by: Oddbjørn Grødem <29732646+oddgrd@users.noreply.github.com>

* feat: verify project exists before sending destroy task (#474)

* feat: add a custom domains admin route (#473)

Co-authored-by: Alex Krantz <alex@krantz.dev>

* bug: deployer freezes (#478)

* bug: reduce spawning to have deployer lock up less

* refactor: don't return cargo logs

* ci: green (#482)

* tests: allow longer time for build

* refactor: trim dependencies on persist

* refactor: clippy suggestion

* feat: TLS acceptor with SNI resolver (#471)

* fix: custom domain routing (#484)

* refactor: more metrics (#475)

* feat: add more metric dimensions to deployer

* feat: add more metric dimensions to gateway

* refactor: common metrics code

* refactor: forward account name

* refactor: add backend feature to deployer

* refactor: standardize naming

* refactor: cargo sort

* misc: configurable deployment tags (#486)

* feat: gateway restores removed containers (#485)

* fix: backend bumps and hot fixes (#487)

* Feature/support actix web (#491)

* Support actix-web: impl Service

* Support actix-web: cli + boilerplate code

* Support actix-web: add framework to doc comment

* Support actix-web: add e2e test

* Support actix-web: fmt

* Support actix-web: sorting toml deps

* Support actix-web: add actix-web to ci

* Support actix-web: formatted boilerplate code for actix-web hello world example

* Support actix-web: formatted boilerplate code for actix-web hello world example
Support actix-web: update cargo lock zstd dep

* Support actix-web: simplify example

* Support actix-web: add test, change example dependency

* fix: e2e test assert, reset example module

Co-authored-by: maksim <maksim@blockchaincmdt.io>
Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>

* Doc: Fix command to prime database with docker-compose (#502)

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>

* Doc: Improve contributing documentation (#499)

In the doc it mentions to go to subfolder of examples but it was not
there :).

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>

* feat: static file support for a single folder (#501)

* feat: static-folder resource

* refactor: make local client work

* refactor: make local deployer work

* feat: storage_manager

* refactor: comments

* refactor: remove unwraps and expects

* refactor: update tests

* refactor: clippy suggestion

* refactor: update version

* refactor: update readme

* refactor: update comment

* refactor: change public to static

* ci: add static-folder

* refactor: code fixes

* refactor: update tests

* refactor: update description

Co-authored-by: Damien <damienbroka@mailbox.org>

Co-authored-by: Damien <damienbroka@mailbox.org>

* ci: build binary (#483)

* ci: build binary

* ci: make tar archive

* ci: make GH draft release with artifacts

* ci: add aarch64

* ci: get version from tag

* ci: rename to linux builds

* ci: add windows

* ci: add macos

* ci: test all

* ci: put all the artifacts under a command

* ci: only on tagged

* ci: restore old

* ci: fix windows

* ci: put archive in separate folder

* ci: switch to tar

* ci: restore old

* refactor: better `cargo-binstall` support

Co-authored-by: Isaiah Gamble <77396670+tsar-boomba@users.noreply.github.com>

* refactor: binary name

Co-authored-by: Isaiah Gamble <77396670+tsar-boomba@users.noreply.github.com>

Co-authored-by: Isaiah Gamble <77396670+tsar-boomba@users.noreply.github.com>

* feat: bump rust to 1.64, bump dependencies (#495)

* feat: bump rust to 1.64, correct contributing.md bug

* feat: bump service deps, change example branch

* fix: pin chrono to 4.22

* chore: upgrade workspace dependencies

* feat: remove aws-smithy-types

ref: smithy-lang/smithy-rs#1760

* feat: upgrade axum to 0.6 in common

* feat: upgrade to axum 0.6.0 in deployer

* feat: upgrade gateway to axum 0.6.0

* feat: upgrde sqlx to 0.6.2 in gateway

* feat: bump chrono to 0.4.23

* feat: replace deprecated chrono functions in deployer

* feat: bump chrono in common

* feat: ignore actix integration test

* feat: implement new state extractor in gateway

* feat: interactive project initialization (#498)

* feat: bump pinned rust version to 1.65 (#504)

* feat: bump pinned rust version to 1.65

 lockfile update is from binary dist PR #483

* ci: install newer protoc

* fix: special module name warning

* fix: clippy

* feat: install newer protoc in dockerfile as well

* misc: env updates (#509)

* ci: update toolchain for binaries

* misc: update dev environment

* misc: remove patches from Cargo.lock

* feat: make the folder configurable (#508)

* feat: make the folder configurable

* Update resources/static-folder/README.md

Co-authored-by: Damien <damienbroka@mailbox.org>

Co-authored-by: Damien <damienbroka@mailbox.org>

* feat: implement workspace inheritance (#506)

* refactor: switch away from cargo package (#507)

* feat: add spinner wait for `project new` and `project status --follow` (#503)

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>
Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>

* fix: capitalise correctly (#511)

* fix: make nice (#512)

* feat: find (soon to be) invalid project names (#479)

* feat: find (soon to be) invalid project names

* refactor: move logic to admin client

* refactor: missed axum 0.6 update (#513)

* feat: build tests in release profile, limit build workers (#514)

* feat: build tests in release profile, limit build workers

* fix: typo in manifests

* feat: create `init` project from correct dir (#518)

* misc: interactive init gif (#519)

* Feat/set examples submodule to main (#520)

* feat: revert examples submodule back to main

* feat: un-ignore actix tests

* chore: 0.8.0 (#521)

* chore: bump examples (#522)

* bug: hacking static folders (#524)

* fix: actix integration with state (#523)

* feat(gateway,deployer): add more tracing events (#500)

* feat(deployer): add more tracing events

* feat(gateway): add more tracing events

* feat: canonicalize before trace (#531)

* feat: 'clean' subcommand (#530)

* feat: 'clean' subcommand

* refactor: output cleaning is done

* Feat/set cpu limit (#529)

* feat: remove redundant actix worker limit

* feat: remove redundant build job limit

* feat: set cpu limit for deployer container

* fix: limit actix worker and build jobs to 4

* feat: add panamax for mirroring crates.io (#525)

* feat: build queue (#532)

* feat: per-project parallelism (#533)

* feat: temp validation of project name in gateway (#534)

* fix(deployer): keep Cargo.lock between deployments (#517)

* ci: remove build and push req on build binaries (#535)

* refactor: don't crash when failing to release slot (#536)

* refactor: release build slot parse type correctly (#538)

* refactor: remove prefetch (#539)

* feat: add cron job for syncing mirror (#537)

* chore: bump cargo-shuttle to 0.8.1 (#540)

* chore: bump cargo-shuttle to 0.8.1

* chore: cargo.lock

* refactor: update manifests and lockfile

implement workspace inheritance for runtime, and remove default-features from shuttle-service

* refactor: fmt

* fix: unused imports, add runtime to deploymentmanager

* feat: move storagemanager to common

* fix: storage manager imports

* fix: codegen test with outdated expected error

* fix: service tests after removing service default features

* fix: deployer tests after removing service default features

* fix: typo

* ci: attempt to fix protoc error

* fix: deployer runtime client connection

* refactor: remove redundant feature flags

* refactor: remove storage_manager from deployer crate

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>
Co-authored-by: Pieter <pieter@chesedo.me>
Co-authored-by: Alexander Krantz <alex@krantz.dev>
Co-authored-by: Damien <git@damien.sh>
Co-authored-by: Ivan <ivancernja@gmail.com>
Co-authored-by: Peter Mertz <peter.s.mertz@gmail.com>
Co-authored-by: XyLyXyRR <39663597+XyLyXyRR@users.noreply.github.com>
Co-authored-by: Damien <damienbroka@mailbox.org>
Co-authored-by: Nereuxofficial <37740907+Nereuxofficial@users.noreply.github.com>
Co-authored-by: Maxim <59533214+biryukovmaxim@users.noreply.github.com>
Co-authored-by: maksim <maksim@blockchaincmdt.io>
Co-authored-by: Federico Guerinoni <41150432+guerinoni@users.noreply.github.com>
Co-authored-by: Isaiah Gamble <77396670+tsar-boomba@users.noreply.github.com>
Co-authored-by: Stijn ("stain") Seghers <stijnseghers@gmail.com>

* refactor: one store per request (#510)

* refactor: read from file

* refactor: make store for every request

* refactor: remove the serenity runtime (#549)

* Feat: shuttle next bump deps (#551)

* fix: axum tests

* feat: bump wasmtime crates

* feat: return streaming body from wasm router (#558)

* feat: stream body to and from router

* fix: reader to stream hack didn't work in wasm

I added a post endpoint and test, which proved that my hacky stream from BufReader didn't work on the wasm side

* refactor: test string

* feat: update codegen with axum-wasm changes

* refactor: clean up

* refactor: typo

* feat: guard against large request bodies

* feat: embed runtime into client and deployer (#559)

* refactor: runtime startup code

* feat: hook cli to runtime

* feat: hook up logs

* feat: custom port

* feat: start a next project with the local runner

* feat: embed executable

* refactor: axum update

* refactor: tonic version from workspace

* refactor: kill runtime correctly

* feat: DB resources for local runs

* feat: static folders for local runs

* feat: secrets for local runner

* refactor: cleanup logs and errors

* refactor: rebuild runtime if it changed

* refactor: more comments

* feat: minimal axum logs

* refactor: unneeded default features

* refactor: fix rerun-if for runtime

* bug: codegen handle_request

* refactor: restore db error

* bug: expanded broken merge (#567)

* feat: refactor router and router inner (#566)

* refactor: clean up imports

* refactor: remove RouterInner

* feat: handle wasi errors

* feat: handle_request error handling

* docs: comments

* refactor: make hyper dev-dep, better error message

* fix: axum-wasm missed import and bracket

* feat: hook in runtime logs (#568)

* feat: simple wasm log to send over unix socket

* feat: hook up logs subscription

* refactor: extra comments

* refactor: simpler interface

* refactor: combine JsonVisitors

* refactor: clippy suggestions

* refactor: update tests

* refactor: extra comment

* refactor: increase channel for better performance

* refactor: move off stderr

* misc: merge main (#572)

* Merge remote-tracking branch 'upstream/main' into shuttle-next

* refactor: touch ups

* fix: missing feature flag in common dep (#573)

* feat: change log read to spawn blocking (#574)

* feat: DX (#577)

* feat: shuttle-next crate for better DX

* refactor: trim dependencies

* bug: fix next check

* ci: add shuttle-runtime

* refactor: activating missing features on dependencies

* refactor: update to shuttle_next::endpoint

* feat: get runtime binary from cargo install (#578)

* feat: get runtime binary from cargo install

* refactor: remove build.rs

* refactor: determine environment with debug assertions

* ci: comment out cargo-sort installation

* fix: clippy

* feat: use cargo home dir, install runtime in prepare.sh

* fix: unused import

* refactor: build from local version in prepare.sh

* fix: local debug run installing from incorrect path

* feat: canonicalize path to debug runtime

* feat: set release runtime install branch to prod

* refactor: move secondary protoc install to common stage

* feat: deployer next (#575)

* feat: propagate next runtime

* feat: store is_next in DB

* feat: runtime manager to allow deployer to start up both runtimes

* feat: make sure tests run

* refactor: better migration query

* refactor: handle runtime errors better

* feat: shutdown runtimes

* bug: missing so

* bug: stop services

* bug: ffi and runtime manager not living long enough

* bug: missing so error

* refactor: run cleanups

* refactor: clippy suggestions

* misc: remove unneeded codegen feature (#584)

* refactor: proto fixups (#585)

* refactor: comment

* refactor: restore binding to external address

* Feat: cleanup fds and tmp (#586)

* feat: remove redundant body fd and /tmp

* feat: update codegen

* refactor: remove tmp from workspace exclude

* ci: next patch and wasi target

* refactor: unwraps and mor (#587)

* refactor: better error handling

* refactor: trim mutexes

* refactor: remove abstract factory

* refactor: remove extension todo

* feat: build our images with the release profile (#583)

* feat: build our images with the release profile

Currently our containerfile builds images with the default debug profile, which means when `get_runtime_executable` is called it will try to install from a checked out local version of `shuttle-runtime`.

* feat: add build arg for cargo profile

* refactor: update comment

* fix: broken codegen/axum test

* feat: install wasm32-wasi target in deploy.sh

* feat: build deps in `CARGO_PROFILE`

* feat: use closure to determine shuttle-runtime path

* fix: path to runtime in docker deployer, clippy

* fix: incorrect function for cargo home

* refactor: always use cargo_home for path to runtime in deployer

* Feature/eng 378 axum wasm multiple handlers per endpoint (#588)

* refactor: clean up if let

* feat: move method validation, add test for it

* test: add test for chained endpoint handlers

* feat: chain endpoints with the same address

* feat: duplicate method error, handler chain bug fix

chaining handlers with a full type path caused a bug where a `,` was inserted, I think the reason is that `path` types can't be followed by `.`. Importing all the routing methods in `app` works, but it is not ideal

* refactor: remove some clones

* refactor: remove unwrap

Co-authored-by: Pieter <pieter@chesedo.me>

* feat: use BTreeMap for building endpoint chains

* fix: only add namespace to first handler in chain

* fix: skip rustfmt for quote in app_to_token test

rustfmt will add a comma to the expected output, which will also be done in the expanded macro output, but not in the stringified output

* refactor: revert btreemap change

* docs: document why we sort the endpoint chains

---------

Co-authored-by: Pieter <pieter@chesedo.me>

* refactor: find code to wrap legacy runtime (#675)

* refactor: find code to wrap legacy runtime

* misc: update README with test instructions

* feat: remove box self from services, remove syncwrapper from axum service (#677)

* feat: remove box self from all services

* feat: remove syncwrapper from axum integration

* Fix/move docker profiles to dev (#674)

* refactor: move docker profiles to dev.yml

* docs: update comment in makefile

* feat: extract next runtime into separate binary (#679)

* feat: extract next runtime into separate binary

* fix: remove startrequestsservice name

* Feature: remove tokio runtime from all resources (#680)

* feat: remove runtime from resources

* fix: compile errors, comments

* fix: remove comment

* feat: trim the service loader, unpin tokio (#681)

* feat: trim the serviceloader, unpin tokio

* fix: tokio sync is needed

* feat: unpin tokio in runtime as well

* fix: static folder tests need tokio rt

* feat: rename loader, build_crate return bin path

* fix: build_Crate invalid args

* chore(typos): fix typos (#682)

* chore(typos): fix typos

* Fix indentation for poise init example (#687)

* Feature/eng 477 make wasm dependencies optional (#688)

* feat: make wasm runtime dependencies optional

* feat: feature gate next lib, update readme

* ci: rename loader feature in ci

* refactor: hyper with server feature in next features

* Feature: eng 465 update all the codegens (#686)

* feat: initial commit of codegen rewrite

* refactor: wrapper to loader

* tests: update the unit tests

* fix: don't unwrap return type

* feat: setup tracing in loader codegen

* fix: pass inn logger to loader

* feat: mvp of shuttle-next local run

* refactor: move logger to runtime

* refactor: move storagemanager export to runtime

* docs: add note about git tags in contrib (#691)

* bug: idle static folder (#692)

* misc: add more logs

* bug: copy static folder

* bug: extra subdirectory

* chore: v0.11.3 (#695)

* Feature: eng 483 trim and fix the tests in shuttle-service (#693)

* tests: update build_crate tests

* tests: remove loader tests

* feat: cleanup service deps

* feat: setup integration tests in runtime

* feat: expected panic message for not_shuttle

* refactor: simplify dummy provisioner

* feat: re-export main and service from runtime

* fix: Respect `Cargo.lock` when building containers (#700)

Building the shuttle containers was failing during the `cargo chef cook`
step because of a cyclic dependency introduced by an upstream crate.
This did not happen when building outside of a container, because the
versions pinned by `Cargo.lock` did not contain any cycles.

See tkaitchuck/aHash#95

* ci: resolve CI errors in shuttle-next (#580)

* test: compile wasm module in axum runtime test setup

* ci: add next patch override to CI

* ci: include wasm32-wasi target in rust install

* fix: deployer tests where runtime fails to start

* fix: incorrect provisioner address

* feat: log service state changes in runtime

* feat: don't send stop req on startup failure

* refactor: unused imports

* refactor: handling legacy panics

* tests: deadlock less

* refactor: fixups

* refactor: clippy suggestions

* tests: mock provisioner

* refactor: restore capture from 'log' and colors

* refactor: clippy suggestions

* tests: longer wait

* tests: don't panic while holding lock

* tests: don't panic on stream closed

* tests: don't filter out state logs

* tests: bigger timeout

* ci: remove duplicate patch

* refactor: comments

---------

Co-authored-by: chesedo <pieter@chesedo.me>

* Merge remote-tracking branch 'upstream/main' into shuttle-next

* Feature/eng 486 update deployer with runtime changes (#696)

* feat: update deployer to work with runtime changes

* feat: pass next feature flag to runtime installs

* fix: revert this change, the error is a rust-analyzer bug

* tests: make deployer test resources bins

* ci: add shuttle-runtime patch to CI

* refactor: consistent naming

* fix: redundant clone of runtime manager

* feat: delete rocket bin from runtime

* feat: make storagemanager a trait object

* docs: add comment about why we use arc

* refactor: use &str in storagemanager

* feat: add runtime patch to deployer

* fix: deployer logic bug, add debug logs

* feat: extract service integrations into separate crates (#702)

* feat: draft of extracting service integrations

* refactor: revert codegen changes

* refactor: depend on shuttle_runtime & service integration

* feat: remove rocket bin, comment out version check

* feat: impl from for integration wrapper

* feat: cleanup shuttle-axum docs, remove version check

* feat: actix-web service integration

* feat: extract poem service integration

* feat: extract poise service integration

* feat: extract rocket service integration

* feat: extract salvo service integration

* feat: extract the serenity service integration

* feat: extract the thruster service integration

* feat: extract warp service integration

* feat: extract the tower service integration

* feat: delete persist from service

* feat: extract tide service integration

* feat: update cargo.lock

* feat: make service integration inner pub

* fix: merge fixes

* refactor: rename integrations, remove comment

* ci: run check-standalone on services

* feat: update test resources

* ci: refactor workspace-clippy job

* fix: add tokio dev dep to services

* fix: remaining services tests

* [next] tests: CI go green (#704)

* tests: thread local tracing

* tests: bind panic in runtime

* tests: post create project params

* tests: fix cargo-shuttle

* misc: std missing from tracing

* tests: update main test for shuttle runtime

* ci: merge workspace jobs

* ci: move service to platform tests

* ci: better names

* refactor: trim dependencies

Trim dependencies to minimize the build times of user projects.

* refactor: catch error and panics in main

* tests: trim cargo-shuttle tests

Make sure there is only one main test for each major vector / feature.

* refactor: missing serde derive

* refactor: missing serde std

* ci: balance machines

* tests: give more time

* ci: bump images more

* ci: bump workspace image

* ci: 2xlarge not available '/

* feat: emit error when shuttle::main is named main (#707)

* Feature: eng 484 update init with codegen changes (#706)

* feat: update init boilerplate with shuttle-next changes

* tests: update shuttle_init_tests

* tests: ignore broken integration tests

* feat: refactor init to create bin

* fix: bug when passing path to init

* fix: codegen clippy (#709)

* fix: codegen clippy

* fix: clippy

* misc: v0.12.0-rc1 (#708)

* misc: v0.12.0-rc1

* misc: publish proto

* refactor: remove relative from build.rs

* refactor: add description to proto

* refactor: make runtime publishable

* refactor: make actix publishable

* feat: make shuttle-axum publishable

* feat: make poem publishable

* feat: make poise publishable

* feat: make shuttle-rocket publishable

* refactor: make warp publishable

* refactor: make tower publishable

* refactor: make tide publishable

* refactor: make thruster publishable

* refactor: make serenity publishable

* refactor: make salvo publishable

---------

Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>

* [next] refactor: update runtime manager (#711)

* refactor: allow multiple runtimes to run at the same time

This is needed so that a new deployment can enter the loading state
without killing currently running runtimes.

* refactor: extra comment

* Feature: eng 497 update or remove the docs in shuttle (#710)

* docs: update service docs

* docs: move service crate-docs to runtime

* docs: update main readme

* tests: fix init test

* refactor: rename legacy runtime to alpha (#713)

* refactor: rename legacy runtime to alpha

* refactor: move next to services (#714)

* refactor: move next to services

* ci: correct next

* refactor: align folder

* [next] refactor: remove ids from runtime (#712)

* refactor: remove deployment id

* refactor: create proto LogItem directly

* refactor: more tracing for local binary

* misc: add kondo to env

* refactor: clippy suggestion

* tests: update

* feat: embed protoc in shuttle-proto (#715)

* feat: embed protoc in shuttle-proto

* fix: path to shuttle-next in runtime next-resource

* [next] bug: communicating resources (#716)

* refactor: send claim to provisioner

* feat: recording of provisioned resources

This returns provisioned resources from runtime so that they can be
recorded by deployer. Closes #ENG-252.

* refactor: restore resources for idle containers waking up

* tests: next tests using wrong folder

* tests: swap rocket for the more stable poem

* refactor: pass auth_uri to runtime

* misc: update all patches

* feat(local): don't install next-runtime from git (#718)

* fix: static folder local run clearing file contents, add missing tests in cargo-shuttle (#717)

* fix: static folder local run clearing file contents

* fix: clippy

* fix: cargo shuttle init tests not logging in

* fix: bail if non-interactive and --new without api-key

* fix: update codegen tests

* [next] refactor: tracing (#719)

* refactor: use tracing setup helper

* refactor: better trace propagation

* refactor: clippy suggestion

* fix: windows local run path bug (#721)

* refactor: pull out vendored protoc from shuttle-proto (#726)

* feat: use grpcio to avoid needing protoc

* refactor: remove vendored protoc, revert to tonic otlp

* [next] bug: misc fixes (#725)

* bug: thread local logging breaking serenity

* refactor: patch all crates

* refactor: extend tokens to 15 minutes

* bug: next runtime not sending stop signal (#728)

* bug: next runtime not sending stop signal

* refactor: better unwrap

* chore: v0.12.0 (#727)

* chore: v0.12.0

* chore: bump examples

---------

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>
Co-authored-by: Damien <git@damien.sh>
Co-authored-by: chesedo <pieter@chesedo.me>
Co-authored-by: Alexander Krantz <alex@krantz.dev>
Co-authored-by: Ivan <ivancernja@gmail.com>
Co-authored-by: Peter Mertz <peter.s.mertz@gmail.com>
Co-authored-by: XyLyXyRR <39663597+XyLyXyRR@users.noreply.github.com>
Co-authored-by: Damien <damienbroka@mailbox.org>
Co-authored-by: Nereuxofficial <37740907+Nereuxofficial@users.noreply.github.com>
Co-authored-by: Maxim <59533214+biryukovmaxim@users.noreply.github.com>
Co-authored-by: maksim <maksim@blockchaincmdt.io>
Co-authored-by: Federico Guerinoni <41150432+guerinoni@users.noreply.github.com>
Co-authored-by: Isaiah Gamble <77396670+tsar-boomba@users.noreply.github.com>
Co-authored-by: Stijn ("stain") Seghers <stijnseghers@gmail.com>
Co-authored-by: Sara Tavares <29093946+stavares843@users.noreply.github.com>
Co-authored-by: HexPandaa <47880094+HexPandaa@users.noreply.github.com>
Co-authored-by: Kieren Davies <kieren@kdavi.es>
  • Loading branch information
17 people authored Mar 20, 2023
1 parent b1c8aba commit 97c0d07
Show file tree
Hide file tree
Showing 210 changed files with 9,263 additions and 7,021 deletions.
111 changes: 47 additions & 64 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,12 @@ executors:
docker-rust:
docker:
- image: cimg/rust:1.65.0
resource_class: small
image-ubuntu:
machine:
image: ubuntu-2204:2022.04.1
docker_layer_caching: true
resource_class: xlarge

# sscache steps are from this guide
# https://medium.com/@edouard.oger/rust-caching-on-circleci-using-sccache-c996344f0115
Expand Down Expand Up @@ -86,18 +88,33 @@ commands:
cat\<< EOF > ~/.cargo/config.toml
[patch.crates-io]
shuttle-service = { path = "$PWD/service" }
shuttle-runtime = { path = "$PWD/runtime" }
shuttle-aws-rds = { path = "$PWD/resources/aws-rds" }
shuttle-persist = { path = "$PWD/resources/persist" }
shuttle-shared-db = { path = "$PWD/resources/shared-db" }
shuttle-secrets = { path = "$PWD/resources/secrets" }
shuttle-static-folder = { path = "$PWD/resources/static-folder" }
shuttle-axum = { path = "$PWD/services/shuttle-axum" }
shuttle-actix-web = { path = "$PWD/services/shuttle-actix-web" }
shuttle-next = { path = "$PWD/services/shuttle-next" }
shuttle-poem = { path = "$PWD/services/shuttle-poem" }
shuttle-poise = { path = "$PWD/services/shuttle-poise" }
shuttle-rocket = { path = "$PWD/services/shuttle-rocket" }
shuttle-salvo = { path = "$PWD/services/shuttle-salvo" }
shuttle-serenity = { path = "$PWD/services/shuttle-serenity" }
shuttle-thruster = { path = "$PWD/services/shuttle-thruster" }
shuttle-tide = { path = "$PWD/services/shuttle-tide" }
shuttle-tower = { path = "$PWD/services/shuttle-tower" }
shuttle-warp = { path = "$PWD/services/shuttle-warp" }
EOF
install-rust:
steps:
- run:
name: Install Rust
command: |
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --target add wasm32-wasi
sudo apt update && sudo apt install -y libssl1.1
install-protoc:
steps:
Expand Down Expand Up @@ -146,33 +163,22 @@ commands:
- << parameters.target >>.env

jobs:
workspace-fmt:
workspace:
executor: docker-rust
resource_class: xlarge
steps:
- checkout
- restore-cargo-cache
- install-protoc
- run: cargo fmt --all --check
- run: cargo install cargo-sort
# TODO: this is incompatible with workspace inheritance, uncomment when
# https://github.com/DevinR528/cargo-sort/pull/29 is merged
# - run: cargo install cargo-sort
# - run: cargo sort --check --workspace
- run: cargo check --workspace --all-targets
- save-cargo-cache
workspace-clippy:
parameters:
framework:
description: "Framework to activate"
type: string
executor: docker-rust
steps:
- checkout
- restore-cargo-cache
- install-protoc
- run: |
cargo clippy --tests \
--all-targets \
--features="codegen,loader,<< parameters.framework >>" \
--all-features \
--no-deps -- \
--D warnings \
-A clippy::let-unit-value \
Expand All @@ -190,9 +196,9 @@ jobs:
- install-protoc
- apply-patches
- run: cargo fmt --all --check --manifest-path << parameters.path >>/Cargo.toml
- run: cargo install cargo-sort
# TODO: this is incompatible with workspace inheritance, uncomment when
# https://github.com/DevinR528/cargo-sort/pull/29 is merged
# - run: cargo install cargo-sort
# - run: cargo sort --check << parameters.path >>
- run: |
cargo clippy --tests \
Expand All @@ -205,20 +211,6 @@ jobs:
-A clippy::format-push-string
- run: cargo test --all-features --manifest-path << parameters.path >>/Cargo.toml -- --nocapture
- save-cargo-cache
service-test:
# Using an image since tests will start a docker container
executor: image-ubuntu
steps:
- install-rust
- checkout
- restore-cargo-cache
- run:
name: Run unit tests
command: cargo test --package shuttle-service --features="codegen,loader" --lib -- --nocapture
- run:
name: Run integration tests
command: cargo test --package shuttle-service --features="codegen,loader" --test '*' -- --nocapture
- save-cargo-cache
platform-test:
parameters:
crate:
Expand All @@ -228,6 +220,7 @@ jobs:
executor: image-ubuntu
steps:
- install-rust
- install-protoc
- checkout
- run: git submodule sync
- run: git submodule update --init
Expand All @@ -244,7 +237,6 @@ jobs:
(cargo test --package << parameters.crate >> --all-features --test '*' -- --list 2>&1 | grep -q "no test target matches pattern") && echo "nothing to test" || cargo test --package << parameters.crate >> --all-features --test '*' -- --nocapture
- save-cargo-cache
e2e-test:
resource_class: xlarge
executor: image-ubuntu
steps:
- install-rust
Expand Down Expand Up @@ -276,7 +268,6 @@ jobs:
key: docker-buildx-{{ .Branch }}
when: always
build-and-push:
resource_class: xlarge
executor: image-ubuntu
steps:
- checkout
Expand Down Expand Up @@ -350,7 +341,7 @@ jobs:
build-binaries-mac:
macos:
xcode: 12.5.1
resource_class: medium
resource_class: xlarge
steps:
- checkout
- run:
Expand Down Expand Up @@ -388,28 +379,9 @@ jobs:
workflows:
ci:
jobs:
- workspace-fmt
- workspace-clippy:
name: workspace-clippy-<< matrix.framework >>
requires:
- workspace-fmt
matrix:
parameters:
framework:
[
"web-actix-web",
"web-axum",
"web-rocket",
"web-poem",
"web-thruster",
"web-tide",
"web-tower",
"web-warp",
"web-salvo",
"bot-serenity",
"bot-poise",
]
- workspace
- check-standalone:
name: << matrix.path >>
matrix:
parameters:
path:
Expand All @@ -418,27 +390,38 @@ workflows:
- resources/secrets
- resources/shared-db
- resources/static-folder
- service-test:
requires:
- workspace-clippy
- services/shuttle-actix-web
- services/shuttle-axum
- services/shuttle-next
- services/shuttle-poem
- services/shuttle-poise
- services/shuttle-rocket
- services/shuttle-salvo
- services/shuttle-serenity
- services/shuttle-thruster
- services/shuttle-tide
- services/shuttle-tower
- services/shuttle-warp
- platform-test:
name: << matrix.crate >>
requires:
- workspace-clippy
- workspace
matrix:
parameters:
crate:
[
"shuttle-auth",
"shuttle-deployer",
"cargo-shuttle",
"shuttle-codegen",
"shuttle-common",
"shuttle-deployer",
"shuttle-proto",
"shuttle-provisioner",
"shuttle-runtime",
"shuttle-service",
]
- e2e-test:
requires:
- service-test
- platform-test
- check-standalone
filters:
Expand All @@ -454,23 +437,23 @@ workflows:
name: build-binaries-x86_64-gnu
image: ubuntu-2204:2022.04.1
target: x86_64-unknown-linux-gnu
resource_class: medium
resource_class: xlarge
filters:
branches:
only: production
- build-binaries-linux:
name: build-binaries-x86_64-musl
image: ubuntu-2204:2022.04.1
target: x86_64-unknown-linux-musl
resource_class: medium
resource_class: xlarge
filters:
branches:
only: production
- build-binaries-linux:
name: build-binaries-aarch64
image: ubuntu-2004:202101-01
target: aarch64-unknown-linux-musl
resource_class: arm.medium
resource_class: arm.xlarge
filters:
branches:
only: production
Expand Down
22 changes: 19 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ git clone git@github.com:shuttle-hq/shuttle.git
cd shuttle
```

> Note: We need the git tags for the local development workflow, but they may not be included when you clone the repository.
To make sure you have them, run `git fetch upstream --tags`, where upstream is the name of the shuttle remote repository.

You should now be ready to setup a local environment to test code changes to core `shuttle` packages as follows:

From the root of the shuttle repo, build the required images with:
Expand Down Expand Up @@ -53,13 +56,26 @@ In order to test local changes to the library crates, you may want to add the be
```toml
[patch.crates-io]
shuttle-service = { path = "[base]/shuttle/service" }
shuttle-common = { path = "[base]/shuttle/common" }
shuttle-proto = { path = "[base]/shuttle/proto" }
shuttle-runtime = { path = "[base]/shuttle/runtime" }

shuttle-aws-rds = { path = "[base]/shuttle/resources/aws-rds" }
shuttle-persist = { path = "[base]/shuttle/resources/persist" }
shuttle-shared-db = { path = "[base]/shuttle/resources/shared-db" }
shuttle-secrets = { path = "[base]/shuttle/resources/secrets" }
shuttle-static-folder = { path = "[base]/shuttle/resources/static-folder" }

shuttle-axum = { path = "[base]/shuttle/services/shuttle-axum" }
shuttle-actix-web = { path = "[base]/shuttle/services/shuttle-actix-web" }
shuttle-next = { path = "[base]/shuttle/services/shuttle-next" }
shuttle-poem = { path = "[base]/shuttle/services/shuttle-poem" }
shuttle-poise = { path = "[base]/shuttle/services/shuttle-poise" }
shuttle-rocket = { path = "[base]/shuttle/services/shuttle-rocket" }
shuttle-salvo = { path = "[base]/shuttle/services/shuttle-salvo" }
shuttle-serenity = { path = "[base]/shuttle/services/shuttle-serenity" }
shuttle-thruster = { path = "[base]/shuttle/services/shuttle-thruster" }
shuttle-tide = { path = "[base]/shuttle/services/shuttle-tide" }
shuttle-tower = { path = "[base]/shuttle/services/shuttle-tower" }
shuttle-warp = { path = "[base]/shuttle/services/shuttle-warp" }
```

Before we can login to our local instance of shuttle, we need to create a user.
Expand Down Expand Up @@ -164,7 +180,7 @@ To run the unit tests for a specific crate, from the root of the repository run:
cargo test --package <crate-name> --all-features --lib -- --nocapture
```

To run the integration tests for a spesific crate (if it has any), from the root of the repository run:
To run the integration tests for a specific crate (if it has any), from the root of the repository run:

```bash
# replace <crate-name> with the name of the crate to test, e.g. `cargo-shuttle`
Expand Down
Loading

0 comments on commit 97c0d07

Please sign in to comment.