[feature] Eldritch & Imix Refactor (#509)

* Refactors eldritch, introduces `eldritch::Runtime`
* Refactors imix, introduces `imix::Agent` and `imix::Config`
* Refactors our protobufs, creating new `eldritch` package for language specific types

README.md

@@ -60,29 +60,7 @@ ENABLE_TEST_DATA=1 go run ./tavern
 
 ```bash
 git clone https://github.com/spellshift/realm.git
-cd realm/implants/imix
-
-# Create the config file
-cat <<EOF > /tmp/imix-config.json
-{
-    "service_configs": [],
-    "target_forward_connect_ip": "127.0.0.1",
-    "target_name": "test1234",
-    "callback_config": {
-        "interval": 4,
-        "jitter": 1,
-        "timeout": 4,
-        "c2_configs": [
-        {
-            "priority": 1,
-            "uri": "http://127.0.0.1/grpc/"
-        }
-        ]
-    }
-}
-EOF
-
-cargo run -- -c /tmp/imix-config.json
+cd realm/implants/imix && cargo run
 
 ```
 

bin/golem_cli_test/eldritch_test.tome

@@ -1 +1 @@
-str(dir(file))
+print(dir(file))

bin/golem_cli_test/hello_world.tome

@@ -4,4 +4,4 @@ def test():
         res = res + i
     return res
 
-str(test())
+print(test())

docs/_docs/dev-guide/introduction.md

@@ -6,11 +6,13 @@ description: Read this before contributing to Realm!
 permalink: dev-guide/introduction
 ---
 # Overview
+
 This section of the documentation is meant for new Realm-contributors, and should be read in it's entirety before submitting your first PR. Below you can learn more about our testing & documentation requirements, project layout, and some of the internals of our codebase.
 
 ## Contribution Guidelines
 
 ### Documentation
+
 Realm is under heavy active development and documentation can go stale quickly if it's not actively maintained. Please take a moment to familiarize yourself with both the **[Developer Documentation](/dev-guide)** you're reading now as well as the **[User-Facing Documentation](/user-guide)**. When submitting a code change, please include updates to the relevant portions of our documentation.
 
 We will do our best during code review to catch changes that require documentation updates, but sometimes things will slip by. If you notice a discrepancy between our codebase and the documentation, please kindly [file an issue](https://github.com/spellshift/realm/issues/new?labels=documentation&title=Documentation%20Discrepancy:&body=Please%20include%20the%20location%20of%20the%20inaccurate%20documentation%20and%20a%20helpful%20description%20of%20what%20needs%20improvement.) to track it or submit a PR to correct it. You can use the ["Edit this page"](https://github.com/spellshift/realm/edit/main/docs/_docs/dev-guide/introduction.md) feature in the right navbar of the documentation to quickly navigate to the appropriate section of documentation that requires an update.
@@ -22,26 +24,30 @@ Realm contains code across a variety of languages and frameworks. Testing helps
 #### Eldritch
 
 Any methods added to the Eldritch Standard Library should have tests collocated in the method's `<name>_impl.rs` file. Here are a few things to keep in mind:
+
 * Tests should be cross platform
-    * Rely on [NamedTempFile](https://docs.rs/tempfile/1.1.1/tempfile/struct.NamedTempFile.html) for temporary files
-    * Rely on [path.join](https://doc.rust-lang.org/stable/std/path/struct.Path.html) to construct OS-agnostic paths
+  * Rely on [NamedTempFile](https://docs.rs/tempfile/1.1.1/tempfile/struct.NamedTempFile.html) for temporary files
+  * Rely on [path.join](https://doc.rust-lang.org/stable/std/path/struct.Path.html) to construct OS-agnostic paths
 
 #### Tavern
 
 ##### Tavern Tests (Golang)
+
 All code changes to Tavern must be tested. Below are some standards for test writing that can help improve your PRs:
+
 * Please submit relevant tests in the same PR as your code change
 * For GraphQL API Tests, please refer to our [YAML specification](/dev-guide/tavern#yaml-test-reference-graphql)
 * For gRPC API Tests, please refer to our [YAML specification](/dev-guide/tavern#yaml-test-reference-grpc)
 * Conventionally, please colocate your test code with the code it is testing and include it in the `<packagename>_test` package
 * We rely on the standard [testify](https://github.com/stretchr/testify) assert & require libraries for ensuring expected values (or errors) are returned
 * To enable a variety of inputs for a test case, we rely on closure-driven testing for Golang, you can read more about it [here](https://medium.com/@cep21/closure-driven-tests-an-alternative-style-to-table-driven-tests-in-go-628a41497e5e)
 * Reusable test code should go in a sub-package suffixed with test
-    * For example, reusable test code for the `ent` package would be located in the `ent/enttest` package
-    * This convention is even used in the Golang standard library (e.g. [net/http](https://pkg.go.dev/net/http/httptest))
+  * For example, reusable test code for the `ent` package would be located in the `ent/enttest` package
+  * This convention is even used in the Golang standard library (e.g. [net/http](https://pkg.go.dev/net/http/httptest))
 * Please use existing tests as a reference for writing new tests
 
 ##### Tavern Tests (Front End)
+
 At the time of writing, the Tavern UI is still in an early stage, and therefore minimal testing exists for it. Once the UI is considered more stable, this documentation will be updated. If the Tavern UI is useable and this documentation still exists, please [file an issue](https://github.com/spellshift/realm/issues/new?labels=documentation&title=Documentation%20Discrepancy:&body=Please%20include%20the%20location%20of%20the%20inaccurate%20documentation%20and%20a%20helpful%20description%20of%20what%20needs%20improvement.).
 
 ### Linear History
@@ -53,47 +59,55 @@ In an attempt to reduce the complexity of merges, we enforce a linear history fo
 Throughout the documentation terms like "agent" or "implant" are used to reference various components (or types of components) in our codebase. Below we attempt to define some of those terms, to add some clarity to that other documentation.
 
 ### Host
+
 A Host is a system that is in-scope for the current engagement. It is used to establish a logical boundary between different systems in an engagement (e.g. between a webserver and a database). This enables operations to target a particular system, for example you may want to list files on a web server in your engagement scope.
 
 ### Implant
+
 References malicious code or persistence mechanisms that are deployed to compromise target systems.
 
 ### Agent
+
 An Agent is a type of implant which retrieves execution instructions by connecting to our backend infrastructure (calling back) and querying for new tasks.
 
 ### Beacon
+
 A Beacon is a running instance of an Agent. A Host may have multiple active Beacons that use the same underlying Agent.
 
 ### Task
+
 A Task represents a set of instructions for an Agent to perform. For example, listing files could be a Task. When listing files across various Beacons, one Task per Beacon will be created for tracking the individual execution output.
 
 ### Eldritch
+
 Eldritch is our Pythonic Domain Specific Language (DSL), which can be used to progammatically define red team operations. Many of the language's built-in features do not rely on system binaries. For more information, please see the [Eldritch section](/user-guide/eldritch) of the documentation.
 
 ### Tome
+
 A Tome is a prebuilt Eldritch bundle, which provides execution instructions to a Beacon. Tomes can embed files and accept parameters to change their behavior at runtime. Tavern's built-in Tomes are defined [here](https://github.com/spellshift/realm/tree/main/tavern/tomes).
 
 # Project Structure
+
 * **[.devcontainer](https://github.com/spellshift/realm/tree/main/.devcontainer)** contains settings required for configuring a VSCode dev container that can be used for Realm development
 * **[.github](https://github.com/spellshift/realm/tree/main/.github)** contains GitHub related actions, issue templates, etc
 * **[docker](https://github.com/spellshift/realm/tree/main/docker)** docker containers for production builds
 * **[docs](https://github.com/spellshift/realm/tree/main/docs)** contains the Jekyll code for the documentation site that you're reading now!
 * **[implants](https://github.com/spellshift/realm/tree/main/implants)** is the parent folder of any implant executables or libraries
-    * **[implants/golem](https://github.com/spellshift/realm/tree/main/implants/golem)** the stand-alone interpreter that implements the eldritch language (Rust)
-    * **[implants/golem/embed_files_golem_prod](https://github.com/spellshift/realm/tree/main/implants/golem/embed_files_golem_prod)** Files and scripts that will be embedded into production builds of imix, golem, and eldritch. These files can be accessed through the [`assets` module.](https://docs.realm.pub/user-guide/eldritch#assets)
-    * **[implants/imix](https://github.com/spellshift/realm/tree/main/implants/imix)** is our agent that executes eldritch tomes (Rust)
-    * **[implants/lib/eldritch](https://github.com/spellshift/realm/tree/main/implants/lib/eldritch)** is the source of our eldritch library (Rust)
-    * **[implants/lib/tavern](https://github.com/spellshift/realm/tree/main/implants/lib/tavern)** is the source of our agents graphql API to interface with Tavern (Rust)
+  * **[implants/golem](https://github.com/spellshift/realm/tree/main/implants/golem)** the stand-alone interpreter that implements the eldritch language (Rust)
+  * **[implants/golem/embed_files_golem_prod](https://github.com/spellshift/realm/tree/main/implants/golem/embed_files_golem_prod)** Files and scripts that will be embedded into production builds of imix, golem, and eldritch. These files can be accessed through the [`assets` module.](https://docs.realm.pub/user-guide/eldritch#assets)
+  * **[implants/imix](https://github.com/spellshift/realm/tree/main/implants/imix)** is our agent that executes eldritch tomes (Rust)
+  * **[implants/lib/eldritch](https://github.com/spellshift/realm/tree/main/implants/lib/eldritch)** is the source of our eldritch library (Rust)
 * **[tavern](https://github.com/spellshift/realm/tree/main/tavern)** is the parent folder of Tavern related code and packages, and stores the `main.go` executable for the service
-    * **[tavern/auth](https://github.com/spellshift/realm/tree/main/tavern/auth)** is a package for managing authentication for Tavern, and is used by various packages that rely on obtaining viewer information
-    * **[tavern/internal/ent](https://github.com/spellshift/realm/tree/main/tavern/internal/ent)** contains models and related code for interacting with the database (most of this is code generated by **[entgo](https://entgo.io/))**
-        * **[tavern/internal/ent/schema](https://github.com/spellshift/realm/tree/main/tavern/internal/ent/schema)** contains the schema definitions for our DB models
-    * **[tavern/internal/graphql](https://github.com/spellshift/realm/tree/main/tavern/internal/graphql)** contains our GraphQL definitions and resolvers (most of this code is generated by **[entgo](https://entgo.io/)** and **[gqlgen](https://github.com/99designs/gqlgen))**
-    * **[tavern/internal](https://github.com/spellshift/realm/tree/main/tavern/internal)** contains various internal packages that makeup Tavern
-        * **[tavern/internal/www](https://github.com/spellshift/realm/tree/main/tavern/internal/www)** contains Tavern's UI code
+  * **[tavern/auth](https://github.com/spellshift/realm/tree/main/tavern/auth)** is a package for managing authentication for Tavern, and is used by various packages that rely on obtaining viewer information
+  * **[tavern/internal/ent](https://github.com/spellshift/realm/tree/main/tavern/internal/ent)** contains models and related code for interacting with the database (most of this is code generated by **[entgo](https://entgo.io/))**
+    * **[tavern/internal/ent/schema](https://github.com/spellshift/realm/tree/main/tavern/internal/ent/schema)** contains the schema definitions for our DB models
+  * **[tavern/internal/graphql](https://github.com/spellshift/realm/tree/main/tavern/internal/graphql)** contains our GraphQL definitions and resolvers (most of this code is generated by **[entgo](https://entgo.io/)** and **[gqlgen](https://github.com/99designs/gqlgen))**
+  * **[tavern/internal](https://github.com/spellshift/realm/tree/main/tavern/internal)** contains various internal packages that makeup Tavern
+    * **[tavern/internal/www](https://github.com/spellshift/realm/tree/main/tavern/internal/www)** contains Tavern's UI code
 * **[terraform](https://github.com/spellshift/realm/tree/main/terraform)** contains the Terraform used to deploy a production ready Realm instance. See [Tavern User Guide](https://docs.realm.pub/user-guide/tavern) to learn how to use.
 * **[tests](https://github.com/spellshift/realm/tree/main/tests)** miscellaneous files and example code used for testing. Generally won't be used but required for some niche situations like deadlocking cargo build.
 * **[vscode](https://github.com/spellshift/realm/tree/main/vscode)** contains our Eldritch VSCode integration source code **(Unmaintained)**
 
 # Where to Start?
+
 If you'd like to make a contribution to Realm but aren't sure where to start or what features could use help, please consult our [Good First Issues](https://github.com/spellshift/realm/labels/good%20first%20issue) for some starting ideas.

docs/_docs/user-guide/imix.md

@@ -8,61 +8,21 @@ permalink: user-guide/imix
 ## What is Imix
 
 Imix is the default agent for realm.
-Imix currently only supports http callbacks which interact directly with the graphql API.
+Imix currently only supports http(s) callbacks to Tavern's gRPC API.
 
 ## Configuration
 
-By default Imix is configured using a JSON file at run time.
+Imix has compile-time configuration, that may be specified using environment variables during `cargo build`.
 
-The config is specified at run time with the `-c` flag.
-For example:
+| Env Var | Description | Default | Required |
+| ------- | ----------- | ------- | -------- |
+| IMIX_CALLBACK_URI | URI for initial callbacks (must specify a scheme, e.g. `http://`) | `http://127.0.0.1:80` | No |
+| IMIX_CALLBACK_INTERVAL | Duration between callbacks, in seconds. | `5` | No |
+| IMIX_RETRY_INTERVAL | Duration to wait before restarting the agent loop if an error occurs, in seconds. | `5` | No |
 
-```bash
-./imix -c /tmp/imix-config.json
-```
-
-The imix config is as follows:
-
-```json
-{
-    "service_configs": [
-        {
-            "name": "imix",
-            "description": "Imix c2 agent",
-            "executable_name": "imix",
-            "executable_args": ""
-        }
-    ],
-    "target_forward_connect_ip": "127.0.0.1",
-    "target_name": "test1234",
-    "callback_config": {
-        "interval": 4,
-        "jitter": 1,
-        "timeout": 4,
-        "c2_configs": [
-        {
-            "priority": 1,
-            "uri": "http://127.0.0.1/grpc"
-        }
-        ]
-    }
-}
-```
+## Logging
 
-- `service_configs`: Defining persistence variables.
-  - `name`: The name of the service to install as.
-  - `description`: If possible set a description for the service.
-  - `executable_name`: What imix should be named Eg. `not-supicious-serviced`.
-  - `executable_args`: Args to append after the executable.
-- `target_forward_connect_ip`: The IP address that you the red teamer would interact with the host through. This is to help keep track of agents when a hosts internal IP is different from the one you interact with in the case of a host behind a proxy.
-- `target_name`: Currently unused.
-- `callback_config`: Define where and when the agent should callback.
-  - `interval`: Number of seconds between callbacks.
-  - `jitter`: Currently unused.
-  - `timeout`: The number of seconds to wait before aborting a connection attempt.
-  - `c2_config` Define where the c2 should callback to.
-    - `priority`: The index that a domain should have.
-    - `uri`: The full URI of the callback endpoint.
+At runtime, you may use the `IMIX_LOG` environment variable to control log levels and verbosity. See [these docs](https://docs.rs/pretty_env_logger/latest/pretty_env_logger/) for more information. When building a release version of imix, logging is disabled and is not included in the released binary.
 
 ## Installation
 

go.mod

@@ -12,6 +12,7 @@ require (
 	github.com/google/go-cmp v0.6.0
 	github.com/hashicorp/go-multierror v1.1.1
 	github.com/mattn/go-sqlite3 v1.14.16
+	github.com/prometheus/client_golang v1.18.0
 	github.com/stretchr/testify v1.8.2
 	github.com/urfave/cli v1.22.5
 	github.com/vektah/gqlparser/v2 v2.5.10
@@ -45,7 +46,6 @@ require (
 	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
-	github.com/prometheus/client_golang v1.18.0 // indirect
 	github.com/prometheus/client_model v0.5.0 // indirect
 	github.com/prometheus/common v0.45.0 // indirect
 	github.com/prometheus/procfs v0.12.0 // indirect