Skip to content

Commit

Permalink
Feature/create (#5)
Browse files Browse the repository at this point in the history
* chg: fix: small fixes

* new: feat: alpha version of new bootstrap approach

* chg: docs: update docs

* new: feat: add new create command

* chg: fix: deprecate some parameters

* chg: feat: finalize create command and do some refactor
  • Loading branch information
felipemarinho97 authored Jul 16, 2022
1 parent 2a7af66 commit 42e671e
Show file tree
Hide file tree
Showing 19 changed files with 1,065 additions and 223 deletions.
108 changes: 0 additions & 108 deletions BOOTSTRAP.md

This file was deleted.

41 changes: 41 additions & 0 deletions BOOTSTRAPPING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# How to bootstrap a Dev Space

This section describes how to bootstrap a Dev Space.


## Create a key pair to SSH into the instance

Now, create a key pair and store it in a file, if you already have a key pair, you can skip this step.

```bash
$ aws ec2 create-key-pair --key-name MyKeyPair --query 'KeyMaterial' --output text > MyKeyPair.pem
```

Remember to give the right permissions to the file.

```bash
$ chmod 400 MyKeyPair.pem
```

## Use the CLI to bootstrap a Dev Space

Download one of the following bootstrap scripts:
- [Bootstrap Script for Arch Linux](https://raw.githubusercontent.com/felipemarinho97/dev-spaces/master/examples/templates/arch.yaml)

Edit the `key_name` and `instance_profile_arn` fields to match your key pair and instance profile you just created.

You can also edit the other fields to match your requirements, like the availability zone, the instance type, the EBS volume size, etc.

Then, you can use the CLI to bootstrap a Dev Space.

```bash
$ dev-spaces bootstrap --template arch.yaml --name MyDevSpace
```

The CLI will take care of creating the Dev Space.

Once the Dev Space is created, you can use the CLI to start it.

```bash
$ dev-spaces start -n MyDevSpace --min-cpus 2 --min-memory 4 --max-price 0.08
```
89 changes: 89 additions & 0 deletions CREATING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Creating a DevSpace

This document describes how to bootstrap a Dev Space using the command `dev-spaces create`. This new version has a better approach on creating the environment because it's more generic and it's easier to manage. Also, it's much more faster (just take few seconds to bootstap a space) than the previous version.

## Instance Profile

There is no need to create an instance profile with EBS permissions because the `start` command will wait until the instance is ready and attach the EBS Volume by itself. You can still specify the instance profile in the create command if you want to.

## SSH Key Pair

Make sure you have a SSH key pair in your AWS account. You can see [here](BOOTSTRAPPING.md#create-a-key-pair-to-ssh-into-the-instance) how to create one using the `aws cli`. If you want your key pair to be availiable in all regions, you can follow [this tutorial](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-ssh-key-pair-regions/) from a AWS Support.

## Creating the Space

The command below will use the pre-defined template to create an space with Amazon Linux 2022 AMI.

$ dev-spaces create -n MyAmazonLinux2022 -k MyKeyPair -i ami-034b81f0f1dd96797

This AMI have the advantage of supporting running docker inside the Dev Space.

Once created, you can use the command `dev-spaces start` to start the space.

$ dev-spaces start -n MyAmazonLinux2022

Now, your instance will be available to ssh into from the port `2222`.

## Command Parameters

_Parameter_|_Alias_|_Description_|_Example_|
|:--:|:--:|:--:|:--:|
|`--name`|`-n`|The name of the Dev Space|`MyDevSpace`|
|`--key-name`|`-k`|The name of the SSH key pair|`MyKeyPair`|
|`--ami`|`-a`|The AMI ID|`ami-034b81f0f1dd96797`|
|`--prefered-instance-type`|`-t`|The type of the instance|`t2.micro`|
|`--instance-profile-arn`|`-p`|The ARN of the instance profile|`arn:aws:iam::123456789012:instance-profile/MyInstanceProfile`|
|`--custom-host-ami`|-|Custom AMI to use for the host - use this flag in combination with `--custom-startup-script`|`ami-034b81f0f1dd96797`|
|`--custom-startup-script`|-|Custom startup script file to use for the host|`./myscript.sh`|
|`--security-group-ids`|-|A list of IDs of the security groups to use|`sg-12345678`|

## Troubleshooting

If for some reason you can't SSH into the instance, you can troubleshoot what's wrong by looking at the logs of the host machine.

Also, try logging into the instance using the root user (port 2222).

SSH into the host instance (port 22):

$ ssh -i MyKeyPair.pem -p 22 ec2-user@<IP_ADDRESS>
ec2-user:~$ cat /var/log/user-data.log
ec2-user:~$ cat /var/log/cloud-init-output.log
ec2-user:~$ cat /var/log/cloud-init.log

Common problems are user-script failing to run due to wrong filesystem mount type, wrong permissions, etc.

## Extra: Building a Ultra-Optimized DevSpace

This section will teach you how to build a DevSpace with a very thin host image, this will make the startup time of the DevSpace much faster and will also reduce the costs of the host EBS volume.

### Step 1: Create a Host Image

Clone [this repository](https://github.com/felipemarinho97/packer-images):

$ git clone https://github.com/felipemarinho97/packer-images

You will need to install the [packer](https://www.packer.io/) tool, follow the instructions on the [packer website](https://www.packer.io/docs/).

Make sure your AWS credentials are set up correctly.

Inside the `packer-images` directory, run the following command to build the host image:

$ cd packer-images
$ packer build base

Once the image is built, `packer` will output the AMI ID of the host image on your console.

### Step 2: Use the base-minimal image to create the optimized DevSpace

Now, you can use the AMI ID of the `base-minimal` image to create the optimized DevSpace. Also, you will need to provide a startup script compatible with your custom host AMI. You can use [this sample](examples/scripts/startup-script.sh) directly on the create command, or you can create your own startup script.

```bash
$ dev-spaces create -n MyDevSpace \
-k MyKeyPair \
-i <your-devsapce-ami> \
--custom-host-ami <base-minimal-ami-id> # <-- this is the AMI ID of the image created by packer
--custom-startup-script 'https://raw.githubusercontent.com/felipemarinho97/dev-spaces/master/examples/scripts/startup-script.sh'

```

Done! This optimized DevSpace will take only few seconds to start.
93 changes: 76 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,29 +3,39 @@
This is a CLI to help creating on-demand development spaces using EC2 Spot Intances.

Currently, the following commands are availble:
* start
* stop
* status
* [start](#starting-a-devspace)
* [stop](#terminating-devspaces)
* [status, list](#listing-my-devspaces)
* [create](#creating-a-devspace)
* [destroy](#destroying-a-devspace)


```bash
$ dev-spaces --help
NAME:
dev-spaces - CLI to help dev-spaces creation and management on AWS
dev-spaces - CLI to help dev-spaces creation and management

USAGE:
main [global options] command [command options] [arguments...]
dev-spaces [global options] command [command options] [arguments...]

AUTHOR:
Felipe Marinho <felipevm97@gmail.com>

COMMANDS:
start --help
stop --help
status --help
help, h Shows a list of commands or help for one command
ADMINISTRATION:
create -n <name> -k <key-name> -i <ami> [-p <instance-profile-arn> -s <storage-size> -t <prefered-instance-type>]
bootstrap -t <template> [-n <name>]
destroy -n <name>
DEV-SPACE:
start -n <name> [-c <min-cpus> -m <min-memory> --max-price <max-price> -t <timeout>]
stop [-n <name>]
status [-n <name>]
list [-o <output>]

GLOBAL OPTIONS:
--help, -h show help (default: false)
--region value, -r value AWS region (default: "ap-south-1") [$AWS_REGION]
--help, -h show help (default: false)
```

# FAQ
Expand All @@ -34,19 +44,25 @@ GLOBAL OPTIONS:
A DevSpace is a elastic development environment on AWS. Because there is no need to build a machine if you can cheaply develop on the Cloud!


## How I can use it?

Please, follow the steps below: [How to bootstrap a Dev Space](BOOTSTRAP.md)

If you have any issue during the bootstrap progress, contact the author for more details on how to proceed.

## My progress is lost when I stop my DevSpace?

No! When you `stop` a DevSpace, the CLI only destroys the instance, leaving the attached EBS Volume intact.
When you call `start` again, the EBS Volume will be attached on the new instance and you can just continue from the point you stop.

This means you are running a _stateful_ workloads on spot instances.

## How I can use it?

```bash
go install github.com/felipemarinho97/dev-spaces@latest
```

Please, follow the steps in this document: [How to create a Dev Space](CREATING.md)

For the legacy way of bootstraping (the hard way), please, follow these steps: [How to bootstrap a Dev Space from scratch](BOOTSTRAPPING.md)

If you have any issue during the bootstrap progress, contact the author for more details on how to proceed.

# Exemples
### Starting a DevSpace

Expand All @@ -60,6 +76,8 @@ instance-id=i-001f2561a626115f5
instance-type=m1.large
```

DevSpaces will be listening by default on SSH port `2222`.

### Listing my DevSpaces

You can list the most recent (last 48h) created DevSpaces.
Expand All @@ -71,12 +89,53 @@ MySpace active sfr-fac050b3-2db3-4d2f-9efa-2403eb239650 2022-0
teste cancelled sfr-6bce6369-7a7b-4d0e-a65e-1498eb5aba90 2022-02-13T13:48:13Z
```

It's also possible to see all the created (regradless if they are active or not) DevSpaces using the command `list`.

```bash
$ dev-spaces list -o wide
SPACE NAME ID CREATE TIME VERSION [...] PUBLIC IP

MySpace lt-0639c1eccbb51e345 2022-07-07 22:55:01 1 [...] 52.23.206.106
arch lt-08fb20577838aa54d 2022-07-05 22:02:00 1 [...] 52.91.16.131
al2022-05 lt-0ca2cf57f06544590 2022-07-05 23:01:10 1 [...] -
```

### Terminating DevSpaces

When you are done, you can use the `stop` command to destroy the DevSpace instance(s).
When you are done, you can use the `stop` command to terminate the DevSpace instance(s).

Note: If you want to destroy all running DevSpaces, ommit the `--name` parameter.
Note: If you want to stop all running DevSpaces, ommit the `--name` parameter.

```bash
$ dev-spaces stop -n MySpace
```

This will not delete your files, just terminate the DevSpace instance.

---

### Creating a DevSpace

The example below shows an example on how to create a DevSpace using the `create` command.

```bash
$ dev-spaces create --name MySpace --key MyKey --ami ami-1234567890
```

You can also optionaly specify the instance profile ARN `--instance-profile-arn`, the storage size (in GBs) `--storage-size`, and the preferred instance type `--preferred-instance-type`. See all the options [here](CREATING.md#command-parameters).

The `--preferred-instance-type` option helps to create your DevSpace in an avaliability zone with the best possible price for that instance type (this is important because once created, the DevSpace will be locked in that zone).

### Destroying a DevSpace

The command below will destroy the DevSpace instance and all it's associated resources like EBS Volumes, Launch Templates, Security Groups, etc.

```bash
$ dev-spaces destroy -n MySpace
✓ Destroying security group sg-0b48ecc167b8a81c7 (0/-, 0 it/min)
✓ Destroying launch template lt-01d0e11ac8523614f (0/-, 0 it/min)
✓ Destroying volume vol-069210dc254fcdc6b (0/-, 0 it/min)
OK
```

**This WILL destroy everythng, including all your files.**
Loading

0 comments on commit 42e671e

Please sign in to comment.