easy-cass-lab

This is a tool to create lab environments with Apache Cassandra in AWS.

We use packer to create a single AMI with the following:

• Multiple versions of Cassandra
• bcc tools, learn about these tools here
• async-profiler, learn about it here
• easy-cass-stress
• AxonOps agent (free monitoring up to six nodes)

easy-cass-lab provides tooling to create the AMI and provision the environments.

Pre-requisites

The following must be set up before using this project:

• Setup AWS Account API Credentials
• Install Docker

Usage

To install easy-cass-lab, you can use Homebrew or download a release. Alternatively clone the project and build it.

Install A Release using Homebrew

brew tap rustyrazorblade/rustyrazorblade
brew install easy-cass-lab

Download A Release

Download the latest release and add the project's bin directory to your PATH.

export PATH="$PATH:/Users/username/path/to/easy-cass-lab/bin"

You can skip to Build the Universal AMI

Optional: Build the Project

If you've downloaded a pre-built release, you can skip this step.

If you're using the project repo, you'll need to build the project. Fortunately, it's straightforward.

The following command should be run from the root directory of the project. Docker will need to be running for this step.

git clone https://github.com/rustyrazorblade/easy-cass-lab.git
cd easy-cass-lab
./gradlew shadowJar installdist

Build the Universal AMI

Using easy-cass-lab requires building an AMI. This can be done once, and reused many times. The AMI should be rebuilt when updating easy-cass-lab.

The first time you build an image, easy-cass-lab will ask you for your AWS credentials.

bin/easy-cass-lab build-image

At the end, you'll see something like this:

==> Builds finished. The artifacts of successful builds are:
--> cassandra.amazon-ebs.ubuntu: AMIs were created:
us-west-2: ami-abcdeabcde1231231

That means you're ready!

Optional: Set the AMI

When you create a cluster, you can optionally pass an --ami, or set EASY_CASS_LAB_AMI.

If you don't specify an AMI, it'll use the latest AMI you've built.

Read the Help

Run easy-cass-lab without any parameters to view all the commands an options.

Create The Environment

First create a directory for the environment, then initialize it, and start the instances.

This directory is your working space for the cluster.

mkdir cluster
cd cluster
easy-cass-lab init -c 3 -s 1 myclustername # 3 node cluster with 1 stress instance

You can start your instances now.

easy-cass-lab up 

To access the cluster, follow the instructions at the end of the output of the up command:

source env.sh # to setup local environment with commands to access the cluster

# ssh to a node
ssh cassandra0
ssh cassandra1 # number corresponds to an instance
c0 # short cut to ssh cassandra0

Select The Cassandra Version

While the nodes in the cluster are up, a version isn't yet selected. Since the AMI contains multiple versions, you'll need to pick one.

To see what versions are supported, you can do the following:

easy-cass-lab list

You'll see 3.0, 3.11, 4.0, 4.1, and others.

Choose your cassandra version.

easy-cass-lab use 4.1

easy-cass-lab will automatically configure the right Python and Java versions on the instances for you.

Optional: Modify the Configuration

You'll see a file called cassandra.patch.yaml in your directory. You can add any valid cassandra.yaml parameters, and the changes will be applied to your cluster. The listen_address is handled for you, you do not need to supply it. The data directories are set up for you.

You can also edit jvm.options. Different versions of Cassandra use different names for jvm.options. easy-cass-lab handles this for you as well.

easy-cass-lab update-config # uc for short

Start The Cluster

Start the cluster. It will take about a minute to go through the startup process

easy-cass-lab start

Log In and Have Fun!

Important Directories:

# The ephemeral or EBS disk is automatically formatted as XFS and mounted here.
/mnt/cassandra 

# data files
/mnt/cassandra/data

# hints
/mnt/cassandra/hints

# commitlogs
/mnt/cassandra/commitlog

# flame graphs
/mnt/cassandra/artifacts

Multiple cassandra versions are installed at /usr/local/cassandra.

The current version is symlinked as /usr/local/cassandra/current:

ubuntu@cassandra0:~$ readlink /usr/local/cassandra/current
/usr/local/cassandra/4.1

This allows us to support updating, mixed version clusters, A/B version testing, etc.

Profiling with Flame Graphs

https://rustyrazorblade.com/post/2023/2023-11-07-async-profiler/

Using easy-cass-lab env.sh, you can run a profile and generate a flamegraph, which will automatically download after it's complete by doing the following:

c-flame cassandra0

The data will be saved in artifacts/cassandra0

Or on a node, generate flame graphs with flamegraph.

There are several convenient aliases defined in env.sh. You may substitute any cassandra host.
You may pass extra parameters, they will be passed along automatically.

Command	Description
c-flame	CPU Flame graph
c-flame-wall	wall clock profiling, picks up I/O, parked threads filtered out
c-flame-compaction	More specific wall clock profiling to compaction
c-flame-offcpu	Just tracks time spent when cpu is unscheduled, mostly I/O
c-flame-sepworker	Request handling, by default this is CPU time. You can add -e wall to make it wall time.

Aliases

On each node there are several aliases for commonly run commands:

command	action
c	cqlsh (auto use the correct hostname)
ts	tail cassandra system log
nt	nodetool
d	cd to /mnt/cassandra/data directory
l	list /mnt/cassandra/logs directory
v	ls -lahG (friendly output)

Shut it Down

To tear down the entire environment, simply run the following and confirm:

easy-cass-lab down

Tools

bcc-tools is a useful package of tools

https://rustyrazorblade.com/post/2023/2023-11-14-bcc-tools/

Interested in contributing? Check out the good first issue tag first! Please read the development documentation before getting started.