I was speaking to the CEO of a developer tools company earlier this year. He’s an engineer and still occasionally contributes to his company’s product. He told me that the biggest obstacle to contribution is his local development environment. As an occasional contributor, his environment is always a little behind the current release. To make the contribution he wants, he first has to spend time updating and configuring his local development environment. This activity often consumes the time he’d allocated to contribute and dulls his enthusiasm for contributing.

For a regular line engineer, this is more than an annoyance. A large portion of your productivity is having your local development environment running and available. I’ve seen engineers lose hours or even days debugging local development environment issues. It’s even worse when starting a new job. There’s little more disappointing and disempowering than to excitedly begin your job and then get stalled because you can’t get your local development environment working. Every company should aim to get their developers as productive as quickly as possible. How long does it take a brand new developer to get code in the hands of customers when they start at your company?  Is it measured in months, weeks, days or hours?

Let’s explore what people do wrong with local development environments and then investigate some approaches to reduce the overhead and the risk of lost developer productivity.

But First: Other Onboarding Best Practices

Before we jump in too far into building better development environments, it’s important to highlight that this is just a facet of onboarding best practices. If you want your new hire to be successful - maximize their potential success by ensuring you address the other aspects too.

Mentoring

Providing every new engineer with a mentor they can work with, shadow, and learn from is key to onboarding. Sure, a new engineer can dive straight into the code base or your workflow management tool to find out how things work in your organization. But this is time-consuming and often leads people down paths that they could have avoided with input from a more experienced colleague. Mentoring, especially combined with pair programming/pairing, is a high bandwidth way to get a new colleague contributing quickly and effectively. A mentor is also the perfect person to ensure your new colleague has access to the right systems, tools, and resources they need to be successful and productive.

Documentation

No mentoring is perfect, and a mentor isn’t going to be available all the time. Documentation fills that gap. Document your team culture, coding standards, pull request review protocol, CI/CD process, APIs, operations, and anything else that makes your engineering team function. This documentation is crucial for remote and distributed teams. Information can flow less smoothly remotely; if someone in a different time zone needs information from a colleague in another time zone, getting an answer can be delayed.

Communication

Ensure you have good lines of communication (and they are documented!). Ensure a new colleague is invited to the correct channels, added to the appropriate meetings, email groups, and has 1:1s scheduled with their manager. Their manager (and their mentor) should regularly communicate and check-in with a new hire in their first few weeks. Often folks are shy or uncomfortable about asking questions: make it clear there’s no shame in asking questions and that you’re happy to make time to get them started.

Culture

Introduce new people to the organization or team and set up introductory meetings with everyone they’ll be working with. This process is essential in remote and distributed teams. Help them understand how the team works, what the culture is like, and welcome them. Starting a new job is scary.

With all this squared away, let's get back to local dev environments.

What is a local development environment?

A local development environment is the source code and infrastructure needed to develop your application, available locally on your laptop or desktop. It’s usually one or more checked-out Git repositories combined with the means to run your application. This can be as simple as a script to run an HTTP server right up to a multi-pod Kubernetes cluster containing databases, caches, and web services. You can think about a local development environment as a test environment lite.

NOTE: We’re primarily going to be discussing local development environments for more modern development. There’s generally no easy way to simulate your ERP or legacy AS/400 system locally.

So why have local development environments?

Local development environments allow developers to write code remotely without needing a shared, centralized development environment that requires maintenance. It allows developers to run and test code with (hopefully) no risk of breaking that shared environment or potentially interfering with customer-facing environments. Finally, it is generally low-cost, relying on an asset you’ve already paid for, and hence potentially cheaper than running the Cloud resources needed for one or more development environments. 

How did we get here or “the bad place?"

The first development environments built were often shell scripts that locally installed software using tools like Homebrew. Later iterations still relied on locally installing software packages but automated the installation process, often using configuration management tools like Puppet, Chef, and Ansible. Or “box builder” projects that wrapped around these sorts of tools. Many organizations still use this locally installed software approach. These environments are often high maintenance and cumbersome to maintain. They are very fragile and are broken by operating system upgrades, the installation or upgrade of software, or by adding or upgrading the dependencies inside your application. Additionally, your production application often runs on a different platform than your local development environments; for example, your production environment is Linux and your development environment is MacOS. This difference means that an application dependency that works on Linux might not exist, might not work, or might not work the same way on macOS.

That’s why many folks only upgrade locally installed environments when necessary, playing a kind of software Jenga where any change could bring down the whole environment. This fragility also means that the first person to discover a new breakage is usually an incoming developer installing the environment for the first time, sparking the kind of onboarding pain no one wants. It’s a common statement to hear:

“Oh, the new engineer can or will fix, document, or update the local development environment and leave it better than they found it.” 

Essentially you’d be dumping your unmaintained and painful pile of technical debt onto a new and potentially inexperienced engineer and expecting them to make do. This introduction is not the experience you want for a new teammate, nor should your team take the productivity hit in time and effort as they get ramped up.

But things get better, right?

If I’ve just described your current local development, then start a project to rework that environment right now. The locally installed approach is painful to maintain, increases friction, and reduces developer productivity. If you do start that project, there are, thankfully, some clear directions for the evolution (or replacement) of your local development environment. The two paths worth exploring here are:

• Containerized local development environments

• Pseudo-local environments, which combine local editing and remote compute resources.

Let’s look at each.

Containerized local development environments

A revolutionary change for local development came with containerization. One of the reasons containers became so popular quickly was that they allow simple replication of application environments and platforms locally on your laptop or desktop. Instead of installing all your myriad dependencies locally, you’re now only required to install the container platform. All your dependencies bundle into a container image. You can also run multiple containers to simulate different components in an environment: web services, databases, caches, etc.

This capability has gotten even more sophisticated with orchestration tools like Docker Stacks, Docker clients able to run a local Kubernetes cluster, VSCode’s devcontainer.json feature, and tools like Minikube that allow running Kubernetes locally on a variety of virtual machines.

This is a boon, especially for folks who use containers or Kubernetes to run their production applications since you likely already have container images that contain your application’s configuration and dependencies. You can use these or adapt them to run your development environment. In most cases, you would then mount your source code or development binary inside the containerized environment.

If you’ve already got a process in place to build your container images, then you’ve also got a semi-automated update process for your development environments. If you don’t have a process in place, you can expand your CI/CD pipeline to include image generation. A common solution is to build your production image and then use that image as an inherited base for your development image, including any dev tools you might want to use in your environment.

You can then configure your development environment to download updated images upon start or request and ensure everyone is working from the latest version of your dependencies and applications. This process eliminates the need for manual updates and multi-platform configuration for dependencies.

Kubernetes is complex, especially its configuration. Hence achieving a productive workflow can be complicated, and some aspects of locally installed environments—for example, hot reload of applications and services—require additional work. There are several tools available to help you manage your workflow locally. These include, amongst others, Skaffold, Tilt, Garden, and Telepresence. All of these tools provide wrappers around Kubernetes, allowing for the auto-detection of code changes in your environment to rebuild images and restart or reload Kubernetes resources.

There are significant advantages to replicating your application platform locally. There is, however, another alternative that combines the benefits of containerization with some centralized management.

Pseudo-local environments

Pseudo-local environments combine a local or even browser-based editor or IDE with a remote compute destination, usually a standalone virtual machine or cloud instance used by a single developer where your actual code and applications execute. These environments have become common in some of the larger web and Cloud companies with both Google and Facebook being exponents of this approach. Like a local containerized environment, you can replicate your production platform, Kubernetes for example, inside that remote instance. Examples of pseudo-local environments include GitHub’s Codespaces, Amazon’s AWS Workspaces, or solutions that use standard cloud or VM instances. These tools can be partnered with an editor or IDE that supports remote editing; for example, Visual Studio Code and CLion have deep remote development support.

These environments are both simpler and more complex than local containerized environments. They are more straightforward in that they usually just require a local editor or IDE and some means to connect to the remote compute resources installed locally. They are also potentially more complex in that you now need to administer and secure those remote resources.

So why choose a pseudo-local environment? Put simply: compute power, enhanced remote working capabilities, centralized management, updating, and control of developer environments. 

First, laptops—the standard-issue device for most engineers—have CPU and memory constraints. That complex, micro-services architecture running in Kubernetes underneath a hypervisor is CPU and memory-intensive. In some cases, compiling or running local tests can be time-consuming and resource-intensive. A remote instance can generally be scaled up as needed to accommodate an additional need for capacity. In a recent role, a large Rust application took sufficiently long to compile on my laptop that I had time to get coffee. We eventually moved development to remote instances.

Next, more and more engineers are remote due to the Covid pandemic and, in general, as people seek more flexible working arrangements. Onboarding is far easier if you don’t have to set up a local development environment but instead send credentials and access instructions for a remote development environment. It’s generally faster to spin up or connect to a previously provisioned development environment than wait for a local download or wait for local dependencies to complete installation. It’s also simpler to off-board developers when they have few local assets on their laptop or desktop.

You can also manage the configuration of your remote resources centrally. You can include the configuration and image definitions of your development environment with your application code, much like you might do with your infrastructure as code definitions. Rather than any configuration on a developer’s laptop, you can update container images or instance images and not need individual developers to deploy or manage any updates. Like a local containerized installation, changes to your configuration can be deployed via your existing CI/CD pipeline. Albeit, you only need to worry about building for one type of target. This also means that if an engineer breaks their remote development it’s easier to reset it back to a good state by simply replacing or rebuilding the instance.

Remotely managed instances also mean one central point for security controls. You control the authentication, transport security, and network controls enforced on your remote instances. These controls allow engineers to limit what is stored on their devices and, if you’re subject to any compliance regime, helps you manage the security of your development lifecycle.

Summary

The CEO I talked to, and his team, decided they were sick of the frustration their local development environment was causing them. They scrapped their local development environment totally and migrated it to a pseudo-local environment using remote instances with local editing. Now a new engineer receives a laptop, with login credentials for their personal development environment, installs the editor of their choice locally, and connects to their ready-made development environment. Their engineers can now have new code running in the hands of customers from day one.  Better local dev environments, and thus better onboarding, is possible if you commit to making a change.