Infrastructure as code

[ghost]

References:

• https://en.wikipedia.org/wiki/Configuration_management
• https://en.wikipedia.org/wiki/Version_control
• https://en.wikipedia.org/wiki/Infrastructure_as_code
• GitOps

I don't think it is possible to have a "dev ops" environment without some kind of Configuration Management Tool.
They make it possible to configure an environment through scripts.
The big advantage of a tool like this is that new virtual machine or container instances can be easily created or updated.
The most common solutions here I think are Ansible, Puppet, Chef and Salt.

It is probably best to pick one tool and have practical scripting exercises how to an enviroment can be configured programmatically.

[monperrus]

Hi Göran,

Thanks for your suggestion.

Among Ansible, Puppet, Chef and Salt, what's the one with the best documentation, in particular getting started and tutorials pages to your opinion?

[monperrus]

Terminology: infrastructure as code (IaC) #22

[monperrus]

Reading material:
GitOps: A Path to More Self-Service IT
https://cacm.acm.org/magazines/2018/9/230599-gitops/fulltext

"Use Cases for GitOps. DNS is an obvious place to start, as are VM creation, container maintenance and orchestration, firewall rules, website updates, blog posts, email aliases and mailing lists, and just about any virtual infrastructure or one with a configuration file or API."

[MatsJonsson]

We've done this with Ansible and Salt in a couple of big projects, only to end up with huge amounts of stuff just to get the actual stuff working.

Terraform

The move to Terraform and Hashicorp Configuration Language (HCL) made life a lot easier https://www.terraform.io/.

[bittermandel]

We're running Salt extensively on thousands of nodes. It's working great for us and gives us great power in controlling our "standard" environments

https://saltproject.io/

[alanmcg]

I have seen this implemented with puppet, both badly and very well, in different organisations.

[monperrus]

How good is your puppet? an empirically defined and validated quality model for puppet
https://pure.tudelft.nl/portal/files/37386939/how_good_is_your_puppet.pdf

[bbaudry]

Molecule is a tool for testing ansible scripts
https://molecule.readthedocs.io/en/stable/#

[monperrus]

Using Testinfra with Ansible to verify server state
https://opensource.com/article/19/5/using-testinfra-ansible-verify-server-state

[monperrus]

Python library to create AWS CloudFormation descriptions
https://github.com/cloudtools/troposphere

[monperrus]

Pulumi - Infrastructure as Code
https://www.pulumi.com/

[bbaudry]

The 'as Code' Activities: Development Anti-patterns for Infrastructure as Code
http://arxiv.org/abs/2006.00177

[bbaudry]

Automating web applications proxying, DNS registration and TLS termination with ansible

https://bpetit.nce.re/2018/03/automating-web-applications-proxying-dns-registration-and-tls-termination-with-ansible/

[bbaudry]

kustomize lets you customize raw, template-free YAML files for multiple purposes

[monperrus]

Free version of Morris' book on infrastructure as code: https://us-east-1.linodeobjects.com/marketing-assets/Infrastructure_as_Code_2E-ER_Linode.pdf

[monperrus]

See accepted papers at CONFLANG, workshop on the design, the theory, the practice and the future evolution of configuration languages.

https://2021.splashcon.org/home/conflang-2021#event-overview

[monperrus]

Luke Hoban on Infrastructure as Code IEEE Software

[monperrus]

Dozer: Migrating Shell Commands to Ansible Modules via Execution Profiling and Synthesis. (arXiv:2203.12065v1 [cs.SE])

[monperrus]

CUE: Configure Unify Execute "Validate, define, and use dynamic and text-based data"
https://cuelang.org/

[monperrus]

Dhall is a programmable configuration language that you can think of as: JSON + functions + types + imports
https://dhall-lang.org/

[matsskoglund]

Tool for vulnerability scanning of Infrastructure as Code https://www.checkov.io/

[monperrus]

Modus is a language for building Docker/OCI container images, it uses logic programming to express interactions among build parameters, specify complex build workflows, automatically parallelise and cache builds, help to reduce image size, and simplify maintenance.
https://modus-continens.com/

Paper: "Modus: a Datalog dialect for building container images."

cc/ @mechtaev @barr

[mechtaev]

Nickel's purpose is to automate the generation of static configuration files - think JSON, YAML, XML, or your favorite data representation language - that are then fed to another system. It is designed to have a simple, well-understood core: it is in essence JSON with functions.
https://nickel-lang.org/

This is relevant to Dhall.

[mechtaev]

Earthly is a CI/CD framework that allows you to develop pipelines locally and run them anywhere. Earthly leverages containers for the execution of pipelines. This makes them self-contained, repeatable, portable and parallel.

[monperrus]

HashiCorp Packer
Packer is a free and open source tool for creating golden images for multiple platforms from a single source configuration.
https://www.packer.io

[monperrus]

GLITCH: an Intermediate-Representation-Based Security Analysis for Infrastructure as Code Scripts
http://arxiv.org/pdf/2205.14371

[monperrus]

GitOps: The Evolution of DevOps? (IEEE Software)

[monperrus]

Dozer: Migrating Shell Commands to Ansible Modules via Execution Profiling and Synthesis.

[monperrus]

Infrastructure as code for dynamic deployments.

[bbaudry]

Material for the Ansible Up & Running book: https://github.com/ansiblebook

[monperrus]

Infrastructure From Code: The Next Generation of Cloud Lifecycle Automation

[monperrus]

Skaffold handles the workflow for building, pushing and deploying your application, allowing you to focus on what matters most: writing code.
https://skaffold.dev/

[monperrus]

Automatically Generating Dockerfiles via Deep Learning: Challenges and Promises

[monperrus]

Mars is an infrastructure-as-code tool for Ethereum
https://github.com/TrueFiEng/Mars

[monperrus]

CircleCI orbs: package management ecosystem for CircleCI configuration
https://circleci.com/docs/orb-intro/

[bbaudry]

Vulnerability Scanner for Containers and other Artifacts with trivy

[bbaudry]

Mining for Cost Awareness in the Infrastructure as Code Artifacts of Cloud-based Applications: an Exploratory Study.
http://arxiv.org/abs/2304.07531

[monperrus]

Testing idempotence for infrastructure as code
https://dsg.tuwien.ac.at/team/hummer/docs/2013-middleware-iac.pdf

[monperrus]

werf: CD and gitops for Kubernetes
https://werf.io/

tutorial by https://killercoda.com/jarns-zeiher/scenario/Simple-End-to-End-GitOps-with-Werf

[monperrus]

Automated Code generation for Information Technology Tasks in YAML through Large Language Models. (arXiv:2305.02783v1 [cs.SE])

[monperrus]

OpenTofu lets you declaratively manage your cloud infrastructure.
https://github.com/opentofu/opentofu

[monperrus]

Infrastructure-as-Code Ecosystems
https://link.springer.com/chapter/10.1007/978-3-031-36060-2_9

[monperrus]

Starlark (formerly known as Skylark) is a language intended for use as a configuration language. It was designed for the Bazel build system, but may be useful for other projects as well.
https://github.com/bazelbuild/starlark

Heavily used in the tensorflow repo

[monperrus]

Rego is a general-purpose policy language, which means that it works for any layer of the stack and any domain. The primary purpose of Rego is to accept JSON/YAML inputs and data that are evaluated to make policy-enabled decisions about infrastructure resources, identities, and operations.

https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-api-access-authorization/rego.html

[monperrus]

Crossplane is a framework for building cloud native control planes without needing to write code.
https://www.crossplane.io/

[bbaudry]

qq, a query language that supports the main formats for config files
https://github.com/JFryy/qq

[monperrus]

Bicep is a declarative language for describing and deploying Azure resources
https://github.com/Azure/bicep