Observability part 4 - Pulumi hands-on

Terraform is king... right?

In the DevOps world, everybody has heard about Terraform at some point. It's the most popular IaC tool according to the latest Stack Overflow survey. It utilizes HCL - HashiCorp Configuration Language. The syntax of HCL is inspired by JSON (you can even serialize HCL to JSON) and is supposed to be both human- and machine-friendly. However, after some time of working with HCL, you notice its limitations. It tricks you into thinking it can do whatever any scripting language can do. On one hand, there are variables, objects, functions, etc., but:

• loops work in a specific way
• values can’t be passed to other functions freely
• Terraform modules in their vanilla (non-Terragrunt) variants can be reused, but can't dynamically figure out input variables.

There are many more examples of why Terraform and HCL can feel stiff and restrictive. A solution to those issues might be Pulumi.

Purpose of this article

We decided to write an article about Pulumi as a part of our observability series. That's because we've used it to deploy infrastructure for Meerkat, our observability stack bootstrapping tool for JVM-based apps. We want to make sure you're familiar with the basics of Pulumi before you dive deep into Meerkat code - that way you will have an easier time understanding it.

What is Pulumi?

Pulumi is a whole ecosystem of different services and components, but the one we're going to focus on today is Pulumi SDK. Pulumi is an open-sourced, multi-language infrastructure as a code tool. It’s similar in concept to AWS CDK, with the difference that it's not bound to a specific public cloud. Like Terraform, it can be used with many providers to communicate with numerous APIs. Currently, Pulumi supports Node.js (both JavaScript and TypeScript), Python, Go, .NET (C#, F#, VB), Java, and Pulumi YAML (as configuration for Pulumi CLI). That means, in order to use Pulumi, you need to write a program that will utilize Pulumi as one of its dependencies.

JavaScript is the language we're going to be using for our hands-on today. Its syntax is simple and can be easily translated to other languages if necessary.

Getting started

Requirements:

• text editor (anything you like, I use VS Code)
• Pulumi CLI
• node.js (can be LTS, you can install it using your system package manager - however I prefer to use nodenv)
• Docker (we will deploy a Docker container for purposes of this tutorial)

First, we need to go to an empty directory and generate some files using a built-in Pulumi command:

pulumi new javascript

Pulumi has generated a few files for us. Basically, it's a new JavaScript application with Pulumi dependency added. Pulumi has also declared it wants to create a stack. A stack in Pulumi is an isolated instance of a program. One project can have many stacks associated with it. By issuing the pulumi new command, apart from generating files, we're creating a new stack. You can alternatively create a new stack by using the pulumi stack init <your-stack-name> command.

In the project’s directory, there are two YAML files - one of them stores the variables used by our stack and the other stores metadata and some configuration options.

We can run pulumi up. It will update our stack and try to create, modify, or delete any resources according to our Pulumi code.

If we run this command (press Enter if you didn't set any password in the installer), we should see something like this:

Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your passphrase to unlock config/secrets
Previewing update (dev):
     Type                 Name                 Plan
 +   pulumi:pulumi:Stack  pulumi-tutorial-dev  create

Resources:
    + 1 to create

info: There are no resources in your stack (other than the stack resource).

Do you want to perform this update?  [Use arrows to move, type to filter]
> yes
  no
  details

Great, Pulumi asks us if we want to create a stack. Let's do that.

Do you want to perform this update? yes
Updating (dev):
     Type                 Name                 Status
 +   pulumi:pulumi:Stack  pulumi-tutorial-dev  created (0.28s)

Resources:
    + 1 created

Duration: 1s

Our stack is created. Now, let's deploy something. Because you have Docker already running, we can use a Docker provider to deploy a container. Let's go to index.js and add the following lines:

"use strict";
import * as pulumi from "@pulumi/pulumi";
import * as docker from "@pulumi/docker";

const stack = pulumi.getStack();

We're importing two modules:

• base Pulumi (used in every Pulumi app)
• Docker provider (to pull and run memcached)

"use strict";
import * as pulumi from "@pulumi/pulumi";
import * as docker from "@pulumi/docker";

const stack = pulumi.getStack(); // programatically gets stack, we can delete this though
const app = "memcached";

const appImage = new docker.RemoteImage(`${app}Image`, { // we set up what image we want to pull 
    name: "memcached:1.6.29",
});

const appContainer = new docker.Container(app, { // we're creating a container out of our pulled image
    name: app,
    image: appImage.imageId,
});

We're using the ES6 import style, so if Pulumi tells you there's a ModuleStack error, try adding this to package.json:

{
    "name": "pulumi-tutorial",
    "main": "index.js",
    "type": "module", // <- this one
    "dependencies": {
        "@pulumi/docker": "^4.5.4",
        "@pulumi/pulumi": "^3.0.0"
    }
}

We can tidy up our code a bit:

"use strict";
import * as docker from "@pulumi/docker";
// we can remove an unused import because we got rid of the getStack() function below

// we removed the getStack() function, it was autogenerated and we weren't using it
const app = "memcached";
const appVersion = "1.6.29"; // a variable that will store the app version in one place

const appImage = new docker.RemoteImage(`${app}Image`, {
    name: `${app}:${appVersion}`, // we can use these variables to build a string with the image name and version
});

new docker.Container(app, { // we can remove the variable assignment here
    name: app,
    image: appImage.imageId,
});

Okay, let's run it and see what happens:

pulumi-tutorial ➤ pulumi up
Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your passphrase to unlock config/secrets
Previewing update (dev):
     Type                         Name                 Plan
 +   pulumi:pulumi:Stack          pulumi-tutorial-dev  create
 +   ├─ docker:index:Container    memcached            create
 +   └─ docker:index:RemoteImage  memcachedImage       create

Resources:
    + 3 to create

Do you want to perform this update?  [Use arrows to move, type to filter]
  yes
> no
  details

Do you want to perform this update? yes
Updating (dev):
     Type                         Name                 Status
 +   pulumi:pulumi:Stack          pulumi-tutorial-dev  created (6s)
 +   ├─ docker:index:RemoteImage  memcachedImage       created (5s)
 +   └─ docker:index:Container    memcached            created (0.36s)

Resources:
    + 3 created

Duration: 7s

Pulumi pulled an image for us and deployed a memcached Docker container.

Once we're done with experimenting, we can use pulumi destroy and our stack will be wiped clean of any resources.

What I like about Pulumi

Pulumi is a very fun and refreshing way to deploy infrastructure. Having used Terraform for years, it feels good to be writing code using a general-purpose language like JavaScript. This can be especially useful if you're a developer or a DevOps person working in a team of developers, since there's a high chance collaboration will be easier using JS or any other supported language instead of HCL. The feedback loop also feels snappier to me. I must mention that using Pulumi documentation was more pleasant than the one for Terraform.

Check other parts of our Observability series:

• part 1: building blocks overview
• part 2: building a local try-me environment
• part 3: configuring OpenTelemetry components