ZIO environment: episode 3

ZIO is nearing the release of version 2.0; one of the headline features is an improved and simplified approach to managing dependencies. In the ZIO world, this is known as the environment.

We’ve covered two previous incarnations of the ZIO environment. The feature has been getting better with each release; since the changes in 2.0 are really significant, our ZIO-env-series definitely needs a new episode! If you’re interested in history, here are the two previous parts: 1 and 2.

The basics

In ZIO, a computation is described as a value of type ZIO[R, E, A]. It’s helpful to think about such a value as an asynchronous version of a function R => Either[E, A]. The meaning of the three type parameters is:

• R: the environment in which the computation takes place; that is, the dependencies of the program
• E: the type of the errors that might occur while running the program
• A: the type of the result of running the program

Here, we’ll focus on the first type parameter, R.

A simple application skeleton

Following the previous episodes, let’s try to create a skeleton for a simple application. Our application will emulate registering users and consists of a couple of components:

• DBConfig is a data class with the configuration of the database
• ConnectionPool is an external resource that we need to manage and which requires configuration
• DB: using a ConnectionPool, we can build a higher-level service, which will allow us to run database queries
• UserModel: this component gives us the possibility to insert new users to the database
• UserNotifier is used to send email notifications to users
• UserRegistration implements the main flow, that is first inserts a new user using UserModel and then sends a welcome email using UserNotifier

Dividing the application code in this manner is quite common: we’ve separated the infrastructure layer (DBConfig, ConnectionPool and DB) from the application layer, which is further divided into internal components (UserModel, UserNotifier) and the main entry point to our business logic, UserRegistration.

Implementing using ZIO

In the latest incarnation of ZIO, the style of implementing “services” and “business logic” is different. I use quotes on purpose here, as the meaning of these terms is overloaded, so we can’t really hope for a precise definition of either. We’ll see where intuition leads us, though.

In ZIO 2.0, services should be implemented using constructor-based dependency injection and the business logic should leverage the environment for dependency management. Here’s how John de Goes, the main architect of ZIO, summarises this:

The 3 Laws of ZIO Environment:

1. Methods inside service definitions (traits) should NEVER use the environment

2. Service implementations (classes) should accept all dependencies in constructor

3. All other code ('business logic') should use the environment to consume services https://t.co/iSWzMhotOv

— John A De Goes (@jdegoes) November 23, 2021

Services

Let’s start from the infrastructure layer. We’ll use the service approach to create instances of ConnectionPool and DB. Let’s start with the latter. First, we’ve got a trait that describes the interface of our service:

trait DB {
  def execute(sql: String): Task[Unit]
}

When executing an SQL query, as a result we get a Task, which is an alias for a ZIO computation that can throw any exception, doesn’t use anything from the environment, and returns a Unit: type Task[A] = ZIO[Any, Throwable, A]. This follows the first no-environment “law” that John describes above.

Next, let’s implement the service. We’ll resort to a dummy implementation, which, however, will require an instance of a ConnectionPool. We use constructors to express this dependency:

object DB {
  // implementation
  case class RelationalDB(cp: ConnectionPool) extends DB {
    override def execute(sql: String): Task[Unit] =
      Task {
        println(s"Running: $sql, on: $cp")
      }
  }
}

Instead of talking to a database, we are printing to the console (which, in ZIO, should be done using the Console service, but for the needs of our example we’ll stick with println).

Layers

But that’s not all! We still need to provide a way to consume our service, shielding clients from the details of the service’s construction. That’s the task of a ZLayer. Drawing many similarities with ZIO, a ZLayer[RIn, E, ROut] describes how to create ROut values, given RIn values, while E errors might occur. The creation might be effectful (that is, creating a value might be described using a ZIO). It might also need allocation and deallocation steps (a resource; in such a case, you typically deal with a wrapped ZManaged).

Moreover, layers can be composed, yielding a layer that produces multiple dependencies. For example, you might have a layer that creates a Clock with DB, which means that the layer creates both a Clock instance and a DB instance. Note that the R/ROut type does not correspond directly to the type of the object you get from the layer: there is no instance of Clock with DB, as you can’t really create one given a Clock and DB. Instead, the A with B encoding specifies that the dependency map that resides inside the layer has entries both for A and B dependencies.

Going back to our running example, here’s the layer for our DB service:

object DB {
  // layer
  val live: ZLayer[ConnectionPool, Throwable, DB] = ZLayer {
    for {
      cp <- ZIO.service[ConnectionPool]
    } yield RelationalDB(cp)
  }
}

Finally, each service method needs an accessor method so that it can be used in the business logic, using ZIO’s environment; note that in the result, the dependency is expressed using the environment type parameter:

object DB {
  // accessor
  def execute(sql: String): ZIO[DB, Throwable, Unit] = 
    ZIO.serviceWithZIO(_.execute(sql))
}

Resources

We can implement ConnectionPool in a similar way, with one notable difference - the pool is a resource that needs explicit allocation and deallocation logic. Hence, we are first creating a ZManaged, and later converting this to a ZLayer:

// procedural, externally defined interface
class ConnectionPool(url: String) {
  def close(): Unit = ()
  override def toString: String = s"ConnectionPool($url)"
}

// integration with ZIO
object ConnectionPoolIntegration {
  def createConnectionPool(
      dbConfig: DBConfig): ZIO[Any, Throwable, ConnectionPool] =
    ZIO.attempt(new ConnectionPool(dbConfig.url))

  val closeConnectionPool: ConnectionPool => ZIO[Any, Nothing, Unit] =
    (cp: ConnectionPool) => 
      ZIO.attempt(cp.close()).catchAll(_ => ZIO.unit)

  def managedConnectionPool(
      dbConfig: DBConfig): ZManaged[Any, Throwable, ConnectionPool] =
    ZManaged.acquireReleaseWith(
        createConnectionPool(dbConfig))(closeConnectionPool)

  val live: ZLayer[DBConfig, Throwable, ConnectionPool] =
    ZManaged.service[DBConfig]
      .flatMap(dbConfig => managedConnectionPool(dbConfig))
      .toLayer
}

Business logic

When implementing the business logic, we’re no longer using constructors, layers, and the accessor, but instead we are leveraging ZIO’s environment. The components of our application service are defined in a completely different manner.

We don’t even need to define them as classes; instead, they can end up as objects. The logic is still parameterized, though this parameterization moves from constructor parameters to a non-trivial ZIO environment type. After all, both constructors and ZIO’s environment are in some way equivalent to an ordinary function.

Without further ado, here’s our business logic:

object UserModel {
  def insert(u: User): ZIO[DB, Throwable, Unit] = 
    DB.execute(s"INSERT INTO user VALUES ('${u.name}')")
}

object UserNotifier {
  def notify(u: User, msg: String): ZIO[Clock, Throwable, Unit] = {
    Clock.currentDateTime.flatMap { dateTime =>
      Task {
        println(s"Sending $msg to ${u.email} @ $dateTime")
      }
    }
  }
}

object UserRegistration {
  def register(u: User): ZIO[Clock with DB, Throwable, User] = {
    for {
      _ <- UserModel.insert(u)
      _ <- UserNotifier.notify(u, "Welcome!")
    } yield u
  }
}

Thanks to the variance of the environment type, everything infers nicely and, in the end, the registration method ends up being a description of a computation, which needs two dependencies: a Clock and a DB instance.

Tying everything together

We still need to run our application. For this, we need to implement a main method. When extending ZIOApp, this ends up being quite straightforward, as we only need to provide the environment type, the layer with the dependencies and the description of the application itself. Here’s how this looks for our example:

object Main extends ZIOApp {
  override type Environment = DB

  override val tag: Tag[Environment] = Tag[Environment]

  override val layer: ZLayer[ZIOAppArgs, Any, DB] =
    ZLayer.make[DB](
      ConnectionPoolIntegration.live,
      DB.live,
      ZLayer.succeed(DBConfig("jdbc://localhost"))
    )

  override val run: ZIO[DB with Clock, Any, Any] =
    UserRegistration
      .register(User("adam", "adam@hello.world"))
      .map { u => println(s"Registered user: $u (layers)") }
}

The entire code is available on GitHub, in a runnable form so that you can experiment at your leisure.

The invocation of UserRegistration.register is quite straightforward, but let’s examine layer creation in more detail!

Making layers

A really nice feature of ZIO 2.0 is how creating layers is simplified. As demonstrated in the above example, to create a layer that yields a DB instance, we simply provide all of the necessary layers to the make macro (Scala code that is invoked at compile-time). The macro figures out if all the required dependencies are met, and if not, prints out a user-friendly error message. For example, if we omit the layer providing the configuration:

Or if we provide unused dependencies, there’s a warning:

We can provide the layers in any order. make will figure out the correct sequencing to create the DB instance, given DBConfig, DBConfig => ConnectionPool and ConnectionPool => DB values.

But what’s a service?

This all works nicely, but quite quickly, a fundamental question arises. Where’s the boundary between a service and business logic? When is a code fragment a service, and when can it be considered business logic? I’m definitely not the only one asking this question!

IMO "business logic" is just the implementation of another service, so no need for an environnement. I only use an environment for request-level data: tracing id or database transaction

— etorreborre (@etorreborre) November 24, 2021

I’m afraid there’s no clear answer or definition that might be given. This ambiguity might be a source of a number of “rules of thumb” and might spark many more or less productive discussions. I think at least some “official” guidance on when a given functionality can be considered e.g. a service will be needed to direct such dilemmas to a fruitful resolution.

Moving further from facts to opinion, on one hand, defining top-level application services using the environment (as in UserRegistration) does feel more light-weight than having to add a dependency to the constructor and later using it in the method body. Type inference is great: when using a number of services, the compiler or IDE will typically suggest the correct type of the resulting program.

However, this convenience has a cost - creating a service does require some boilerplate, as in addition to the trait + implementation, we need to provide the layer value and an accessor for each of the trait’s methods. Hence, you’ll probably want to minimise the number of services, or use services from libraries.

Which brings us to constructors - is adding dependencies via constructors really that bothersome that it justifies the business logic / service split, the machinery of the ZIO environment, and the dilemmas that we’ll have to solve to decide what’s the nature of a given piece of code? I’m not convinced - maybe not as ergonomic as ZIO’s environment, but passing dependencies via constructors is really simple - essentially a way to extract common parameters of a group of methods, and this typically would be my first choice when writing applications. But I’ll be happy to be proven wrong!

What the caller knows

There’s an important difference between dependencies passed via the environment and using constructors. In the first case, the caller knows exactly what the downstream logic needs. In the second, this is hidden from the caller.

I’ve written about this in an article on how the reader monad is not an alternative, but a complement to dependency injection, as well as in the previous article on the ZIO environment, hence if you’re interested in that subject, please take a look there.

Contextual computations

There is, however, one area that I think is a very promising application of the ZIO environment, especially in its latest, simplified form. Eric hinted as to what this might be in the tweet above, and John echoes this:

In ZIO 2, layers become eliminators for environmental effects:

val effect2 = clockLayer(effect)

...will eliminate `Clock` from the environment, supplying it to the effect (and adding whatever environmental effects the `Clock` layer requires).

— John A De Goes (@jdegoes) November 24, 2021

There are different “kinds” of objects in an application. So far, we’ve been mainly interested in the graph of dependencies - the DB, UserNotifier, etc. instances. There are also data objects, such as the User instance that is passed to the register method. I think it’s quite intuitive that these two instances are different in their nature. The former are static nodes in the dependency graph, the latter - dynamic, changing with each invocation.

But static-dynamic is only one axis along which objects differ. Another is “how global” an object is. A User instance is very local (and dynamic); on the other hand, there are objects that provide context for a given invocation and are propagated along multiple method invocations (but they are still dynamic, unlike static dependencies). Eventually, such context parameters are eliminated using an appropriate context handler.

To make this more concrete, let’s look at executing database queries using a relational database. This typically requires a transactional context: something that represents an open connection or a running transaction. An approach of representing such transactional computations as values has been pioneered by Slick (DBIOAction) and Doobie (ConnectionIO), however, in both these cases, dedicated data types have been created to describe the computations. Because of that, mixing transactional and non-transactional logic is difficult, if at all possible.

ZIO opens up a possibility to improve this situation. A value of type ZIO[Connection, Throwable, User] might be a description of a transaction fragment, which reads a user from the database. We could potentially compose multiple such fragments into bigger transactions. In the end, some kind of a transactor service might eliminate this effect, taking a connection from the connection pool, starting & finally committing the transaction.

I don’t think this has been implemented as a library yet, but I’m keenly waiting!

Note that this might be viewed as an alternative to Scala 3’s context functions. The following twitter thread might provide some insights!

My intuition is that the environment should come first, i.e. `R => Input => E|A`. That way, `Input` can depend on `R`. E.g. the environment might then define a type that's used in the input. But YMMV.

— Martin Odersky (@odersky) August 11, 2021

Changes compared to 1.0

If you’re interested in comparing the 1.0 and 2.0 versions of the same code, take a look at this diff. It migrates the implementation of the above example from ZIO 1.0 to 2.0

One thing that is immediately clear is that a lot of code got removed. Definitely an improvement! The main “culprits” here are the removal of Has and the departure of everything-is-a-service approach.

We’re getting the same functionality in a much leaner package - what’s not to like!

Alternative approaches

I’ve been researching approaches to dependency injection for a long time, first using Java, then using Scala. This resulted in advocating constructor-based dependency injection (see the DI in Scala guide); mainly as an alternative to what Java offers via Spring and JEE, but also as an alternative to various Scala-based approaches to DI, which I think are unnecessarily complex.

As a disclaimer, I’m also involved in the development of macwire, a lightweight Scala DI library that uses macros to create the final object graph. Hence I’m not entirely impartial!

Together with Mateusz Borek, we are also working on an improved version of MacWire (see the issue, docs and PR with improvements), which will automatically properly wire the object graph, taking into account stateful dependencies, given as a cats.effect.Resource or zio.ZManaged. In some ways, this is similar to how ZLayer.make works, and hence can be seen as competition.

I’d love for ZIO to succeed, and I wouldn’t mind macwire being replaced by a superior solution, however as the ZIO environment already moved from the services-everywhere approach to a mixed constructor-based DI + ZLayer one, I’m not ruling out further evolution in ZIO 3.0 :).

Summing up

The ZIO Environment definitely takes a step in the right direction. It’s a pleasure to use, much simpler both to write and, as I suspect, also to read and understand the code later. One problem is the non-obvious distinction between services and business logic, causing hard decisions to be made as to which style: service with constructor-based DI or the environment to use. But this might be ironed out given enough examples and documentation.

The ergonomics are also improved, and the macro-generated error messages deserve special attention, giving quick and clear feedback as to what needs fixing in a particular program.

I’m also especially looking forward to seeing how the contextual capabilities of the ZIO environment will end up being used. I think there’s great potential to provide novel and easy-to-use libraries covering functionalities such as database access or security.