Do you still need testcontainers with Spring Boot 3.1?

Spring Boot 3.0, released in November 2022, brings us a lot of new exciting features, we’re only a few months after the release, yet the spring team doesn’t slow down, and they’ve just released Spring Boot 3.1. There are tons of new features available for us, and in this article, we will focus on a brand new module spring-boot-docker-compose which can automatically start and stop services defined in docker-compose for you in local development. Once I saw it, I started to think, “So can I now use good old docker-compose.yml to run my containers in tests, and I don’t need testcontainers anymore?”. Ok… let me explain what I mean.

Why do I need all this?

We all write code that uses some kind of external services like databases or queues to store some data. Using docker containers for external dependencies used by our application can simplify our development workflow a lot. We can run our application with the exact same versions of external dependencies as on production so we can be sure that our application is working properly. But starting and stopping them manually is not a good idea, especially when we need to run some integration tests to verify if our application is working properly with a selected database, for example.

Thankfully we have a great library for this called testcontainers which can run and stop any container we want and it manages the container lifecycle for us. So with testcontainers, we can run our integration tests with applications connected to our dependencies in the same versions as on production, and all happens automatically. But that’s not all. We can even start our application in a local environment, and testcontainers can run those containers for us. It's a great feeling when you can just run one command or a button in your IDE and you have your application up and running! But can we achieve the same with spring-boot-docker-compose? Can we use it in tests and in local development so we can replace our beloved testcontainers and use only spring boot features? Let’s find out.

What do we want to achieve?

We’re going to explain what we will try to achieve with spring-boot-docker-compose on an application that uses testcontainers.
Our application will use MongoDB, we will store two simple documents there, so we can execute two simple tests on separate collections, but we want to use only one container, and this is one of our goals: run a single instance of a specific container for all tests so our tests can be fast.

These are our MongoDB documents

@Document
public record Article(@Id ArticleId articleId, Author author, Content content) {
}
public record ArticleId(String id) {
}
public record Author(String name) {
}
public record Content(String text) {
}

@Document
public record User(@Id String id, String name) {
}

And we will use spring-data-mongodb module to create repositories for those documents.

public interface UsersRepository extends MongoRepository<User, String> {
}

In the end, we will expose simple HTTP endpoints to retrieve data from those repositories.

@RestController
@RequestMapping("/api/users")
public class UsersController {

    private final UsersRepository usersRepository;

    public UsersController(UsersRepository usersRepository) {
        this.usersRepository = usersRepository;
    }

    @GetMapping
    public Users getAllUsers() {
        var users = usersRepository.findAll();
        return new Users(users);
    }
}

public record Users(List<User> users) {
}

The repository and rest controller for articles looks almost the same, so I will omit them here. All code can be found on our GitHub repository.

Integration tests setup with testcontainers

In the next step, we will prepare tests and testcontainers configuration to run our tests connected to MongoDB running in docker.

Here is an example test case for users collection. We are saving some user to the database, and we’re checking if this user is returned from the API.

@Test
void shouldReturnSavedUser() {
    // given
    var user = new User(UUID.randomUUID().toString(), "username");
    usersRepository.save(user);

    // when
    Users users = testRestTemplate.getForObject("/api/users", Users.class);

    // then
    assertThat(users).isEqualTo(new Users(List.of(user)));
}

Again almost identical tests exist for articles.

Now we need to define our container with testcontainers.

@TestConfiguration(proxyBeanMethods = false)
public class TestContainersConfiguration {

    @Bean
    @ServiceConnection
    public MongoDBContainer mongoDBContainer() {
        return new MongoDBContainer("mongo:6.0.5");
    }
}

To share our spring context among all our integration tests, we usually have a base class with @SpringBootTest annotation that we extend in our classes with tests, so when we run all our tests only one spring context will be created for us, thus our tests will be fast. The same thing we need to do with our containers (to start them only once). And all we need to do is to import this TestContainersConfiguration into our BaseIntegrationTest

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@Import(TestContainersConfiguration.class)
@ActiveProfiles("test")
public class BaseIntegrationTest {

Psst… If you haven’t noticed, we’re using another great feature from Spring Boot 3.1 which is Service Connections. Now you don’t need to additionally define connection configuration to our container (like host and port). If you annotate your bean with @ServiceConnection, connection configuration will be extracted from the running container for you automatically.

And that’s all that you need to configure. Now we can run our tests, and testcontainers will spin up a fresh MongoDB container for us, and our application will use it in our tests.

Container mongo:6.0.5 is starting: 89bd7f517e0eacedae7af96c04cdcf8c8a5878cc8fcec3e57560d1c9b054914c
Container mongo:6.0.5 started in PT1.040507S

Migrate to Java 21. Partner with Java experts to modernize your codebase, improve developer experience and employee retention. Explore the offer >>

Using testcontainers at development time

Now as we have our tests running with automatically started MongoDB, we will configure our application on a local environment that way so it will use testcontainers to start MongoDB for us, and this is our next goal: use the same configuration of containers to run tests and application on local env. 

To do that, we need to create a new main application in our test sources so it will have access to our test classpath (testcontainers and container configuration).

public class TestApplication {

    public static void main(String[] args) {
        SpringApplication.from(Application::main)
                .with(TestContainersConfiguration.class)
                .run(args);
    }
}

Here we run SpringApplication, where all the configuration is loaded from our main Application and additionally, we’re adding our MongoDB testcontainers configuration and a simple class that will sample to our database at startup on a local environment.

In addition to that, we want to initialize our database with sample data on the local environment to indicate that local and test data doesn’t interfere with each other. To do that, we will use spring profiles and spring conditional auto-configuration to enable data initialization only in the local environment. In our case, there are two special profiles, local and test. And this is the content of our data initializer.

@Configuration
public class InitialDataForDevelopment {

    @Bean
    @ConditionalOnProperty(name = "data.initialize", havingValue = "true")
    public CommandLineRunner commandLineRunner(UsersRepository usersRepository) {
        return args -> usersRepository.save(new User(UUID.randomUUID().toString(), "name"));
    }
}

As you can see, we will initialize sample data only if we set data.initialize=true property, and by using spring profiles, we can set data.initialize=true only in the local profile, so our tests will run without any data. And here comes the next important goal: If we run our main application first, it will populate our db with some data, and while it’s running, we want to execute our test that way so our test won’t see this data, in short: we want to have isolation between running container in local env and tests. And this is achieved in testcontainers by default. As I’m running my application locally, it starts one container.

Creating container for image: mongo:6.0.5
Container mongo:6.0.5 is starting: cd242f72eb33b89542aeb8710c0a0b9fed12cd4fb0feffb5fa443e2b0199f3e7

And while my app is running with my initial data initialized, I run my tests, and we can see that testcontainers started a new MongoDB container for my tests.

Container mongo:6.0.5 is starting: 40aa3198e15416f98e029f43b0a7846d17161ad707dfd5ffde6246f6861302bd

And now, I have two running containers on my machine, and my tests don’t interfere with my local development environment.

 ~/Workspace/demo-spring-boot-testing> docker ps | grep mongo

40aa3198e154   mongo:6.0.5                 "docker-entrypoint.s…"   49 seconds ago   Up 48 seconds   0.0.0.0:58327->27017/tcp   quirky_hawking
cd242f72eb33   mongo:6.0.5                 "docker-entrypoint.s…"   29 minutes ago   Up 29 minutes   0.0.0.0:57885->27017/tcp   cool_jennings

How cool is that, right?!

There’s also one more important goal that wasn’t shown in the previous examples: run any container as you need. In our simple example, we run a simple MongoDB database but in more complex environments, you need to run many more services, for example, Kafka, Mockserver or Neo4j, and all this is supported by testcontainers.

To sum up, we have these goals:

1. Ability to run containers for local development
2. Ability to run containers for tests (only a single instance of a specific container should be run)
3. Isolation of containers between tests and local development
4. Ability to run any container we want

New spring-boot-docker-compose module in action

Let’s check if we can accomplish our first goal: The ability to run containers for local development.

Well, that’s the whole point of the module, so it should be pretty easy. The first step is to include spring-boot-docker-compose in our dependencies, we’re using maven, so we need to put this in our pom.xml file.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-docker-compose</artifactId>
</dependency>

Next, we need to describe our external dependencies in the docker-compose.yml file.

version: '3.9'
services:
  mongodb:
    restart: always
    image: mongo:6.0.5
    ports:
      - 27017:27017
    tmpfs:
      - /data/db

And that’s it. Now If we run our application, spring boot will detect that we have docker-compose.yml file, and it will start it for us. The first goal is achieved.

.s.b.d.c.l.DockerComposeLifecycleManager : Found Docker Compose file 'MY_HOME_DIRECTORY_PATH/demo-spring-boot-testing/docker-compose/docker-compose.yml'
o.s.boot.docker.compose.core.DockerCli   : Network docker-compose_default  Creating
o.s.boot.docker.compose.core.DockerCli   : Network docker-compose_default  Created
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Creating
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Created
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Starting
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Started
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Waiting
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Healthy

That was pretty easy, right? I have to admit that I had to adjust one setting in my project because I created a multi-module project, and I had to tell spring boot where to find my docker-compose.yml file but this is just a simple setting in our application.properties. This is because it will look for a configuration file in the current working directory, and as I have a multi-module project, my working directory changes when I run tests or main class from my IDE, that’s why I set it manually, and of course, running commands like ./mnvw test also works. But If you have a single module project you just need to put this docker-compose.yml file in the root directory of your project, and you’re good to go.

spring.docker.compose.file=./docker-compose/docker-compose.yml

Onto the next goal: The ability to run containers for tests (only a single instance of a specific container for all tests should be run)

At first, I thought that I didn't have to do anything and it would just work (we have our docker-compose.yml file already, so why not?) But it turns out that spring boot won’t start containers defined in docker-compose.yml in tests by default, it is just skipped, but there is a flag with which we can control this behavior, so let's change it.

spring.docker.compose.skip.in-tests=false

And with this little configuration, we can run our MongoDB container from docker compose in tests! Here are the logs from the test run indicating that the container started. Of course, all tests turned green, and our container started only once for all of our tests.

.s.b.d.c.l.DockerComposeLifecycleManager : Found Docker Compose file 'MY_HOME_DIRECTORY_PATH/Workspace/demo-spring-boot-testing/docker-compose/docker-compose.yml'
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Created
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Starting
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Started
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Waiting
o.s.boot.docker.compose.core.DockerCli   : Container docker-compose-mongodb-1  Healthy

We can mark our second goal as completed.

Our third goal was: Isolation of containers between tests and local development

And here I’ve found the first drawback. Unfortunately, running our containers in a local environment in isolation from containers running in tests is not so easy to configure. That’s because this new module tries to find out if it should start new containers with only a simple check of results from docker compose ps command. If there are any services already running, it won’t spin up new containers for us, and we will end up with tests connected to our container started for local development. Here and here, you can find out how it works under the hood. So any attempt to run these containers with different databases, docker networks or even docker compose profiles won’t work.

The only solution I’ve found is to create separate docker-compose.yml files for each environment (local and test in our case) and to put them in separate directories. It’s not ideal to have multiple docker-compose.yml files with the same configuration, but fortunately, we have Extensions in docker compose, so to not repeat ourselves, we can define one common docker-compose.yml with almost all configurations of our services, and in docker-compose-local.yml and docker-compose-test.yml we can reuse this common definition with a little customization, in our case of MongoDB we need to expose different ports so we can run two containers at the same time with different ports exposed. Here are my new docker-compose files.

./docker/docker-compose-common.yml

version: '3.9'
services:
  mongodb:
    restart: always
    image: mongo:6.0.5
    tmpfs:
      - /data/db

./docker/local/docker-compose-local.yml

version: '3.9'
services:
  mongodb-local:
    extends:
      file: ../docker-compose-common.yml
      service: mongodb
    ports:
      - 27017:27017

./docker/test/docker-compose-test.yml

version: '3.9'
services:
  mongodb-test:
    extends:
      file: ../docker-compose-common.yml
      service: mongodb
    ports:
      - 27018:27017

I know it’s not perfect, and we had to play a little bit with docker compose, but in the end, I think that we can mark our goal as achieved.

Last but not least: Ability to run any container we want

And here we have a second but a lot smaller problem. Currently, we’re not able to run any containers we want out of the box, we’re limited to several containers (which will probably cover most of the use cases), for example, we’re able to use MongoDB, Redis, PostgreSQL, Cassandra, Elasticsearch, without any extra code, but if you want to use something not yet included in the module, for example, Kafka or Neo4j, you will need to provide your own implementation of DockerComposeConnectionDetailsFactory so it will create ConnectionDetails for you, but this is another piece of code which you need to maintain so it’s not an ideal situation. But it’s probably just a matter of time when more DockerComposeConnectionDetailsFactoryies will be added to the spring-boot-docker-compose.

Conclusion

As we saw, this new module spring-boot-docker-compose is pretty powerful and easy to use. Maybe it’s not so easy to configure container isolation, and we have to introduce some redundancy of our 'docker-compose.yml' files, but it can do all the same as what we did with testcontainers. Out of the box, it also doesn’t support as many containers as we have in testcontainers (but this shouldn’t be a problem in most cases, and I’m sure more will be added in later releases). It looks like we have a new big player in town who will try to disrupt the status quo.

To sum up, sadly, I cannot give you the answer to whether you should now use spring-boot-docker-compose instead of testcontainers. You have to choose it by yourself. I think that we covered most of the use cases for which you would like to use docker containers in your development workflow in this article, but remember that this module is pretty new, so it doesn’t have all the features which you have in testcontainers, for example, parallel container startup or container reuse to name a few.

All the code can be found on our GitHub repository.

Reviewed by: Rafał Maciak and Dariusz Broda