Testing Microservices - Contract Tests

Testing monolithic applications is relatively easy. We implement unit, integration, and probably end-to-end tests, and try to keep the testing pyramid as perfectly shaped as possible. But when it comes to testing microservices, things complicate a little. The main difference stands for communication which is essential in a microservice architecture. Luckily, for every challenge, we have a pattern and in this case, it’s contract testing. Let’s see how it helps to test microservices.

Monolith vs Microservices

One of the advantages (or disadvantages, depending on the context) of the modular monolith is the fact that it works in a single process. The modules can communicate using local function calls. This means that they don’t need to use a network to deliver business functionalities. Thus, the unit or integration tests that confirm the correctness of those modules, are implemented on its public API level. Changing the API of the module causes a compilation error (e.g. when we change a method's signature). Thus, it's hard to break the communication contract of modules in the monolith.

Interesting read: How to communicate Java microservices?

Things get complicated when it goes to microservices. They need to communicate over the network to provide business functionalities. We cannot rely on the compilator. The modules are independent and distributed. They are two separate applications probably with different testing and deployment pipelines.

How to ensure that services get along before we go to production?

Integration testing

Probably the first thing that comes to mind is integration testing. This is the quite obvious and partially correct answer to this question. But before we go further, let me clarify my understanding of integration tests. Different people understand it differently. I’ll refer to the integration test as the one which validates whether independently written parts of the system cooperate successfully and deliver the functionalities according to the requirements. This may be a test that validates

• exposed API (e.g. REST API)
• communication with another service
• producing and consuming messages
• integration with the database

The integration test should run in isolation. This means that we don’t need services up and running to execute the tests. Actually, we shouldn’t even need to be connected to the network to run them.

Integration tests are good candidates to test whether the service (or module) correctly operates with other parts of the system (database, other services, etc.). The main goal should be to validate whether the business functionality works as expected. Unfortunately, quite often I saw that they are used to test communication contracts between services. This approach has some disadvantages. Let’s discover them.

Assume we want to test communication between two services - order and payment. The second one exposes the REST API which allows initiating the payment. The order-service handles order creation. It calls the payment-service to create a payment for this order. To confirm this communication we have to implement at least two integration test sets:

• validation that API fulfills requirements. This validates whether, for a certain request, the service responds with the expected output. It includes validation of path, headers, and body structure. Such tests must be implemented in the payment-service. It should simulate calling the API by order-service (see Code 1).
• validation that payment-service’s API is correctly called by order-service. This checks whether the HTTP client correctly calls the payment-service. Such tests must be implemented in the order-service. They use the stubs of the payment-service (see Code 2).

In the payment-service, we tested the exposed API using the following integration test. It’s written using Spock and RestAssured.

def "should initiate payment"() {
   given:
   //skipped

   def request =
           """
           {
               "accountId": "$accountId",
               "orderDescription": "My first order",
               "quota": 128.5,
               "paymentType": "TRANSFER_ONLINE",
               "dueDate": "$dueDate" 
           }       
           """

   expect:
   given()
           .contentType(APPLICATION_JSON_VALUE)
           .accept(APPLICATION_JSON_VALUE)
           .and()
           .body(request)
           .when()
           .put("/payment/%s".formatted(paymentId))
           .then()
           .assertThat()
           .statusCode(200)
           .body("status", equalTo("FINISHED"))
           .body("paymentExternalId", equalTo("ext-%s".formatted(paymentId)))
}

Code 1. Integration test of payment-service API

This test confirms that API works as designed by the payment-service team. Test checks that the API accepts the defined request and produces the expected response. It should also execute important business logic (checking skipped in the test).

On the other hand, the client team of order-service wrote integration tests of order creation logic. This test also is written using Spock and uses WireMock to stub the payment-service.

def "should create order and initiate payment"() {
   given:
   //skipped

   def expectedRequest =
       """
       {
           "accountId": "$accountId",
           "quota": $quota,
           "paymentType": "TRANSFER_ONLINE",
           "dueDate": "$dueDate"                       
       }       
       """

def paymentServiceResponse =
       """
       {
           "paymentExternalId": "ext-$orderId",
           "status": "FINISHED"
       }       
       """

and:
wiremockServer.stubFor(put(urlMatching("/payment/([a-fA-F0-9-]*)"))
       .withRequestBody(equalToJson(expectedRequest))
       .willReturn(
               aResponse()
                       .withStatus(200)
                       .withHeader("Content-Type", "application/json")
                       .withHeader("Accept", "application/json")
                       .withBody(paymentServiceResponse)
       )
)

expect:
sut.createOrder(accountId, orderId, quota).paymentProcessed() == true

}

Code 2. Integration test of order creation process that stubs payment-service

This test uses stubs of payment-service to validate the logic of order creation. It makes some assumptions about how the called service operates (the stubs).

What’s wrong with those tests? Actually, they work correctly as soon as both service teams perfectly know how the other side works. They must know what the expected request and returned response look like. Ideally, if they don’t need to change the contract between them.

This causes the service’s operability to rely on humans’ communication and knowledge. Imagine the payment-service team member who noticed an inconsistent naming in the response. He decided to change the field status to paymentStatus. It’s easy to change the field name and adjust the integration test that it passes, so we can go to production. However, without adjustments in the client’s code, the communication will be broken. After deploying the producer to production, the communication will fail. The service will be unavailable, and the company will lose some money.

In order to communicate the planned breaking change in the API to the clients, the payment-service team has to know who they are. What's more, they have to assess whether this change impacts them or not. For that, they need to know how the clients use their API. It’s not possible to know it based on the integration tests. They only reflect some assumptions about the contract. They don’t ensure that this is how the clients use API in the production system.

Another disadvantage of this approach is doubling the work. In order to test the communication of two services, we have to implement two sets of tests. Usually, they are stored separately in both client’s and producer’s repositories. What's worse, they are not synchronized. If client teams change the contract, it’s not reflected in the producer’s code and vice versa.

Image 1. Integration tests of both sides are stored in different repositories and are not synchronized.

Last, the integration tests of API are usually quite boring and repeatable. In the when section it executes the tested API. Then it asserts whether the expected output was returned. This must be done field by field, for all the fields in the response.

How can we test it better? We will see this later in this article.

End-to-end testing

First, I'll explain what I mean by end-to-end tests. Those are the tests that validate whether the whole system works as expected. They run on real instances of the services without isolation. Usually, they are deployed on a so-called stage environment. They validate whether important business scenarios work as expected.

The end-to-end tests are resistant to the problem mentioned above. However, they also have some disadvantages. One of the biggest is that all microservices must be deployed and running to execute the tests. This makes it challenging to set up and maintain such an environment. It’s also quite an expensive approach, as it requires infrastructure and time to execute the tests. If we want to check a contract between two services, we can’t test it in isolation. Instead, we have to use infrastructure with deployed code. That’s awkward and against the microservice’s spirit of loosely coupled, independently deployable services.

Contract Tests to the rescue

Somewhere between integration and e2e tests the contract tests appear. This concept helps to confirm the correctness of communication between two services in isolation. It is based on the following assumption. If the consumer and producer are tested independently using the same contract, it means that they can properly communicate. This is a key idea behind contract testing. There is only one, a shared contract that defines how the services talk to each other. They may communicate in both synchronous and asynchronous ways. The tests of the consumer and producer are based on this contract. This is in contrast to integration testing. There we have two sets of independent tests without automated synchronization.

Physically, the contract is a document with communication described using a DSL (Domain Specific Language). Depending on the framework it may be written in various languages and notations. You can use Java, Groovy, JSON, YAML, and so on. Following is an example of a contract in Java (Pact framework).

builder
      .given("test PUT)
        .uponReceiving(“PUT REQUEST")
        .path(“/payment”)
    .headers(expectedHeaders)
        .method(“PUT”)
      .willRespondWith()
        .status(200)
        .headers(headers)
        .body(“{\”accepted\”: true}")

Producer and consumer?
While talking about contract tests those two terms are commonly used.
A producer (or sometimes provider) is a service that exposes API. In the case of message-driven communication, this is a service that publishes the messages.

A consumer is a service that consumes API exposed by the producer or consumes published messages.

In this article, I refer to the producer as the service that exposes API, and the consumer, as the one that uses it.

How does contract testing help?

Now let’s see how it fixes the problems of integration testing.

The first challenge concerned a situation where one side breaks the contract without informing the other one. It was possible in integration tests, however, it’s hard to do it with contract tests, because we have a single, shared definition of this contract. If the producer (payment-service team) changes for example the response, it must be reflected in the contract. Otherwise, its tests will fail. Since it’s changed in the contract, the consumer’s tests are also updated and probably fail, if the change was not backward compatible. This forces the teams to inform each other about the changes, based on the contract.

Because all the communication details are written down in the contract, the producer team (payment-service) exactly knows who and how is using its API. The contracts usually reside in the producer’s repository (or in a dedicated one). In case multiple clients use the same API, every integration has a different contract (contracts are defined per client). Thus, based on the contracts, the producer team can identify the callers easily. Imagine we want to remove or change the field in the response. Having the contracts agreed upon with all the clients, we can simply check whether the field is used or not. This gives us more flexibility in evolving the APIs.

Depending on the framework we don’t need to maintain any tests or stubs. Producer’s tests are generated from the contract. We don’t have to implement them on our own. They confirm that API works as defined in the contract. On the other side, the consumer tests use the stubs which are also generated from the same contract. This makes sure that both sides are synchronized in terms of communication and its tests. Also, it reduces the code we have to write by hand.

Finally, the contract is a kind of documentation of services’ coupling. It’s created and agreed upon by both parties of communication, thus it documents their agreements. Moreover, some frameworks allow to generate OpenAPI documentation from the contract. That creates a single point of truth about our API.

To sum up, only a contract needs to be maintained in this approach. The contract describes how the services communicate. Producer and consumer tests are run against the rule written down there. This gives confidence that both sides are implemented correctly and can operate.

Who owns the contract?

I explained that in contract testing there is a single point with contract definition. This may raise a question - who is responsible for defining and maintaining it? Depending on the relations between the involved services (and their teams) there are two ways of testing:

Consumer-Driven Contract Testing

If a producer is a downstream service, which should try to conform to the consumers’ requirements, the Consumer Driven strategy is used. In this approach, the consumers drive the contract of communication. They tell the producer what the API should look like. Such agreement should be of course reviewed, discussed, and confirmed by the producer team. Usually, it’s done by creating a Pull Request to the repository with contracts by the consumer team. Then it’s reviewed by the producer team and finally implemented.

This solution gives some advantages:

• since the contract is written down and approved, the consumer team doesn’t have to wait for the producer to implement the feature. Instead, they can use the contract to generate the stubs and test the integration of this service.
• the producer knows who and how is using its API

In this approach, for example, the following flow may be used to orchestrate the work of consumer and producer teams:

1. Let’s say the team responsible for order-service (consumer) implements a new feature. It requires integration with payment-service (producer). While doing TDD they realize that they need a stub of the API.
2. The teams discuss the contract and define it using DSL. It’s created in and committed to a payment-service repository.
3. The tests of the producer’s API are generated based on the contract. For now, they don’t pass due to missing implementation.
4. The contracts and tests are pushed to the repository. Also, the pull request may be created to be taken by the producer team.
5. The stubs of payment-service are generated together with the tests. They may be published to the artifact repository (e.g. Artifactory).
6. The consumer team may finish the tests using the stubs.

This shows that implementation doesn’t have to be finished by the producer in order to let the consumer progress with its implementation. Of course, the solution cannot be deployed to production until implemented by the producer team. However, this doesn’t block the consumer.

Finally, the producer team has to implement the API according to the contract. If all the tests pass, the service may be deployed to production. Both the consumer's HTTP client and the producer’s API were implemented and tested based on a single contract. This ensures that the services will correctly communicate in production.

However, there is one challenge here. This makes sense only if all the consumers accept the game rules. Having at least one consumer who doesn’t create a contract may break the idea. Usually, it’s easy to achieve it by forcing the consumer to create and agree on the contract in order to integrate with our service. This may be done technically, e.g. by not allowing the ingress to our service.

Producer-Driven Contract Testing

On the other hand, our service may be the one which is used by multiple parties which are probably not known. In this case, we don’t know exactly who wants to use our API and how. This approach is also used when we don’t want to allow the clients (consumers) to affect our API. In this case, the producer is an upstream service. It defines the contract of the API and informs consumers how to integrate with it. Such contracts may be used by consumers as documentation. They can also use it to test the integration with the producer, without deploying the code to the server.

Contract testing’s trap

Quite often I saw people overuse contract testing. I went this way too. By overusing I mean using contract tests in place of unit/integration testing. Contract testing should be used to test the contract of services that communicates synchronously or asynchronously. It shouldn’t validate the logic.

To give some examples, let’s consider the payment-service’s API. It returns a field status in the response. Depending on the scenario, this may have value FINISHED, IN_PROGRESS, or FAILED. However, the value of this field is an internal logic of this business process. It shouldn’t be tested by contract tests but probably by unit tests. One shouldn’t create a contract test scenario for every field value. There should be the only one which validates that this field was returned, with any of those values.

In contrast to this, there is a situation where for a specific request, the response is different. This may be for example a flag, which indicates that an extra field is added to the response. Or this may be an error scenario, where for an incorrect request, the producer returns 400 HTTP status. In this case, this should be tested by contract test, as this is a specific communication situation.

Going deeper, if 400 code is returned in 3 scenarios, but the response differs only with the message, only one contract test scenario should be created for that.

Summary

This article started by pointing out the challenges of testing communication in a microservice architecture. It also reviewed two common approaches to testing it - integration and e2e testing. Then, contract testing was introduced as a solution for the aforementioned problems.

Contract testing is a concept that allows testing communication (both synchronous and asynchronous) between the services in isolation. The main idea behind it is a contract. This is a place where communication rules are agreed upon and written down by the sides. The contract is a single point where the contract is defined. The parties don’t have to synchronize when the contract changes. Both producer’s and consumer’s tests are based on this contract. It ensures that if the tests pass, the services get along in production.

Next, I explained the difference between Consumer Driven and Producer Driven approaches and when to use them. Finally, I gave you a tip on how not to fall into the trap of overusing contract testing.

In the next articles, I will introduce the two most popular frameworks used for contract testing - Spring Cloud Contract and Pact. Stay tuned!