Contract Testing For Dummies

Nowadays service-oriented applications are becoming very popular.

There is no "one" backend.

There are multiple services that all communicate with each other.

This is all nice and dandy but when you get into the details it gets a bit messy. One of the common fallacies is how do you test these services.

Well, you might say we can test each service individually.

Yes, that would work but there are lots of things that can go wrong when services communicate with each other.

But how do we test that?

Well, that's what we will talk about today.

By the end of this article you will be able to answer these questions:

• How do we usually test services?
• What problems does the "usual" method have?  
• What is contract testing and how is it better than the old approach?
• What are the common terminologies used in contract testing?
• What are the types of contract testing?
• When should we use contract testing?

Testing Services

The main method of testing multiple services is to simply have integration tests. Integration tests are simply tests that help us validate that multiple services are working together.

So for example, you can test an API call that uses two services and verify the response that you get back.

Another example is end-to-end integration testing, where you deploy all your services locally in a somewhat "production" environment and run a series of test cases on them.

Whilst this approach would simulate what the user would experience, they have several major drawbacks:

• They are slow – Because you have to boot up the whole system and run each test sequentially, each test may take several seconds to several minutes to complete, especially if you got a prerequisite setup before every test such as data preparation.
• They are hard to maintain – Because they require all services to be in the correct state before testing, so if the state of one service is incorrect then all tests may fail.
• They can be unreliable or flakey – As stated above all services should be in the correct state before testing them and that is usually a hard thing to maintain. So tests mostly fail due to configuration issues than breaking code changes.
• They are hard to debug – When a test fails, there is no proper stack trace that can show you at which part of the code the test fails because of the remote nature of the services.
• They do not scale well – The more you test your code like this, the more the tests get entangled, which means that the tests become slower and releases get clogged in automation pipelines.
• They find bugs too late – Because of the nature of the tests, we usually run them in our pipelines after the code gets committed. The bugs get found out by a separate testing team days afterwards. This delay in feedback is extremely costly to modern, agile delivery teams.

I probably missed a couple of problems but you get my point.

Smart engineers were fed up with these problems so they created a new testing method. It's called contract testing.

What is Contract Testing?

Contract testing is a testing methodology that ensures that two services are compatible and are able to communicate with each other.

The way it does that is it captures the interaction that is being exchanged between each service, storing them in a contract, which can then be used to verify that both parties adhere to it.

You might say isn't this API documentation like Swagger?

No, because it goes beyond schema validation. It requires both parties to come to a consensus on the allowed set of interactions and allowing for evolution over time.

The goal of this method is that it allows each service to be tested independently and the contracts are generated from the code itself, hence making it up to date with reality.

Contract Testing Workflow and Terminologies

Now that we know what contract testing is, let's see how it's being implemented.

But before we do all that, let's cover the basic terminology:

• Consumer – A service that initiates a request to another service. The kind of request doesn't matter it can be HTTP, RPC, message-broker, etc...
• Provider – A service that responds to the request.
• Mock Service Provider – This is used on the consumer side of things to mock the provider meaning that integration-like tests can be run without requiring the actual service provider to be available. This is also used to generate consumer contracts.
• Contract – Is the JSON serialised interactions (requests and responses) defined in the consumer tests.
• Contract Broker – Our contract files will be stored here and will allow both our providers and consumers to access them.

Depending on what tool you use for contract testing there might be more or fewer components.

In general, you should see these five components.

Anyways, now that we know our terms, let's go over the steps in the diagram above.

PS. The code examples are all in JavaScript using some imaginary library. You can implement your own contract testing framework or use the I recommend in the recommendations section.  

Consumer Side

Step 1:  The consumer unit tests its behaviour under a provider mock. This will be used to create our consumer contracts.

Here's an example of creating and testing against provider mocks:

const provider = new Provider({
  port: 8080,
  consumer: "ConsumerAPI",
  provider: "ProviderAPI"
});

// Start the mock service!
await provider.setup()

// Your expected response
expected response = {
	"status": "All Good!!!"
}

describe('Consumer Test Against Provider API', () => {
  describe('given there are x', () => {
    describe('when a call to the API is made', () => {
      before(() => {
        return provider.addInteraction({
          state: 'there is a need to request from provider',
          uponReceiving: 'a request for provider',
          withRequest: {
            path: '/something',
            method: 'GET',
          },
          willRespondWith: {
            body: response
            status: 200,
            headers: {
              'Content-Type': 'application/json; charset=utf-8',
            },
          },
        })
      })

      it('will receive the expected response from provider', () => {
        return expect(callProvider().response).to.be(response)
      })
    })
  })
})

Once we run this test, everything should work because we forced the desired response. But there is an extra hidden behaviour that the provider did for us, it generated a contract of the expected response to the contract broker.

This is the response for a successful 200 response, in real life you would have tests for each different response.

Step 2: Share the consumer contract with the providers.

If your consumer contract was submitted to the contract broker, you can simply tell the providers to access it via HTTP, or some sort of SDK.

Step 3: Test that the provider verifies the consumer's contract

Right now, that the provider has access to the consumer contract, we can test whether it is valid or not.

describe('Contract Verification', () => {
  const port = 1234
  const opts = {
    provider: providerName,
    providerBaseUrl: `http://localhost:${port}`,
    contractBrokerUrl: 'https://my-contracts.com/',
    contractBrokerUsername: 'dXfltyFMgNOFZAxr8io9wJ37iUpY42M',
    contractBrokerPassword: 'O5AIZWxelWbLvqMd8PkAVycBJh2Psyg1',
    publishVerificationResult: true,
    tags: ['prod'],
    providerVersion: '1.0.' + process.env.HOSTNAME,
  }

  before(async () => {
    server.listen(port, () => {
      console.log(`Provider service listening on http://localhost:${port}`)
    })
  })

  it('should validate the expectations of Consumer', () => {
    return new Verifier().verifyProvider(opts)
  })
})

In just three easy steps, we created tests that check whether our two services are compatible.

I also illustrated the three different kinds of contract testing.

• Consumer-Driven – We test from the consumer's point of view.
• Provider Driven – We test from the provider's point of view.
• Bidirectional – We test from both the consumer and provider point of view.

Benefits of Contract Testing

Contract tests generally have the opposite properties to multi-service integration tests.  

• Speed – Contract tests are fast because they don't require multiple systems to be running.
• Maintainability – They are easier to maintain because you don't need to understand the whole ecosystem.
• Debugability – They are easier to debug and fix because the problem is only in your single service. They also run locally, so the developers can find bugs immediately.
• Scalability – Because each service can be independently tested, build pipelines don't increase linearly / exponentially in time

They also have other unexpected benefits such as:

• You can develop the consumer before the provider.
• It helps you drive out the requirements from your provider.
• You get a good set of documentation for your consumer-provider interactions.
• The ability to see exactly which fields each consumer is interested in, allowing unused fields to be removed, and new fields to be added to the provider API without impacting a consumer.
• You immediately know which consumer is broken if a breaking change is made to a provider.

When Should I Contract Test?

Contract testing is one piece of the puzzle, you can't only have those. The same thing can be said about other types of testing. But nevertheless, it's an essential piece.

So comes the million-dollar question, at which state should I contract test?

As you see above there are some prerequisites needed for contract testing. Let's go over the things you need before you start implementing contract testing:

• Architecture – Contract testing was essentially made for distributed systems but can work with a monolithic system if you have one client and one backend server.
• Communication – You must have good communication between the different teams in the software lifecycle. So all your developers, QA's, and infra team must be in sync to make contract testing work.
• Testing Semantics – Contract testing is not functional testing. It only tests whether the request/response is valid or not in relation to the contract.
• Unit and Component Tests –  If you implement contract tests without having any unit or component tests, then your contract doesn't have much of an impact. It's better to focus on unit/component testing instead.

Recommend Resources

• Pact – This is by far the most popular contract testing framework out there.
• OpenAPI Specs – This can be used as your contract file.
• Protobuf – You can also use this as your contract file.
• Contact Testing Crash Course

Conclusion

Contract testing is not some fancy tool that should be used everywhere.

It's just a tool that has its use cases.

So before you jump right in, make sure that it fits your use case.

If you enjoy content like this, feel free to follow me on Twitter at @tamerlan_dev.

Thanks for reading.

References

• What is Contract Testing?
• Practical Contract Testing with Spring Cloud Contract by Oleksandr Romanov
• Pact Terminology
• Pact FAQ
• Introduction to Contract Testing