API Contract Testing Is Not Enough

Teams often tell me that their tests ensure that their components adhere to their "API contracts". They are referring to their REST, gRPC or other technical spec for their microservice or Web service.

Is that enough? Or is integration testing still needed?

First of all, testing against contracts is essential; but it is not sufficient.

• Well defined contracts don’t define the meaning of contract elements – only their data type and range.
• Contracts don’t define temporal behavior – e.g., what a component should do if a message is not received in a certain period of time, or if duplicates are received, or if the rate of requests is too high to sustain.
• Services are stateless, but they modify state indirectly by changing databases, sending messages that sit in durable message histories such as Kafka, and they often interact with external systems that maintain state. Thus, there are many kinds of system-wide state changes that are not represented by a service's contract, but that might affect how a service responds or how a service's response is interpreted.

I have also sometimes heard teams say,  “If you don’t use our contract correctly, it’s not our fault”. That might be true, but it is not a good answer for an enterprise: if someone is using your contract wrong, the organization needs to know. The testing pipeline must test that all contracts are being used correctly by other enterprise apps – one cannot afford to break other apps in production, even if they are using your contract wrong.

I have sometimes made a tabulation of the root causes of production incidents organizations that claim to be "DevOps". I usually discover that about half of the production issues tend to be integration issues. This is true even in organizations that claim to do fairly thorough testing at a unit and component level. This tells me that more is needed.