Generic APIs – a Designed Gap
During the design phase of integration projects, sometimes, the topic of a “Generic API” design comes up usually driven by the need for an extensible API design that can keep up with the changing business requirements.
This technique is typically introduced by novice technical people as a “silver bullet” design that can presumably minimize the integration efforts. In some cases, it is even stated as a disguised technical requirement among the business requirements with the same rationale.
First, let’s start with what a Generic API usually means, and then we will explore the key design trade-offs and challenges that are often overlooked by using this approach.
Generic API – deliberate information gap
This is an API endpoint where its description deliberately omits detailed information about the structure and semantic of the exchanged messages and operations in favor of being able to support multiple current and future use cases under that same API endpoint.
The generic API design-time description would merely contain information about the endpoint’s location, and a top-level wrapper of the message structure which allows different message contents and operations to be injected at runtime.
The following are a couple of examples of generic message payloads:
It is worth noting that it is one thing to leverage an extension point for specific message node(s) in a well-defined message structure for extensibility purposes, but quite another to treat the entire message payload in an abstract manner as in the generic API approach.
As mentioned, extensibility is usually the perceived benefit of this approach. However, challenges arise from using generic APIs which can become a hinderance in the long term, especially in complex integration solutions.
Complexity Shift
The contract definition (schema) in a generic API is purposely too simplified, however, the essential business complexity, in the form of the business domain models and its operations, has to be reflected somewhere – you guessed it – it has to be implemented in the logic of the integrated systems.
This means that the complexity is not eliminated in a generic API, it is merely shifted from the design-time API contract definition to the API provider- and the consumers-sides respective systems to deal with multiple integration use cases at runtime according to the invoked API operations and message structures.
Increased Coupling
Normally, coupling creates dependencies between the integrated systems which would eventually make it harder for these systems to evolve independently.
The API provider and the consumers need to somehow align on the message structures being exchanged and on the set of defined operations and expected behaviors.
The lack of a guiding contract in the generic API approach would increase the sematic and syntactic coupling between the API provider and the consumers. This type of coupling creeps between the integrated systems as internal details inadvertently seeps between them during implementation which increases the coupling between them.
In contrast, a well-defined, expressive API contract definition will create an explicit boundary by defining the syntax and the semantics details to be exchanged between the integrated systems.
Notice that while the provider and consumers each would be coupled to the API contract itself, however, this would be a relatively weaker form of coupling as it is controlled by the consciously defined boundary.
API Adoption
We create an API to be consumed, but how easy is it for the consumers to adopt our API?
Clear Expectations
An expressive contract would eliminate any ambiguity or assumptions by the API stakeholders – the provider and consumers – regarding the behavior-related data fields in the exchanged messages.
It will ensure a common, and hopefully intuitive, understanding of the syntax and semantics of the business and technical fields that are shared between them.
Following a contract-first approach will help the API stakeholders to align on the key, shared knowledge in advance and to iron out any uncertainties while it is still cheap to do so, that is, before actual implementation has taken place.
Developer Experience
Developer experience on the consumer side is an aspect that is often overlooked in API design.
IDEs and integration tools typically have time-saving development capabilities for consuming APIs that rely on the expressiveness of the defined API contract.
These development tools would generate the necessary boilerplate code and/or configuration artifacts for developers which will significantly facilitate the consumers’ API onboarding process.
public static class EntityExtensions
{
public static CustomerDto setCustomerInfo(this Customer entity)
{
return new CustomerDto
{
Id = entity.Id,
Name = entity.Name,
Address= entity.Address,
};
}
}
In contrast, developers consuming a generic API would have to manually construct the message structures to inflate the request payloads for the target API operation, in addition to blindly parse and validate the response payloads at runtime.
Needless to say, this can be time consuming and error-prone especially when handling large or complex message structures and relying on an offline, and possibly, stale integration document.
Traceability and Control
Generic APIs don’t capture much about the API message content or the supported operations, so API stakeholders haphazardly share the necessary integration knowledge in an offline manner, using emails, meetings, calls, and integration documents.
This means that the API stakeholders won’t be able keep track of the changes being made to the message content nor map those changes to the target API operations or to specific endpoint versions.
In addition, it won’t be possible to achieve fine-grained control using API platforms at the provider-side; capabilities such as API usage reporting and API governance which rely on granular message- and operation-levels would not be easily applied, if at all.
Final thoughts
Whenever the word “generic” is mentioned in design workshops, it should sound the alarm bell. It simply means that we are about to defer some of the design decisions. In the API world, this could fly under the radar in simple and limited integration solutions; however, it will significantly hinder the complex ones.
In my experience, the generic API topic is usually driven by tight delivery deadlines, or by someone looking for a shortcut or a silver bullet solution to address a challenge without going to the necessary depth in assessing the solution or in addressing the real challenges.
Generic APIs give a false sense of controlling change while disregarding the aforementioned key trade-off aspects.
In practice, expressive APIs will initially entail additional effort in terms of design and development, but it will go a long way as it forms the guardrails that will mitigate coupling and will promote API adoption and governance. Accompanied by a versioning strategy, the API will also be able to evolve with the business requirements in a controlled manner.