Composable Martech: Orchestration & Federation

Part 3 in our “unpack the stack” series on composable martech is all about the data – specifically, access to the data – the middle layer of the stack. The next set of capabilities we’re exploring is Orchestration and Federation. These two capabilities go well together because they are very similar and have some overlap, so let’s unpack ’em.

Orchestration and Federation in a Composable Architecture

At a high level, the “orchestration and federation” category represents the underlying access and routing to data across a variety of back-end martech products – from PIM, CMS, Order Management, DAM, Marketing Automation, internal and external proprietary databases, etc. While the prior topics of FEaaS and Experience Builders focus on the visual expression of content, data, and layout, orchestration and federation capabilities provide access (and intelligence!) to the actual content and data to hydrate those experiences. Let’s better understand the differences here.

Orchestration vs. Federation

The reality is these terms are often used interchangeably, so the definitions below are my take based on how they are often used in reality and… a bit of the dictionary:

Federation means bringing multiple sources of data/content together into a consolidated and predictable “place” – in reality the “place” may be a martech tool that holds a copy of all of the data/content, or simply an API facade that sits on top of the underlying systems’ APIs. More on this in a bit. The key point here is it’s a unification layer in the martech stack, a single entry point to get access to the back-end systems via a single API.
Orchestration is the same as Federation, however, is brings a bit more logic to the data party, providing some level of intelligence and control on exactly what data/content is provided for consumption. It’s like air traffic control for the flow of data from back-end systems.

Examples of Content Federation

Content Federation is a unification capability where you can combine multiple back-end sources together in a composable stack. A few examples include:

Hygraph Content Federation

Hygraph Remote Sources unify multiple back-end source system APIs directly into the Hygraph API so the consumer (e.g. a web app) only needs to access the Hygraph API and not all of the underlying other systems directly. You can read more about the content federation concept from Hygraph or see it live in a video! One thing to note is that Hygraph does not actually fetch and store external data inside Hygraph, instead the remote source’s API schema is added to the Hygraph API so a single call to the Hygraph API will make an “under the hood” call to the external API from Hygraph at query time.

Contentful Content Orchestration’s External References

Contentful External References (ironically) is a feature of Contentful “content orchestration” (see what I mean by these being used interchangeably?). External References allows system users to register external API sources that get merged into the Contentful GraphQL API so a consumer only needs to use one API. This is nearly identical in capability to Hygraph, however, one important thing to note is that Contentful allows for bi-directional editing of external data.  That means a CMS user can directly edit the external data from the Contentful CMS UI (assuming the API is setup to handle that). One key advantage of bi-directional editing is that a business user does not need to log into the other systems to make edits, they can stay inside the Contentful interface to do all of the editing.

Netlify Connect

Netlify Connect is another good example of federation following a similar model to Hygraph and Contentful. In Netlify Connect you can configure multiple “data layers” to back-end systems using pre-built integrations provided by Netlify, or to your own proprietary system using the Netlify SDK. A great use case for this custom approach is if you have a proprietary system that is difficult to get data out of and requires custom code.

The most notable difference with Netlify Connect is that it actually fetches and caches your external data into its own database and exposes snapshots of the historical data. This means you can use historical data revisions to query a specific set of data at a point in time, especially if you need to troubleshoot or rollback the state of an experience.

Optimizely Graph

Unlike the prior examples, Optimizely is a more traditional DXP that is leaning heavily into headless with the likes of Sitecore, Adobe, dotCMS, and others.

Optimizely Graph is the new GraphQL API to serve headless experiences built on Optimizely. One subtle (and maybe overlooked?) feature of Graph is the ability to register external data sources and synchronize them into Graph. Based on the documentation as it stands today, it appears this work is primary developer-driven and requires developers to write custom code to fetch, prepare, and submit the data to Graph. That said, the benefits mentioned previously still stand. This allows headless experiences to consume content from a single API while behind the scenes the synchronization process fetches and stores the external data into Graph.

Enterspeed

Enterspeed is a good example of a pureplay product that focuses on unification as the middle layer in a composable architecture. It allows you to ingest external data, transform that data, and deliver that data to various touchpoints, all via a high-speed edge network.

WunderGraph Cosmo

WunderGraph provides GraphQL microservice federation. It’s an open source and hosted product that helps you manage multiple back-end databases, APIs, authentication providers, etc. Additionally, its designed in a way for developers to declare the type of compositions of APIs they want using code, following a Git-based approach, instead of requiring UI-based setup and configuration.

Hasura

Hasura provides GraphQL federation similar to WunderGraph. It provides a single GraphQL API to consumers with the ability to connect several underlying systems such as REST APIs and databases (e.g. Postgres, SQL Server, Oracle, etc.).

Examples of Orchestration

Digital Experience Orchestration with Conscia

DXO is an emerging capability pioneered by Conscia.ai to help solve the problem of integrating many back-ends to many front-ends. DXO helps to orchestrate the complexity in the middle via a unified layer that all back-end services and systems of records communicate with, as well as a unified front-end API for experiences to consume:

A key tenet to this approach is to continue to leverage real-time APIs from the back-end systems, for example, a pure play headless CMS and a commerce engine. The DXO not only acts as a façade in front-end of these back-end system (similar to an API gateway), but it also provides other benefits:

Unifies data across back-end systems of record like you see with Federation
Provides enhanced business logic and rules by allowing business users to chain APIs together, avoiding static logic written into code by developers
Offers performance improvements by caching the real-time calls to the back-end APIs as well as pushing as much computation (e.g. business logic) to the edge closer to the end users

One key value proposition of Conscia’s DXO is a business user-friendly canvas to integrate multiple API call responses and create chains of calls. For example, the response of one API call might become an input to another API, which is often hard-coded logic written by developers:

Conscia’s DXO provides two key capabilities:

DX Engine as the federation layer to unify multiple back-end sources of data and content, as well as a rules engine and intelligent decisioning to orchestrate the right content and data for the individual experience
DX Graph as a centralized hub of all data, especially useful if you have legacy back-end systems or proprietary systems with hard to access data. The DX Graph can connect to modern APIs to visualize (via a graph!) all of your data, but crucially it becomes a centralized hub for proprietary data as well that may require scheduled sync jobs, batch file processing, and similar ways to integrate.

Similar patterns: API Gateways & BFFs

Is this like an API Gateway?
Yes and no. An API gateway provides a façade on top of multiple back-end services and APIs, however it mostly performs choreography as an event broker between the back-end and front-end (client). An orchestration system puts the brain in API gateway, being a centralized hub, and allows business users to control more of the logic.

Is this similar to the BFF (backend for frontend) design pattern?
Sort of. If the specific federation or orchestration tooling you are using allows you to control the shape of your API responses for specific consumers (e.g. frontend clients in a BFF), then you can accomplish a BFF. This is definitely a good use case for Conscia.

Why do orchestration and federation matter in composable architecture?

In a truly composable stack, we need to consider the fact that multiple systems in use means multiple sources of truth: CMS, another CMS, maybe another, PIM, DAM, OMS, the list goes on. It is absolutely possible to integrate directly with all of these systems directly from your head (the experience implementation you are providing such as a web site, mobile app, etc.). However, direct integrations like this tend to break down when you scale to multiple experiences since all of the back-end data integration logic is in a specific experience implementation/head (e.g. the website application code).

So, what’s the alternative to putting the integrations directly in your head?

Abstract it out and build a DIY integration layer: this sounds like a lot of work, but it certainly is possible. However, it may be hard to scale, add features, and maintain since it will turn into a bespoke product within your architecture.
Buy a federation/orchestration tool: why build it when there are products that already handle this? Focus on your specific business instead of building (and maintaining!) a bespoke product (like a CMS, and PIM, and OMS, etc.)

A dedicated federation/orchestration layer offers the following key benefits:

A single unified API for consumers (marketing site, web portal, native mobile app, data syndication to other systems/channels, etc.)
Promotes the concept that systems of record should truly own their data and avoids needing to write custom middleware to handle the orchestration and logic across many systems (e.g. head-based integration or a custom integration layer)
Encourages reuse of data and content: it offers data-as-a-service, so you can focus on how to activate it on your channels.
May provide contextual intelligence to control and personalize API responses to individual visitors in a dedicated data layer to power tailored omnichannel experiences. 

We have it all, so what’s next?

Seems like we have everything we need here, what else is there? Let’s (re)package up the first three capabilities into a larger topic – stay tuned for part 4 where we will talk about Digital Experience Composition (DXC).

Source: Read More