There are cases where ownership of the data consumed by a fleet of microservices is distributed across several systems of record. Derived data is then scattered across multiple storage solutions, each designed to solve a specific problem using a subset of it. It may be mirrored and synchronized into a specific format (consider a graph database), or enriched with features that exist only in another store (consider a relational database with custom metadata). These read models can derive from relational or document databases, resources exposed through APIs, or a mixture of them.
The question in this case is how to achieve eventual data consistency across all the data stores keeping in mind that source-of-truth ownership is distributed across multiple data stores and third-party APIs, each with its own set of shortcomings. Moreover let's assume that consolidating them into one data store is not an option - in this specific case a strategy within a set of constraints is required in order to achieve an adequate level of consistency which can guarantee the correct resolution of end-user specific use-cases.
First it becomes clear that the criteria for each data store's consistency need to be properly defined - how fresh must the data be? does access require bidirectional synchronization—reads and writes—or is it just a read model within a specific type of data store to optimally solve a problem e.g. moving the relations between users into a graph database in order to easily make connections between them and generate insights that can then be easily consumed through a micro-service by various clients (mobile, web apps, etc.)
Read-only consistency
The simplest case is a read-only mode with little or no metadata. It is also important to know whether the metadata is encapsulated or leaks into other microservices’ business logic. If we're talking about the former then the rules for data freshness are dictated by its consumers.
Bidirectional consistency
The next one is when we have to deal with bidirectional consistency i.e. a client updates something in our data store and we need to sync that back with all the relevant systems of record. Now, this becomes quite problematic, because we have a dependency tree with side-effects leaking on each leaf. Existing criteria dictated by the consumers will thus require syncing the data to lowest acceptable interval, otherwise we will break the consistency contract we have with our clients. This becomes quite complex with metadata that is shared between multiple micro-services.
Shared metadata and queues
What about metadata? This is now a full-fledged cross-cutting concern that becomes dissonant with itself as it needs to be synced in multiple places and via different protocols (think about having similar metadata in a graph database and a relational database). Optimally, metadata that cuts across multiple microservices should reside in one system of record but let's say this is not an option.
This quickly transforms into an aching architectural issue - the solution resides not in a shared data store but within a shared mechanism of keeping track of these changes. One solution could emerge in the form of queues with ack/retry capability backed by persistence (something à la RabbitMQ, Kafka, etc.). It is now self-evident that this adds another complication: how do we deal with persistent events in the queues that are now inconsistent due to how much time they resided (a simple example would be a set of unacked events that after some time "T" become invalid as the record/object they have a reference to doesn't exist anymore) as a result: this will require another mechanism to drop those events from the queues or add another high level abstraction leak as the clients will have to deal with the issues of out-of-date data.
Eventual consistency in a complex system with distributed systems of record with an added complication of cross cutting metadata within a fleet of micro-services is quite an interesting problem to solve - in its most basic form we deal with a graph of (inter)dependencies that should govern the orchestration of both the micro-services and the syncing of the various data stores.
Finally, the solution would emerge from the graph of (inter)dependencies in the form of an attached mechanism which abstracts the distributed ownership behind one coherent view that stays within the confines of its consumers, thus delivering the expected business value on all fronts.