When it comes to data meshes that have been constructed or are currently under development, I have not observed any instances where the four core data mesh principles have been utilized to their fullest extent. This ideal, which I term the ‘pure’ data mesh, demands strict adherence to its defined principles. However, in practice, all implementations deviate from this pure form, each with its own set of exceptions. Among these, some of the most frequent exceptions I have seen are:
- Data is kept in one enterprise data lake with each domain getting its own container/folder instead of each domain having its own data lake. This is popular with users of Microsoft Fabric and its OneLake. This helps with performance by not having to join data from multiple data lakes that can be physically located all over the world. It also helps with governance as all data is in one location
- Limiting the technologies each domain can use, which is almost always at least limited to one cloud provider. And usually within the cloud provider, limited to certain technologies. For example, each domain can only use Microsoft Fabric and not other technologies such as Azure Synapse. This reduces complexity and cost
- Central IT creates the data ingestion piece for each domain instead of each domain creating their own. This speeds up development as central IT will know the data ingestion piece well and can reuse a lot of the code
- Some services (for example, compute, MDM) are shared instead of each domain having their own. For example, a handful of Databricks clusters can be used by all the domains instead of each domain having their own Databricks cluster. This can greatly reduce costs
- IT builds the entire solution for each domain, including creating the analytical data. This will speed up development as IT will become experts in building these solutions and can repeat the process for each domain
- Using a product (like Microsoft Purview) for data cataloging all domain data instead of each domain using their own product and providing API calls to access their domain metadata
- Using a virtualization tool (like Microsoft Fabric with OneLake shortcuts) to access all domain data instead of each domain providing API calls to access data
- Having data products that simply contain data and metadata and not implementing the more complicated concept of using a data quantum as a data product (in which it holds not only the domain-oriented data and metadata but also the code that performs the necessary data transformations, the policies governing the data and the infrastructure to hold the data)
- Not using consumer-aligned domains or aggregate domains
- Using a shared model (there is only one domain that is shared by multiple business units) instead of a federated model (there are multiple domains that may be aligned to business units, products, or data producers/consumers)
It’s important to recognize that many of these exceptions can diminish or completely negate a key advantage of a data mesh: its ability to facilitate organizational and technical scaling. This scaling is crucial in preventing bottlenecks caused by people and infrastructure limitations.
This leads to the question: At what point do you have so many exceptions it should not be called a data mesh?
We can call that point the Minimum Viable Mesh (MVM). How many exceptions would be allowed for a solution to be considered a MVM? Can we all even agree on the four data mesh principles? If the four principles aren’t universally accepted in total, we can’t even try to answer that question. Do we need an industry committee to come up with a revised definition of a data mesh and a MVM? How do we avoid feedback based solely on self-interest?
Unfortunately, I don’t have the answers. If you are looking to build a data mesh, I suggest learning all you can about a data mesh and other architectures to see what would work best for your particular use case. My upcoming book might help with that. The eBook version will be available in early February. It’s got a pretty cover now 🙂
The post Common Data Mesh exceptions first appeared on James Serra's Blog.