How Your API Strategy Is Fundamental to Any Data Mesh Strategy

The data mesh approach has gained popularity over the last couple of years as organizations look for reliable ways to break down data silos. At first, data lakes looked like a good way to improve data management and make information more discoverable. Unfortunately, data lakes — and data warehouses — don’t always conform to business needs. They’re often slow and even unresponsive to queries. Potentially even worse, they can still lead to data silos.

Transitioning to a data mesh strategy could help reduce these and other problems. Of course, you need a reliable way to implement a strategy that takes advantage of a data mesh’s potential benefits. That’s where an informed API strategy becomes critical.

Here's the 5 things you need to know about how your API strategy is fundamental to any Data mesh strategy:

• Zhamak Dehghani introduced the concept of data meshes in 2019, focusing on their principles and architecture to enhance understanding of their implementation and potential benefits.
• The four core principles of data mesh include domain-oriented decentralized data ownership and architecture, viewing data as a product, establishing self-serve data infrastructure as a platform, and adopting federated computational governance.
• In data meshes, independent data services manage small data segments within a decentralized architecture, improving access for authorized users, with APIs playing a crucial role in connecting and reusing these data sources.
• Data mesh encourages treating data as a product, leading to improved data accessibility and customization, with APIs enabling the combination of various data sources for tailored user needs.
• The approach of self-serve data infrastructure in data meshes empowers users to create specific data sets, enhances data governance, and facilitates access control, with APIs contributing significantly to these aspects.

Table of Contents:

The Four Data Mesh Principles

Zhamak Dehghani published her first blog post about data meshes in 2019. A year later, she went deeper into the concept by explaining four principles of data mesh and high-level logical architecture (among many other things. It’s worth reading How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh and Data Mesh Principles and Logical Architecture to improve your understanding of data mesh strategies, how to implement them, and what they can achieve.).

The four principles of data mesh focus on:

• Domain-oriented, decentralized data ownership and architecture
• Data as a product
• Self-serve data infrastructure as a platform
• Federated computational governance

Don’t worry if some of that sounds overly technical. Here’s a breakdown of what each data mesh principle addresses.

Domain-Oriented Decentralized Data Ownership and Architecture

A data mesh relies on independent data services responsible for managing small portions of an organization’s information. They don’t contribute to data silos, though, because they’re deployed across a decentralized architecture that improves access for all authorized users.

The Role APIs Play: You can conceptualize this similarly to the way you think about microservices. By weaving together decentralized sources, you gain access to the services (data) you need. Also, like microservices, you can use APIs to connect and reuse sources as needed.

Data as a Product

Organizations treat data differently when people think of it as a product. Once you conceptualize data as a product, you want to make it easier to locate and use. You also want to give users opportunities to customize the data they need so they can streamline workflows and create business process automations.

The Role APIs Play: APIs make it possible to combine data products tailored to the user’s unique needs. For example, someone tracking sales numbers could connect multiple data sources — such as a CRM and various e-commerce platforms — that contain sales information.

Self-Service Data Infrastructure as a Platform

Data mesh architecture should empower users to create data sets particular to their needs. Traditionally, IT teams have maintained control over access. While this approach makes sense when you want to control access and prevent unauthorized users from viewing private information, it creates bottlenecks that prevent teams from doing their jobs.

Self-service data infrastructure as a platform helps eliminate this and related issues so all teams can complete projects.

The Role APIs Play: A self-service data strategy doesn’t mean everyone has access to every piece of information. In fact, APIs can contribute to data governance while helping authorized positions find the information they need.

For example, you might restrict members of your machine learning team to the data they need to train products. You maintain full control over who can access your APIs, so you can also control who accesses data. You could even create specific domain teams and assign levels of access to each.

Federated Computational Governance

Federated governance means that each team owns the data it produces. As a data strategy, this means you distribute information throughout your organization. It also means the people who know most about the data get to make decisions about it. An IT team member might not know which data truly matters to a marketing campaign. The marketing team has more insight and experience that tells them how to manage the data, ensuring everyone has access to quality data that adds business value to projects and products.

The Role APIs Play: API connectors can provide access to data managed by small teams. They can also restrict what users can access. When team leaders create APIs as part of a data mesh strategy, they open as-needed access to the right people. Authorized users can feel free to experiment with data, and teams don’t need to worry that someone from outside of the department will alter the information.

Creating Business Value From API-Assisted Data Mesh Strategies

In some use cases, you might need to decide between data mesh or API management. However, you can often use both technologies to create business value from the information distributed throughout your IT ecosystem.

Connecting Diverse Data for Deeper Insights

Your business domains contain vast amounts of information. You can use APIs to support ETL data pipelines that ensure data quality and uniform formatting. This approach makes it easier for you to harness the power of big data, an essential aspect of business success in today’s economy.

When you have a data platform connected to all of your enterprise sources, you can rely on analytics to detect emerging trends, improve customer services, and create personalized marketing plans that drive higher revenues. At that point, you can make data-driven decisions that get results. You never have to make “informed decisions” again. You get guidance from huge amounts of data and analytics tools that can find patterns within the noise.

Diverse Data Integration

Your organization probably has all kinds of data stored in a broad range of locations, including AWS databases, private networks, and on-site servers. Even within those locations, you can probably find structured, unstructured, and semi-structured data.

An API-assisted data mesh strategy makes it possible for you to connect these sources, including SQL and NoSQL. With a little effort, your metadata, structured data, and unstructured data all become useful to your enterprise. Now, you can aggregate the information to find opportunities for improvement.

The Right Data for Every Team

Your company’s departments and teams have different data needs. If you employ engineering teams with data scientists, they probably want to access as much enterprise data as possible so they can build new products, strengthen existing features, and maintain strong security.

The data team within your sales department probably has more niche requirements. They might only want real-time sales numbers that help them determine whether they need to refine their strategies to meet revenue goals. For them, a business intelligence (BI) dashboard matters much more than the automated virtual assistants your customer service agents want.

An API-led data mesh strategy can serve these and other data needs. Decentralization means all data consumers can access information pertinent to their projects.

Adopting a Data Mesh Strategy

A reliable data mesh strategy could unlock numerous advantages for your business. How do you know that it’s the right approach for you, though?

Start With a Pilot Project

An enterprise-level data mesh won’t happen overnight. It could take months or years to accomplish. Start with a pilot project that can help you decide whether you want to commit to a data mesh strategy. Find a team working on a data-driven project that might benefit from data mesh principles. Ideally, someone within your organization already wants to explore domain-oriented ownership and data mesh. Let them try the approach and see how it goes. Then, you can use their successes and failures to inform future steps.

Create Domain Teams That Make Sense for Your Business

Each domain becomes responsible for overseeing a set of data. Your domains don’t have to fall along those traditionally defined by department, though. For example, you might have several product development teams working on different releases and features. It might make sense to establish each group within the department as a separate domain.

You can apply this logic to all of your departments and teams, including those focused on customer services, marketing, sales, and partnerships.

Focus on Quality Data

Whether you choose an API-led data mesh strategy or you decide to take a different path, success always begins with the quality of your data. You want each domain to control its own data. At the same time, you want domain teams to follow guidelines that help ensure quality.

Take some time to determine the data guidelines you want everyone to follow. As long as teams stay within those guardrails, you should have quality data that leads to successful outcomes.

Get (Nearly) Everyone Involved

The decentralized nature of data meshes requires company-wide participation. You can't rely on one team of data scientists to do everything. That’s contradictory to data mesh’s principles. Instead, you want (nearly) everyone involved.

Get your department heads on board first. They can then distribute plans to the teams they oversee. More likely than not, managers will take the lead long before employees get involved. Critically, you must consider whether the people within your organization understand the benefits of the new approach.

Luckily, even a little training can prepare managers and employees for changes they can expect as data mesh becomes the dominant policy within your company.

Start Creating and Managing APIs With DreamFactory

DreamFactory makes it easy for all of your teams to create and manage APIs that contribute to your data mesh strategy. They don’t need any experience building APIs, coding, or managing data.Start your free 14-day trial today to see how DreamFactory’s instant API creation could support your organization.

Related Reading: