Blog

Combining Multiple Data Sources into One API with DreamFactory

Written by Spencer Nguyen | August 20, 2024

Whether you're pulling data from different databases, cloud storage, or external APIs, the challenge is to bring it all together. That's where DreamFactory comes in. It's a powerful platform that generates REST APIs for a wide variety of data sources, making integration a breeze.

In this blog post, we'll be discussing how you can use DreamFactory to combine data from multiple sources into one cohesive API. We'll cover setting up DreamFactory, connecting different data sources, creating unified API endpoints, transforming data on the fly, and ensuring security and performance. By the end, you'll see just how easy it is to integrate and manage your data with DreamFactory.

Data often resides in multiple locations—ranging from SQL and NoSQL databases to cloud storage and third-party APIs. The ability to combine these disparate data sources into a single, unified API is crucial for creating cohesive applications.

Combining Multiple Data Sources: Reasons and Benefits

Data often resides in multiple locations—anywhere from SQL and NoSQL databases to cloud storage and third-party APIs. The ability to combine these disparate data sources into a single, unified API is crucial for creating cohesive applications.

Reasons to Combine Multiple Data Sources

 
  • Centralized Data Access: When data is scattered across different systems, accessing it becomes cumbersome and inefficient. By combining data sources, you centralize access, which will let applications retrieve all necessary information through a single API call.
  • Data Insights: Merging data from various sources allows for more comprehensive analysis. For example, combining customer data from a CRM with transaction data from a financial system can provide insights into customer behavior and business trends.
  • Better Application Performance: By consolidating data retrieval into a single API, you reduce the number of network requests and database queries your application needs to make. This results in faster response times and a more responsive user experience.
  • Easier Integration: Many applications require integrating data from third-party services or external APIs. By combining these sources into one API, you simplify the integration process, which will in turn simplify the complexity of your codebase.

Benefits of Combining Multiple Data Sources

 
  • Data Management: With a unified API, managing data across multiple sources becomes much more straightforward. You can apply consistent access controls, data validation, and transformation rules.
  • Scales With Your Business: As your application grows, the ability to easily add new data sources or modify existing ones without significant changes to your API architecture is a major advantage. DreamFactory's flexible architecture supports this scalability, allowing you to adapt to evolving data needs.
  • Decreased Redundancy: When data is combined into a single API, you minimize redundancy in data retrieval processes. This not only conserves resources but also simplifies debugging and maintenance by reducing the number of points of failure.
  • Improved Security: By centralizing data access through a single API, you can enforce consistent security policies across all data sources. This reduces the risk of security gaps and ensures that sensitive data is protected consistently, regardless of its origin.

High Level DreamFactory Overview

DreamFactory is the Swiss Army knife of API management. It helps you generate REST APIs for a bunch of different data sources without the hassle of manual API development.

Here's a quick rundown:

> REST API Generation: DreamFactory automatically creates REST APIs for your data. No need to write code from scratch. Just plug in your data source, and DreamFactory handles the rest.

> Supported Data Sources: This platform can connect to almost anything - SQL databases like MySQL and PostgreSQL, NoSQL databases like MongoDB, cloud storage services like AWS S3, and even other APIs. If you've got data, DreamFactory can probably connect to it.

> Architecture and Key Components: DreamFactory is built on a solid stack. It's an HTTP-based app using the Laravel framework with an Angular front end for the admin console. It supports multiple system databases (MySQL, SQL Server, PostgreSQL, SQLite) for storing configuration settings. The architecture is n-tier, so you can scale and secure the presentation, logic, and data layers separately. Typically, it runs in Docker containers managed by Kubernetes, often hosted in environments like Azure Government for added security.

Combining Multiple Data Sources

 

 

SQL Databases: How to Connect MySQL and PostgreSQL Databases

 
  1. MySQL:
    • Go to the "API Generation" tab and select the database tab.
    • Click on "the “+" and select "MySQL" as the service type.
    • Fill in the necessary details like hostname (e.g., mysql.example.com), database name, username, and password.
    • Click "Save" and test the connection to ensure it's working correctly.
  2. PostgreSQL:
    • Follow similar steps as for MySQL.
    • Select "PostgreSQL" as the service type.
    • Enter the required connection details.
    • Save and test the connection.

NoSQL Databases: Setting Up Connections to MongoDB

 
  1. Go to the "API Generation" tab and select the database tab. 
  2. Press the “+” button to start generating the API.
  3. Select "MongoDB" from the service type list.
  4. Provide the connection string and any necessary authentication details.
  5. Save the configuration and test the connection to verify it.

Cloud Storage: Connecting to AWS S3 and Other Cloud Storage Services

 
  1. In the "API Generation” tab, select the database dropdown. 
  2. Select "AWS S3" as the service type.
  3. Enter your AWS access key, secret key, and the region where your bucket is located.
  4. Specify the bucket name and any other configuration settings.
  5. Save and test the connection to make sure it works.
  6. For other cloud storage services (e.g., Azure Blob Storage), follow similar steps with the appropriate service type and credentials.

External APIs: Setting Up and Consuming Data from External HTTP-Based APIs

 

 
  1. Create a new service in the "API Generation" tab and choose "HTTP Service." It should be under the “Network” dropdown. 
  2. Enter the base URL of the external API (e.g., https://api.example.com).
  3. Configure any required headers, authentication tokens, or query parameters.
  4. Save the service and test it to ensure it can successfully connect to and retrieve data from the external API.

Creating Unified API Endpoints

 

API Design: Best Practices for Designing APIs that Combine Multiple Data Sources

 
  • Consistency: Ensure consistent naming conventions and response structures across endpoints.
  • Documentation: Provide clear documentation for each endpoint, including parameters and expected responses.
  • Error Handling: Implement robust error handling to manage failures from any of the integrated data sources.
  • Security: Use appropriate authentication and authorization mechanisms to protect data.

Endpoint Configuration: Using DreamFactory’s Interface to Create and Manage Endpoints

 
  1. In the DreamFactory admin console, go to the "API Docs" tab.
  2. Select the service you want to create an endpoint for.
  3. Configure the endpoint by specifying the HTTP method (GET, POST, etc.), URL path, and any parameters.
  4. Use the built-in interface to define how data from multiple sources should be merged.
  5. Save the configuration and test the endpoint using the API docs or an external tool like Postman.

Sample Endpoints

 

Merging User Data from Multiple Databases

 
  • Create an endpoint that fetches user data from both a MySQL and a PostgreSQL database.
  • Combine the results into a single JSON response that includes information from both sources.

Combining Financial Data from Different Sources

 
  • Set up an endpoint that aggregates financial data from an internal SQL database and an external API.
  • Present a unified response with combined financial metrics and insights.

Integrating Third-Party API Data with Internal Data

 
  • Create an endpoint that retrieves data from an internal MongoDB collection and an external HTTP API.
  • Merge the data to provide a comprehensive response, such as combining user profiles with social media activity.

Data Transformation and Manipulation

 

Virtual Fields: Creating and Using Virtual Fields for Derived Data

 

 
  1. Navigate to the "Schema" tab in the DreamFactory Admin menu.
  2. Select the table where you want to add a virtual field.
  3. Click on the "Add Field" button and choose "Virtual" as the field type.
  4. Define the logic for the virtual field using SQL expressions or custom scripts.
  5. Save the configuration and verify that the virtual field appears in the API responses.

SQL Functions: Applying SQL Functions to Manipulate Data

 
  1. In the "Schema" tab, select the table and field you want to manipulate.
  2. Add a database function in the "DB Function Use" section.
    • For example, to format a date field: DATE_FORMAT(record_date, "%Y-%m-%d").
  3. Save the changes and check the API response to see the transformed data.

Custom Scripting: Using Custom Scripts (NodeJS, PHP, Python) to Handle Complex Transformations

 
  1. Go to the "Scripts" tab in the DreamFactory admin console.
  2. Create a new script and choose the language (NodeJS, PHP, Python).
  3. Write your custom script to perform the desired data transformation.
    • Example (NodeJS): A script to calculate and add a new field to the response.
      javascript

      event.response.content.forEach(record => {

record.newField = record.existingField * 2;

});

  1. Save the script and attach it to the appropriate event (e.g., before returning the response).
  2. Test the endpoint to ensure the script runs correctly and the data is transformed as expected.

These sections should provide a comprehensive guide on connecting multiple data sources, creating unified API endpoints, and transforming data using DreamFactory.

Advanced Features and Deployment Flexibility

While the core of combining multiple data sources into a single API has been well-covered, it's important to highlight some of the additional powerful features that make DreamFactory an ideal choice for building robust API ecosystems.

Documented REST APIs with Pre and Post-Processing Scripts

One of the standout features of DreamFactory is its ability to not only connect multiple data sources but also provide automatically documented REST APIs for each connection. This documentation is crucial for developers who need to understand the structure and usage of the APIs without diving deep into the code.

Beyond just generating these APIs, DreamFactory allows you to add pre- and post-processing scripts to modify the incoming and outgoing data. Whether you're working with NodeJS, PHP, or Python, you can inject custom logic at different stages of the API call. This is especially useful for:

  • Data Validation: Ensuring that incoming data meets specific criteria before it's processed.
  • Data Transformation: Modifying or enriching data after retrieval but before it's sent to the client.
  • Security Enhancements: Adding additional layers of security checks or token verifications.

These scripting capabilities provide a level of flexibility that’s often necessary when dealing with complex integrations or custom business logic.

Lightweight and Versatile Deployment Options

Another significant advantage of DreamFactory is its lightweight architecture. Despite its powerful capabilities, it can be deployed in resource-constrained environments, such as a Raspberry Pi, making it an excellent choice for edge computing scenarios or IoT applications. This lightweight nature doesn’t come at the expense of performance or features, allowing you to maintain a robust API ecosystem even in a minimalistic setup.

DreamFactory’s versatility in deployment options is worth emphasizing. It can be hosted in the cloud, which is ideal for scalability and accessibility, or run in Docker containers, providing portability and ease of management. Docker-based deployment is particularly useful in environments where consistent and repeatable builds are required, ensuring that your API ecosystem can be scaled, managed, and secured with minimal overhead.

Flexible Hosting: Cloud or On-Premise

DreamFactory’s flexibility extends to its hosting options. Whether your organization prefers a cloud-based solution for ease of access and scalability or an on-premise setup for enhanced control and security, DreamFactory accommodates both. This is particularly beneficial for industries with stringent regulatory requirements, such as healthcare or finance, where data sovereignty and security are paramount.

Conclusion

DreamFactory takes all your data and serves it up through one unified API. It's like having a universal remote for your TV, cable box, and sound system – everything you need, controlled from one device.

By bringing all your data together, DreamFactory helps you create smarter, faster, and more powerful apps.

The best part? DreamFactory handles all the complex behind-the-scenes work, so you can focus on the fun stuff – using your data to create amazing things and grow your business. 

Want to give it a try? Spin up a a trial version in your environment by talking to our engineers here!
View full post