Combining Multiple Data Sources into One API with DreamFactory

by Spencer Nguyen • June 25, 2024
Table of contents

Integrating and calling other APIs is important in the development process. APIs allow different systems to communicate and share data, enabling developers to build more dynamic and feature-rich applications. Whether it's accessing third-party services, aggregating data from multiple sources, or automating workflows, the ability to effectively integrate and call other APIs is a key component of a successful software strategy.

DreamFactory is an API generation tool that facilitates this process by providing a unified platform for managing and calling external APIs. It handles the complexities of authentication, request formatting, and response parsing. 

DreamFactory_blog_CTA_163x200@2x-May-07-2024-08-15-34-3229-AM

High Level DreamFactory Overview

DreamFactory is like the Swiss Army knife of API management. It helps you generate REST APIs for a bunch of different data sources without a lot of hassle.

Here's a quick rundown:

  • REST API Generation: DreamFactory automatically creates REST APIs for your data. No need to write code from scratch. Just plug in your data source, and DreamFactory handles the rest.
  • Supported Data Sources: This platform can connect to almost anything - SQL databases like MySQL and PostgreSQL, NoSQL databases like MongoDB, cloud storage services like AWS S3, and even other APIs. If you've got data, DreamFactory can probably connect to it.
  • Architecture and Key Components: DreamFactory is built on a solid stack. It's an HTTP-based app using the Laravel framework with an Angular front end for the admin console. It supports multiple system databases (MySQL, SQL Server, PostgreSQL, SQLite) for storing configuration settings. The architecture is n-tier, so you can scale and secure the presentation, logic, and data layers separately. Typically, it runs in Docker containers managed by Kubernetes, often hosted in environments like Azure Government for added security.

Setting Up DreamFactory

Installation

  • Download DreamFactory from the official website.
  • Install Docker and Docker Compose if not already installed.
  • Run the provided Docker installation script to set up DreamFactory.

Initial Configuration

  • Access DreamFactory via your web browser at http://localhost:8080.
  • Create an admin account when prompted.
  • Configure system settings, such as email server details, from the "Admin" tab.
  • Add data sources like MySQL, PostgreSQL, or MongoDB under the "Services" tab.
  • Set up user roles and permissions in the "Roles" tab.

DreamFactory Interface Overview

  • Dashboard: View system status and recent activity.
  • Services: Manage and configure API services.
  • Users: Handle user accounts and authentication methods.
  • Roles: Define and assign roles and permissions.
  • Scripts: Add custom server-side scripts for extended API functionality.
  • API Docs: Explore and test APIs with automatically generated documentation.
  • Admin: Access system settings and logging configurations.
  • Logs: Monitor and troubleshoot with detailed logs of API requests and system activities.

Of course, this is a super high level explanation of setting up DreamFactory. If you’d like to get more granular, feel free to book a call with one of our engineers here!

Overview of API Integration

DreamFactory makes it simple to integrate and call other APIs, providing a unified interface to manage and interact with various external services. Here's how you can leverage DreamFactory to call other APIs effectively:

Creating a Service for External APIs

  1. Define a New Service
    • In the DreamFactory interface, navigate to the "Services" tab and click "Create."
    • Choose "HTTP Service" to define a new service that will call an external API.
  2. Configure the Service
    • Enter the necessary details, such as the service name, base URL of the external API, and authentication method (if required).
    • Save the configuration to make the service available.

Setting Up API Requests

  1. Endpoints and Parameters
    • Define the endpoints you need to call within the external API. This can be done by specifying paths and parameters directly in DreamFactory.
    • Customize headers, query parameters, and request bodies as needed to match the external API’s requirements.
  2. Authentication
    • Configure authentication methods such as API keys, OAuth, or Basic Auth within the service settings. DreamFactory supports various authentication mechanisms to securely connect to external APIs.

Handling Responses

  1. Transform and Process Data
    • Use DreamFactory’s scripting capabilities to transform API responses. For example, you can write server-side scripts in Node.js or Python to parse JSON responses, handle errors, and format data before it’s returned to your application.
  2. Error Handling
    • Set up robust error handling to manage issues such as network failures or invalid responses. DreamFactory allows you to define custom error messages and fallback mechanisms to ensure reliability.

Combining Multiple Data Sources

DreamFactory offers powerful capabilities for aggregating responses from multiple APIs into a single endpoint, providing a streamlined way to gather and present data from various sources. This functionality is particularly useful for applications that need to pull together information from different systems or services to deliver a cohesive response to the client.

How It Works

DreamFactory allows you to use server-side scripting to make multiple API calls and aggregate their responses. By leveraging scripting languages such as Node.js, PHP, or Python, you can fetch data from various APIs, process it, and then return a unified response. This approach simplifies the client-side logic, as the client only needs to make a single API call to DreamFactory to receive a comprehensive dataset.

Steps to Combine Multiple API Calls

Create a Script:

  • In the DreamFactory admin console, navigate to the Scripts tab and create a new script. Choose the appropriate scripting language for your environment (e.g., Node.js, PHP, Python).

Make API Calls:

  • Within the script, use HTTP request functions to call the different APIs. For example, in Node.js, you can use the axios or request module to make these calls.

Aggregate Responses:

  • Collect the responses from the various API calls. You can process and combine this data as needed. This might involve merging JSON objects, filtering data, or performing calculations.

Return Unified Response:

  • Format the aggregated data into a single response object. Ensure the response structure is clear and meets the requirements of your client application.

Assign Script to an Endpoint:


    • Attach the script to a new or existing API endpoint in DreamFactory. This endpoint will now serve the combined response from multiple APIs.

Example

Consider a scenario where you need to gather user information from multiple services. One API provides basic user details, another offers account information, and a third delivers user activity data. Here’s a simplified example using Node.js:

const axios = require('axios');

async function getUserData(userId) {
// Make API calls to different services
const userDetails = await axios.get(`https://api.example.com/users/${userId}`);
const accountDetails = await axios.get(`https://api.example.com/accounts/${userId}`);
const activityDetails = await axios.get(`https://api.example.com/activity/${userId}`);

// Aggregate responses
const aggregatedData = {
user: userDetails.data,
account: accountDetails.data,
activity: activityDetails.data,
};

return aggregatedData;
}

module.exports = async function(req, res) {
try {
const userId = req.params.userId;
const data = await getUserData(userId);
res.json(data);
} catch (error) {
res.status(500).send(error.toString());
}
};

In this example, the script fetches user details, account details, and activity data, then combines them into a single response object. The client application only needs to call this single DreamFactory endpoint to receive all the relevant information.

Benefits

  • Simplified Client Logic: Clients make fewer API calls, reducing complexity and improving performance.
  • Optimized Performance: Aggregating data server-side can be more efficient, reducing latency and improving response times.
  • Centralized Data Processing: Apply consistent business logic and data transformations in one place.

DreamFactory’s capability to combine multiple API calls into a single endpoint improves the efficiency and effectiveness of API management, making it easier to deliver comprehensive and cohesive data to client applications. This feature is particularly valuable for complex applications that need to integrate and present data from diverse sources seamlessly.

Conclusion

Calling other APIs with DreamFactory takes the complexity out of integrating various services, making it straightforward and efficient. With DreamFactory, you get a single platform that handles everything from authentication to data transformation, allowing you to focus on building great applications. Whether you’re connecting to third-party APIs or combining multiple data sources, DreamFactory simplifies the process, ensuring your integrations are secure, reliable, and easy to manage. Give it a try, and see how much smoother your API interactions can be!