Terence Bennett - May 16, 2017Dreamfactory has long touted our ability to securely generate REST APIs from a slew of different sources like SQL,NoSQL,SOAP, Files, AD/LDAP, SNS, SSO but Dreamfactory also allows you to connect existing REST services. In this short tutorial I’ll show you how to configure an HTTP service in DreamFactory using Open Data from data.gov, test it out and even infuse a little Swagger for documentation!
Open Data, particularly Open Government Data has several benefits. The first and most widely discussed benefit is transparency which can lead to civic engagement and foster innovation. However there are often less recognized economic benefits to using open data. For the purpose of this tutorial we will be using the U.S. Energy Information Administration API which provides information on the usage, imports and exports of renewable and nonrenewable energy such as coal, nuclear, electricity and petroleum. You will need to register for an API Key to follow along with this tutorial so if you haven’t done so yet, register and keep your API Key in a safe place for later use.
Now that we have some of the prerequisites taken care of let’s review the layout of an API call in DreamFactory. The layout of a typical DreamFactory API call is as follows:
<verb>http[s]://<server-name>/api/v<api-version>/[<service-name>]/[<resource-path>][?<param-name>=<param-value>]
This breaks down into the following:
- Verb – The HTTP verb matching the API call being used e.g. GET,POST,PATCH,PUT, DELETE.
- Server-Name – The name or IP given to the installed web server that is running the DreamFactory instance.
- Api-Version – This is the version number of the REST API, currently “2”. Performing a GET here returns an array of available services when permissions allow.
- Service-Name – The API name of the service you want to access. Performing a GET here on most of the native services will return an array of available resources.
- Resource-Path – The optional resource path of the service. This path may include multiple sections divided by ‘/’ as such to define the resource. Different resource paths or resource paths that have multiple sections when called upon with GET will often provide different or more granular representations of the requested data.
- Param-Name – URL parameters that are available, which vary by service type. ‘?’ denotes the start of a query or parameter. Additional parameters are separated using a ‘&’
- Param-Value – Values given for URL parameters. Note: Each value must be properly encoded
The above information is the foundation for understanding DreamFactory services and API calls. This information will take you far my friend, know it well.
Most API publishers will provide some accompanying documentation to help you get started with their API. Fortunately the folks over at the U.S. General Services Administration, Technology Transformation Service who developed data.gov did exactly this making getting started with the API that much easier! Looking over the U.S. Energy Information Administration API documentation I chose four endpoints which were Category, Updates, Series and Series Categories to be the focus of this tutorial.
Let’s get started with a basic HTTP service configuration. The minimum requirements to get started using the HTTP service are a service name and base URL. Let’s go ahead and create our service.
Navigate to the ‘Services’ tab in your DreamFactory instance and click on ‘Create Service’, scroll down and select ‘Remote Service’ and ‘HTTP Service’. A form will appear prompting you to give the service a Name, Label and Description. Let’s do this now:
<INSERT FIRST IMAGE>
In my example I’ve named my service ‘EIA’ , labeled it, marked it active and provided a brief description to make the service identifiable to other DreamFactory Users and Admins.
Next, click on the ‘Config’ tab in the DreamFactory UI and add the base URL. From the examples in the documentation provided I arrived at the conclusion the base URL for the API was https://api.eia.gov/
<INSERT SECOND IMAGE>
Now that you have the service configured, it’s time to test it out. Typically in DreamFactory when you add a service type like SQL,NoSQL or File Storage you are presented with an immediately testable service in the form of Swagger in the API Docs tab.This is not the case with HTTP services, DreamFactory is pretty capable with the other service types but unfortunately it’s not psychic so auto-generating Swagger from remote services is quite difficult. So it’s become a common misunderstanding and frequently asked question after a user creates a service to ask:
“Where’s my Swagger and how to use the API?”
<INSERT GIPHY>
The fun part is using the API, but how? With a REST client of course! Something like Postman will work but I like to use DreamFactorys’ built in testing tool which is aptly named ‘Test REST’. To use ‘Test REST’ navigate in your browser here:
https://<your-URL>/test_rest.html
Since I am running my DreamFactory locally I will access ‘Test REST’ at:
https://localhost/test_rest.html
<INSERT THIRD IMAGE>
In the field labeled URL you will see that it is prefixed /api/v2/. Following the layout of the typical DreamFactory API call I showed you earlier and the fact that ‘Test REST’ is located at the same URL as DreamFactory we can safely assume that this translates to:
https://localhost/api/v2/
Following this same convention let’s append the service name and ‘category’ resource to the URL:
https://localhost/api/v2/EIA/category
Nice! Now remember the API Key we registered for earlier? Let’s append this to the URL as a parameter name and value, following the API call layout from earlier this adjusts our URL to:
https://localhost/api/v2/EIA/category?api_key=<your-eia-api-key>
API Key is a user supplied value. For example:
https://localhost/api/v2/EIA/category?api_key=9af8c4be3a7199c6f506c3a319999f21
Perfect! Using the GET method and your DreamFactory Admin credentials in the ‘HTTP Auth username’ and ‘HTTP Auth password’ (basic auth) click the ‘Submit’ button. Scroll down and look for your data under ‘Output’. Let’s do this now:
<INSERT FOURTH ANIMATED IMAGE>
If the call on the ‘category’ resource was successful you should be returned a tidy little JSON response containing a list of category ids and names representative of the subtopic, these will be listed under ‘childcategories’ . Category ids can be used as query parameters in subsequent API calls. Let’s try to query the API using the category id 1292190 which is ‘Crude Oil Imports’. Before we do this let’s try to make our lives a little easier by not having to tote that API Key around with us every time we want to make an API call. Parameter names and values like our API Key can be hard coded into every API call you make through DreamFactory to the API, this also applies to headers and cURL options. This can be achieved via the service config tab by clicking on ‘Manage’ and clicking on the service you will be able to edit the HTTP service. Once you’re back in to the service configuration click the ‘Config’ tab and under parameters click the button that has the ‘+’ symbol to add the API Key as a parameter, you can add other additional parameters, headers and cURL options by clicking the ‘+’ button. In the param-name field put ‘api_key’ and in the param-value field put your API key the U.S. Energy Information Administration generously provided you, check the ‘Outbound’ box to pass the API Key on all API calls and select GET verb only as the API is read-only. Save the changes.
<INSERT FIFTH ANIMATED IMAGE>
Let’s test it out. Return to ‘Test REST’ and this time let’s query with the ‘category_id’ parameter-name and the 1292190 category id for parameter-value. Category id is a searchable parameter of the ‘Category’ resource, in ‘Test REST’ this adjusts the URL for the API endpoint to:
https://localhost/api/v2/EIA/category?category_id= 1292190
The API Key isn’t required as it is in the service configuration now. Let’s call the API endpoint from ‘Test REST’, each call will return a set of ‘childcategories’ which describe different pieces of data that can be subsequently queried.
<INSERT SIXTH ANIMATED IMAGE>
In the above example I made three subsequent queries to the API ‘childcategories’ of ‘Crude Oil Imports’ which were:
- GET https://localhost/api/v2/EIA/category?category_id=1292191
This call returned Crude Oil Imports ‘By Origin’. I chose category id 1292192 for the next API call which references the origin United Arab Emirates.
- GET https://localhost/api/v2/EIA/category?category_id=1292192
This API call returns additional options for Crude Oil Imports such as By State, By Refinery, By Port and By PADD. For the subsequent API call I will query ‘By State’ which is category id 1292271.
- GET https://localhost/api/v2/EIA/category?category_id=1292271
This API call is interesting because it is suggestive, it doesn’t return any ‘childcategories’ in this case but rather another subset ‘childseries’ that contains a list of series ids. The documentation suggests that series ids are a required query parameter when doing a GET on the ‘series’ resource. Let’s query for series id ‘PET_IMPORTS.CTY_AE-RS_CA-ALL.A’ which is from the description in the ‘name’ key, “Imports of all grades of crude oil from United Arab Emirates to refineries in California, Annual”. To do this we will do a GET on the API endpoint:
https://localhost/api/v2/EIA/series?series_id=PET_IMPORTS.CTY_AE-RS_CA-ALL.A
Let’s call the API now using GET:
<INSERT SEVENTH ANIMATED IMAGE>
“The Swagger specification is a powerful definition format to describe RESTful APIs. The Swagger specification creates a RESTful interface for easily developing and consuming an API by effectively mapping all the resources and operations associated with it. It’s easy-to-learn, language agnostic, and both human and machine readable.”
Adding some Swagger to the service will make the HTTP service easier to comprehend for people who might be interested in using the API, this could be a developer, researcher, data scientist or anyone that has an interest in the data coming from the API. Also, what better way to show off what we’ve learned today than infusing some Swagger? DreamFactory supports Swagger definitions written in both JSON and YAML formats. Looking over the specification you’ll quickly notice some recurring themes and similarities with regards to how the definitions are written and the layout of API calls we discussed previously. Let’s add some Swagger now, the Swagger definition below written in JSON covers a few of the endpoints we discussed throughout this blog in addition to some others we didn’t. The definition isn’t comprehensive but will serve as a good starting point for documenting the API in further detail. Copy the below JSON in to your favorite text editor and save with a name you’ll remember using the .json file extension, I saved mine as ‘eia.json’ for reference. Once this is saved navigate to the ‘Services’ tab in the DreamFactory UI, click ‘manage’ and then click on the EIA service to edit. Navigate to the ‘Service Definition’ tab and beneath ’Select Format’ choose ‘JSON’ . Next, click ‘Select Desktop File’ and navigate to the JSON file we saved containing our Swagger definition. Once the definition is visibly in the Editor scroll through it to ensure there are no syntactic errors (the editor will highlight any errors with a red ‘x’). If everything looks good, click save!
<INSERT CODE>
{
"swagger": "2.0",
"info": {
"version": "1.0",
"title": "U.S.Energy Information Administration Swagger",
"description": "U.S.Energy Information Administration API services",
"license": {
"name": "MIT",
"url": "https://github.com/gruntjs/grunt/blob/master/LICENSE-MIT"
}
},
"host": "api.eia.gov",
"basePath": "",
"schemes": [
"http"
],
"consumes": [
"application/json"
],
"produces": [
"application/json",
"application/xml"
],
"paths": {
"/category": {
"get": {
"description": "Category",
"operationId": "https://api.eia.gov/category",
"produces": [
"application/json",
"application/xml"
],
"parameters": [
{
"name": "api_key",
"in": "query",
"description": "The API Key, required to make API requests to U.S. EIA API services.",
"required": false,
"type": "string"
},
{
"name": "category_id",
"in": "query",
"description": "Optional. Category ID is a unique numeric identifier of a category. If left blank the root category is fetched from the API, additional queries can be made using category IDs returned from this API call.",
"required": false,
"type": "integer",
"format": "int32"
}
],
"responses": {
"200": {
"description": "success"
}
}
}
},
"/updates": {
"get": {
"description": "Updates",
"operationId": "https://api.eia.gov/updates",
"produces": [
"application/json",
"application/xml"
],
"parameters": [
{
"name": "api_key",
"in": "query",
"description": "The API Key, required to make API requests to U.S. EIA API services.",
"required": false,
"type": "string"
},
{
"name": "category_id",
"in": "query",
"description": "Optional. Category ID is a unique numeric identifier of a category. If left blank the root category is fetched from the API, additional queries can be made using category IDs returned from this API call.",
"required": false,
"type": "integer",
"format": "int32"
},
{
"name": "deep",
"in": "query",
"description": "Optional. If true, include the series in all descendent categories. If missing or false, only series directly in the start category will be returned.",
"required": false,
"type": "boolean"
},
{
"name": "firstrow",
"in": "query",
"description": "Optional. Integer specifying the zero-based index of the first row to return, providing a means to page through the updated series. Note that it is possible to page through the all of the API's series in this manner",
"required": false,
"type": "integer",
"format": "int32"
},
{
"name": "rows",
"in": "query",
"description": "Optional. Determines the maximum number of rows returned for each request, up to 10,000. Missing or invalid value results a default value of 50 as the maximum rows returned with each call.",
"required": false,
"type": "integer",
"format": "int32"
}
],
"responses": {
"200": {
"description": "success"
}
}
}
},
"/series": {
"get": {
"description": "Series",
"operationId": "https://api.eia.gov/series",
"produces": [
"application/json",
"application/xml"
],
"parameters": [
{
"name": "api_key",
"in": "query",
"description": "The API Key, required to make API requests to U.S. EIA API services.",
"required": false,
"type": "string"
},
{
"name": "series_id",
"in": "query",
"description": "Required. The series ID is a case-insensitive string consisting of letters, numbers, dashes and, periods that create a unique identifier for the EIA series.",
"required": true,
"type": "string"
},
{
"name": "start",
"in": "query",
"description": "Optional. When start is used, values will be returned beginning with the provided value. Values are accepted in the same format as dates in returned API data. Dates are formatted as yyyy, yyyyQq, yyyymm, yyyymmdd for annual, quarterly, monthly, and daily/weekly data when querying API. ",
"required": false,
"type": "integer",
"format": "int32"
},
{
"name": "end",
"in": "query",
"description": "Optional. When end is used, values will be returned ending with the provided value. Values are accepted in the same format as dates in returned API data. Dates are formatted as yyyy, yyyyQq, yyyymm, yyyymmdd for annual, quarterly, monthly, and daily/weekly data when querying API.",
"required": false,
"type": "integer",
"format": "int32"
}
],
"responses": {
"200": {
"description": "success"
}
}
}
},
"/series/categories": {
"get": {
"description": "Series Categories",
"operationId": "https://api.eia.gov/series/categories",
"produces": [
"application/json",
"application/xml"
],
"parameters": [
{
"name": "api_key",
"in": "query",
"description": "The API Key, required to make API requests to U.S. EIA API services.",
"required": false,
"type": "string"
},
{
"name": "series_id",
"in": "query",
"description": "Required. The series ID is a case-insensitive string consisting of letters, numbers, dashes and, periods that create a unique identifier for the EIA series.",
"required": true,
"type": "string"
}
],
"responses": {
"200": {
"description": "success"
}
}
}
}
}
}
Terence Bennett, CEO of DreamFactory, has a wealth of experience in government IT systems and Google Cloud. His impressive background includes being a former U.S. Navy Intelligence Officer and a former member of Google’s Red Team. Prior to becoming CEO, he served as COO at DreamFactory Software.