Kevin McGahey - May 11, 2016Developers often find themselves creating same services and resources across all their DreamFactory instances multiple times. For example, in a simple three-stage software development lifecycle (dev, test, production), developers typically need to copy a set of apps, roles, users, services, lookups and other resources across all or some of their environments. This becomes a tedious and counter-productive task as the number of resources and instances goes up.
To address this challenge and to make it even easier to provision a DreamFactory instance with pre-configured services and resources, DreamFactory introduces the “Packages” feature in version 2.1.2. Packages allow you to export system-wide resources, including your storage files (apps), into a single package file. This package file is a zip file archive of the exported resources. You can take this package file and import it into an existing instance or provision a new instance with it. You can find the new package API in the API Docs under ‘system’ service.
Package File
Before we get into the details of how to export/import packages across multiple instances, let’s go over the structure and content of a package file. As mentioned earlier, the package file itself is a zip file of all the exported resources organized in related folders. Resources are exported in JSON format. Storage folders are exported in the zip file and stored inside folders that correspond to their path and storage service name.
The package always contains a ‘package.json’ file. This file is called the manifest file, which details out all the contents of the package. Here is how package contents look like after extracting the zip file.
You can see that all exported resources are in a *.json file under their corresponding API path. For example, the API for roles is …system/role, therefore the exported roles are stored in the role.json file under the system folder. Similarly, _shema.json is stored inside a folder named with its service name – mysql. We also exported the ‘work’ and ‘my_images’ directories from the ’s3’ storage service. The contents of these directories are zipped into a single file and stored inside folders named with their corresponding path (work, my_images) and service name (s3).
Now let’s take a look at the content of the manifest ‘package.json’ file.
{
"version": "0.1", // Package version
"df_version": "2.1.1", // DF instance version
"secured": false, // Secured flag (coming up)
"description": "", // Optional package description
"created_date": "2016-04-12 14:18:08", // Date this package was created
"service": {
"system": {
"role": [“role1”,“role2”],
"service": ["DB","mailgun","script","s3","mysql","math-py"],
"app": ["add_angular2","my-test-app"],
"user": ["[email protected]","[email protected]"],
"admin": ["[email protected]"],
"custom": ["adminPreferences"],
"cors": [6,7],
"email_template": [“test_email_template","test_template"],
"event": [
"mydb._table.contact.{id}.patch.pre_process",
"user.register.post.post_process"
],
"lookup": ["host","user"]
},
"DB": {
"_schema": [“contact_group","contact_info"]
},
"mydb": {
"_schema": [“contact","todo"]
},
"mysql": {
"_schema": [“contact","todo"]
},
"files": [
"applications/",
"projects/",
"testfiles.zip"
],
"s3": [
"my_images/",
"work/"
]
}
}
As you can see, the manifest file simply lays out the resources and their location in this package. The service – resource structure of the manifest file closely adheres to their corresponding API path.
Export
Exporting a manifest only
Before exporting a package it’s good to know which resources you can export. To see all the exportable resources, simply request the package manifest with the following API call.
curl GET https://df.com/api/v2/system/package
You can use this manifest to export your desired resources. To get just the system resources use parameter ‘system_only=true’
curl GET https://df.com/api/v2/system/package?system_only=true
To download this manifest as a file use parameter ‘as_file=true’
curl GET https://df.com/api/v2/system/package?as_file=true
The manifest file shows all key resources in your instance. Periodically downloading and keeping an archive of the package manifest file will allow you to maintain an audit trail of your instance.
Exporting a package
Exporting a package requires a POST call to the system/package API with a payload similar to the package manifest. For example, the following call will export a package containing roles with name r1, r2, and r3.
curl POST https://df.com/api/v2/system/package -d ‘{“service”:{“system”:{“role”:[“r1”,”r2”,”r3”]}}}’You can also export multiple resources, storage files, database schemas all in one package using a manifest like below.
{
“service": {
"system": {
"role": [“role1”,“role2”],
"service": [
“DB",
“mailgun",
“script",
“s3",
“mysql",
“math-py"
],
"app": [“add_angular2",“my-test-app"],
"user": [“[email protected]",“[email protected]"],
"admin": ["[email protected]"],
"custom": ["adminPreferences"],
"cors": [6,7],
"email_template": [“test_email_template","test_template"],
"event": [
"mydb._table.contact.{id}.patch.pre_process",
"user.register.post.post_process"
],
"lookup": ["host","user"]
},
"DB": {
"_schema": [“contact_group","contact_info"]
},
"mydb": {
"_schema": [“contact","todo"]
},
"mysql": {
"_schema": [“contact","todo"]
},
"files": [
"applications/",
"projects/",
"testfiles.zip"
],
"s3": [
"my_images/",
"work/"
]
}
}
There are few other ways you can export package resources using the export manifest. Here are some examples:
Exporting package resources using id:
{
“service”:{
“system”:{
“role”:[1,2,3] // Exporting roles with id 1,2,3
…
}
}
}
Exporting package resources using filter:
You can use filters to export resources that are filterable. Example: exporting all services that begin with a prefix ‘test_’.
{
“service”:{
“system”:{
“service”:{
“filter”:”name like ’test_%’”
}
…
}
}
}
Exporting package resources with related data:
To export related data , specify their relation name in the export manifest. Example:
{
“service”:{
“system”:{
“role”:{
“ids”:[1,2,3], // Exporting roles with id 1,2,3
“related”:[
“app_by_role_id”,
“role_service_access_by_role_id”
]
}
…
}
}
}
Exporting package resources with related data using filter:
{
“service”:{
“system”:{
…
“role”:{
“filter”:”name like ’test_%’”,
“related”:[
“app_by_role_id”,
“role_service_access_by_role_id”
]
}
…
}
}
}
Exporting storage resources in package:
{
“service”:{
“files”:[ // Name of your storage service
“css/”, // Storage folders to export
“assets/“,
“images/buttons/“,
“app.js”, // Storage files to export
“index.html”
]
}
}
Exporting schemas from a Database service:
{
“service”:{
“db”:{ // Name of your database service
“_schema”:[ // Export _schema resource
“table1”, // Name of your tables
“table2”,
“table3”
]
}
}
}
Exported package storage
By default, all exported packages are stored using the local file service – ‘files’ inside a folder named ‘__EXPORTS’. The response from the package export call shows the full URL of the exported package and a flag indicates whether the package URL is accessible or not. You can download a package using this URL.
{
"success": true,
"path": “https://df.com/files/__EXPORTS/df_2016-04-13_03:23:34.zip”,
"is_public": true
}
You can also use a different storage service and folder for your exported package by specifying the service and folder in your export manifest.
{
“storage”:{
“name”:”s3”, // Name of the storage service
“folder”:”my-exports”, // Folder to store your package in
},
“service”:{
“system”:{
“role”:[1,2,3] // Exporting roles with id 1,2,3
…
}
}
}
If you just provide the storage service name and do not provide a folder, then your package will be stored in a default folder called ‘__EXPORTS’. The default name used for the package file follows the scheme hostname_y-m-d_H:s:i.zip. You can use a different name for your package if desired.
{
“storage”:{
“name”:”s3”, // Name of the storage service
“folder”:”my-exports”, // Folder to store your package in
“filename”:”my-package.zip”
},
“service”:{
“system”:{
“role”:[1,2,3] // Exporting roles with id 1,2,3
…
}
}
}
Exporting a package securely
Normally, when you export system services in your package, all your service configs, which includes sensitive information such as your database host, username, password etc. are all in plain text. You can export your package securely using a password which will encrypt all your service config information.
When you import this package into another instance you will need to provide this password. If you export your users in a secure package, you can export your users with their passwords (encrypted). This way all your users will continue to have the same password when imported in a different instance. To export a package securely, set the ‘secured’ flag to true and provide a ‘password’ on the export manifest.
{
“secured”:true, // Set this true for secure package
“password”:”secret”, // Must provide a password for secure package
“service”:{
“system”:{
“service”:[“db”,”s3”] // Service configs are encrypted
}
}
}
Import
Exporting storage resources in package:
Make a multipart/form-data POST request to system/package API.
curl --form [email protected] https://df.com/api/v2/system/package
Importing package file from a URL:
You can import a package from a URL using the ‘import_url’ parameter.
curl POST https://df.com/api/v2/system/package?import_url=https://me.com/packages/my-package.zip
Importing package from command line:
You can import a package from the command line using the Artisan console command – dreamfactory:import-pkg.
php artisan dreamfactory:import-pkg path
The path to your package file can be a path to a single file or a folder or a URL. When a folder is provided it will import all packages in that folder. To import a secured package, just use the ‘—password’ option to provide the password
php artisan dreamfactory:import-pkg path —password=secret
NOTE: When importing a service or resource that already exists in the target instance, the import process will skip it and leave the existing record unchanged. The import process logs all skipped/troubled items in the system log. It also returns the log messages as the response of the import call.
Import/Export using the Admin app
The admin app provides an easy way to export and import your packages using an UI. You can find this UI under the ‘Packages’ tab inside the admin app.
From the ‘Type’ drop down menu, select your resource type then select your resource from the ‘Name’ menu. Click on ‘Add to Package’ to add it to your ‘Selected Package Content’ list. Select as many resources as you need to export. After that, select a storage service where you want to store your package using the ‘Export to’ menu and provide a folder in the text field next to it. Click on ‘Export’ button to export your package.
I hope this blog post explains how easy it is to deploy your apps from development all the way to production. Let us know what you think in the comments or head on over to the community forum if you have questions! Also check out this screencast to see a quick example.
Kevin McGahey is an accomplished solutions engineer and product lead with expertise in API generation, microservices, and legacy system modernization, as demonstrated by his successful track record of facilitating the modernization of legacy databases for numerous public sector organizations.