NoSQL, No Problem! – Operation Specifics

 
LeeHicksAs a continuation to my initial blog on NoSQL support on the DreamFactory Services Platform (DSP), I would like to give you a little more information on how to use the NoSQL service operations. Designed to be flexible and powerful, yet still adhere to simple REST principles, there are several options for CRUD operations available in the DSP REST API for NoSQL services. Once you have configured your NoSQL service, you can click the document icon to the right of the service listing on the admin console and get the Swagger UI interface for documentation and testing as seen below. DreamFactory MongoDB Rest API Notes about this blog:
  1. I am using MongoDB as our example service but all supported NoSQL types behave similarly, with some exceptions.
  2. Also, I am using JSON format everywhere here, but XML is also supported.
  3. In the cURL request, not all headers are included in request, i.e. authentication, app name, etc. You can use the Swagger UI to perform many of these same calls much easier.
  4. See my notes in the earlier blog about how the HTTP verbs function.

Table Administration

As you see from the list above, all table administration operations utilize the root of the service for the URL. All tables are identified by a ‘name’ field in the operations. All posted or returned data is transmitted as a single object representing a table containing a ‘name’ field at a minimum, or an array of table objects returned as a ‘table’ field value, except where noted. Retrieving Tables Every DSP service, when queried at its root with no extra parameters, returns an array of ‘resources’ available by that service. In the SQL and NoSQL case, these resources are table names. To get a list of currently available tables, just send a GET request to the service’s root, which, if you are using one of our hosted DSPs, looks like this.
curl https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo
The response looks like…
{
  "resource": [
    {
      "name": "test"
    },
    {
      "name": "zipcodes"
    }
  ]
}
To add vendor-specific details to the output, just add the query parameter ‘include_properties’ set to true. In this case, we get MongoDB-specific details about each collection.
curl https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo?include_properties=true
The response looks like…
{
    "table": [
        {
            "name": "test",
            "indexes": [
                {
                    "v": 1,
                    "key": {
                        "_id": 1
                    },
                    "ns": "test.test",
                    "name": "_id_"
                }
            ]
        },
        {
            "name": "zipcodes",
            "indexes": []
        }
    ]
}
Additionally, there is a ‘names’ query parameter accepting a comma-delimited list of table names that allows the client to selectively retrieve information about specific tables
curl https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo?include_properties=true&names=test,zipcodes
Creating Tables Use the POST operation to create one table or multiple tables at once. At a minimum, a value for the ‘name’ field is required to create a table. Additional vendor-specific fields may also be sent. Some vendors may require additional fields. Again, a single object or an array of objects is permitted in the posted data. The output will reflect the input, i.e. an array posted will result in an array received. To create a new table, use the following…
curl -X POST https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo \
     -H 'Content-Type: application/json' \
     -d '{ "table": [ { "name": "test", … } ] }'
The response looks like…
{
    "table": [
        {
            "name": "test",
            "indexes": [
                {
                    "v": 1,
                    "key": {
                        "_id": 1
                    },
                    "ns": "test.test",
                    "name": "_id_"
                }
            ]
        }
    ]
}
Updating Tables Most of the NoSQL vendors accept little to no updates on the meta-data part of the tables. Some do however, and those would be accessed as follows…
curl -X PATCH https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo \
     -H 'Content-Type: application/json' \
     -d '{ "table": [ { "name": "test", “indexes”: {…} } ] }'
Deleting Tables Obviously this should be approached with caution. You can use the ‘names’ query parameter in a delete request to delete existing tables.
curl -X DELETE https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo?names=test
An additional option for delete would be to post the same data format as the other operations.
curl -X DELETE https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo \
     -H 'Content-Type: application/json' \
     -d '{ "table": [ { "name": "test" } ] }'

Record Administration

The DSP REST API provides the following ways for your app to retrieve, create, update and delete data from the tables. The table name is sent as part of the URL, while the other options are sent as query parameters or as part of the posted data. Many of the query parameters can also be sent via the posted data. One special query parameter available on all requests is the ‘fields‘ parameter. This dictates which fields will be returned for the affected records of the operations. It accepts a comma-delimited string of field names when passed as a query parameter, or an array of field names when passed in posted data. For GET request, this parameter defaults to returning all fields, i.e. a ‘*’ value, while all other request types return only the record identifying fields by default. This saves the client from having to do an additional round-trip call to get things like updated or auto-filled field values, or to get a list of records changed when updating by a filter. Again, where applicable, a single object or an array of objects is permitted in the posted data. The output will reflect the input, i.e. an array posted will result in an array received. Creating Records To create a single record …
curl -X POST https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes \
     -d '{ "_id": "95008", "state": "CA", "pop": 39968 }'
To create multiple records…
curl -X POST https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes \
     -d '{ “record” [ { "_id": "95008", "state": "CA" }, { "_id": "30022", "state": "GA"} ] }'
If the DB vendor does not automatically create the identifying fields (primary key), it must be included in the POST request. The applicable identifying fields are always returned for successfully created records on a create request. Retrieving Records There are many ways in which an app can retrieve data through our API. Depending on your app you may want to use one or all of these. Note that the first three also make use of the optional query parameter ‘id_field’ which allows the client to state which field is used as the identifying field for the records in that table. Here is a quick look at the retrieval options.
  • By a single record identifier – In this case, the identifying field (think primary key) value is passed as part of the URL after the table name. This will return a single record or a 404 – Not Found error.
curl https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes/95008
Returns this…
{
    "_id": "95008",
    "city": "CAMPBELL",
    "state": "CA",
    "pop": 39968
}
  • By a list of record identifiers – This case uses the ‘ids’ query parameter sent as a comma-delimited string of id values, or ‘ids’ field sent as comma-delimited string or an array of id values in posted data. If a large number or ids, or the id values are long or have special characters in them, it would be better to pass them as posted data.
curl https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes/?ids=95008,30022
Returns this…
{
    "record": [
        {
            "_id": "95008",
            …
        },
        {
            "_id": "30022",
            …
        }
    ]
}
  • By record – This case allows you to post a single record or an array of records that at a minimum include the identifying field(s) and values. This is useful in retrieving records with more than one key, or for updating a stash of records that you have old or partial data for already. (Note: The below cURL call works for *nix and mac, for Windows, use -X POST -H “X-HTTP-Method: GET” instead of -X GET.)
curl -X GET https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes \
     -d '{ “record” [ { "_id": "95008", … }, { "_id": "30022", … } ] }'
  • By filter – This is the most powerful option for data retrieval. The NoSQL service allows the client to use a simple SQL-like filter string passed as a query parameter (url-encoded), or in some vendor cases, native filters in various formats (like MongoDB’s json format, more on this in later vendor-specific blogs).
For our simple SQL-like filter string, comparison operators supported for every vendor are =, !=, >, >=, <, <=.; Or given as their SQL short form ‘ eq ‘, ‘ ne ‘ (or ‘ <> ‘), ‘ gt ‘, ‘ ge ‘, ‘ lt ‘, ‘ le ‘. Depending on the db vendor, others such as contains, like, and begins-with may also be supported. Spaces surrounding the operator is required. String values on the right side of the comparison must be within single or double quotes. Depending on the db vendor, logical comparisons (AND, OR, NOT) are also supported in order to build out more complicated queries. Other ‘filter helper’ query parameters are also available.
  • limit – Defaults to return all, accepts an integer greater than 0 to limit the number of returned records.
  • order – Accepts a field name followed by space and then ASC or DESC to order the returned records.
  • offset – Accepts an integer greater than 0 to skip that many records in the response. This is useful for setting up paging through records, but may not be supported by all vendors.
For example, if we want to find the first 3 records in the zipcodes table information that have a population of over 20,000 people, returning only the city and state. If you are familiar with SQL, this would look like “SELECT _id,city,state from zipcodes WHERE pop > 20000;”. Here is what it looks like in cURL.
curl https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes?filter=pop+%3E+20000&limit=3&fields=city%2Cstate
And this is what gets returned…
{
    "record": [
        {
            "_id": "28659",
            "city": "NORTH WILKESBORO",
            "state": "NC"
        },
        {
            "_id": "31201",
            "city": "HUBER",
            "state": "GA"
        },
        {
            "_id": "71291",
            "city": "WEST MONROE",
            "state": "LA"
        }
    ]
}
Updating and Merging Into Records The same options available for retrieving data also apply to updating records. As mentioned in the earlier blog, the PUT HTTP verb is used when the whole record is to be replaced with the posted data. The PATCH HTTP verb is used when the client only wants to send the changing fields to the server. The same array of records or a single record format with changes is supported for updating records. To replace the whole record at id of 95008, send the request like…
curl -X PUT https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes \
     -d '{ "_id": "95008", "city": "Campbell", "state": "CA", "pop": 40102 }'
If you only want to merge changes into that record without having to resend everything else, then send the request like…
curl -X PATCH https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes \
     -d '{ "_id": "95008", "pop": 40102 }'
If you only want to update or merge data for one record, and it can be identified by a single key field, then you could also add the id to the end of the URL, and pass only the fields that require change.
curl -X PATCH https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes/29684
     -d '{ "pop": 40102 }'
Two other methods of merging data are by id list or filter (using “ids” or “filter” url parameters mentioned above for retrieving records). In these cases, if not natively supported, the server will query the table for the filtering results, merge in the record changes and push the changes back in an update request. This is an easy way of updating multiple records with the same field-value changes. Deleting Records Deleting supports the same options as updating. The only difference is, when using the id, ids or filter options, no posted data is required. Obvious, right? Using the ‘fields’ query parameter, the client can request the full or partial records before they are deleted from the system. What Next? Stay tuned for more specifics on each NoSQL vendor, next up more on MongoDB.

NoSQL, No Problem!

 
LeeHicksYou may have already seen that DreamFactory Service Platform (DSP) supports SQL databases with its REST API, but did you know that we also support most of the popular NoSQL databases? Currently, DSPs support Amazon Web Services’ SimpleDB and DynamoDB, Windows Azure’s Tables, MongoDB, MongoHQ, and CouchDB.

While the technology behind, and the interface in front of, each of these services vary, some dramatically, they each support storing data and lots of it. Some already have a REST or somewhat  REST-like HTTP interface to access them, which may mean storing your access keys or authorization information on the client and passing them along with every call. Some do not have an HTTP interface at all and must be accessed through server-side code that you have to go find and install or write yourself. Some allow you to design the primary keys or indexes of your storage tables, some do not. Some support JSON natively, some do not. In fact, the formats supported for data entry and retrieval (think filtering) vary almost as widely as the number of vendors. Get the drift? Different creators, different creations. But you say “I just want a simple way for my app to get my data in and out, quickly and easily.” That’s where I think we can help.

NoSQL as a Service

DSP Services allow you to “on-board” web-based or locally running services that you want to use from your app and access from one authenticated REST API. The access to these services can be controlled by app and by user role. There is more on this and how to provision these services in our documentation here.

The same goes for the supported NoSQL services. We securely store any credentials required to access your storage on the server-side, relieving your app from maintaining it. We remove any cross-site hoops your app would have to handle by making it a “native” REST service on the DSP. We even go a step further and provide a common, fully REST-based API to each of the supported NoSQL service types and use a common data format layout, using JSON, for data entry and retrieval. Where we were able, we also provide a common SQL-like filtering language, making it dirt simple to find the data you need.

We like to call this our “blend layer”. The idea is, just like a DAO (data access object), we simplify the process to use your data. In fact, we have made this so “blended” it is nearly identical to our SQL DB REST API, which we think is powerful but simple to use. So how does this help you? Well, you write the app once, and store your data in your favorite storage. If or when your “favorite” storage changes (i.e. if something cheaper or faster comes along), your app and data will only require minor changes (if any) to keep chugging along.

So what does this wonderful interface look like? Well, since you asked so nice and all. Once you get your service setup on your DSP (using the documentation mentioned earlier, or the vendor-specific follow-up blogs to this blog), you can use our Live API documentation interface to see how it works. Of course, you can use cURL or your favorite REST testing browser app, it is up to you. Since I am such the promotionalist, I’ll use the Swagger interface in this blog to show you.

As with the other DSP services, we put a mostly standard meaning to the HTTP verbs in our REST API. Here is the how they are used for NoSQL services (using “mongo” as our demo service).

DreamFactory MongoDB Rest API

Note that the ‘POST’ is used for creating new entities, ‘PUT’ is used for replacing the whole content of an entity and ‘PATCH’ is used to merge in changes to an entity (‘PATCH’ and ‘MERGE’ HTTP verbs are used interchangeably in the REST API). Also, as with the other DSP services, if your transport layer only accepts ‘GET’ and ‘POST’ verbs, using a ‘POST’ command while setting either a url parameter, ‘method=PATCH’, or a header, X-HTTP-Method = PATCH will “tunnel” the command through properly.

Table Administration

Azure calls them “tables”. So does AWS, that is for DynamoDb, but for SimpleDb, they are called “domains”. For CouchDb, they are called “databases”, not to be confused with MongoDb’s “databases” which house what they call “collections”. See what I mean?? While each vendor calls them something different, we will refer to them collectively as “tables”. These are the things that group your sets of data together or partition them, however you see it. You can actually manage most aspects of the table administration through the REST API.

To get a list of currently available tables, just send a GET request to the service’s root, which, if you are using one of our free-trial DSPs, looks like this.

https://dsp-mydspname.cloud.dreamfactory.com/rest/my_service_api_name_here

Every DSP service, when queried at its root with no extra url parameters, returns an array of resources available by that service. In the SQL and NoSQL case, these resources are table names.

{
   "resource": [
      {
         "name": "test"
      },
      {
         "name": "zipcodes"
      }
   ]
}

To include vendor-specific table information, if any is available, you can send ‘include_properties=true’ as a url parameter in the request. You can also limit the returned data by specifying the tables by name, using the ‘names=table1,table2,table3’ url parameter. The returned data will include a table array, where each object in the array includes at least the ‘name’ field and value, as well as, any available vendor-specific properties of the table, i.e. keys, etc.

https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo?include_properties=true&names=test%2Cuploads
{
   "table": [
       {
           "name": "test",
           "indexes": [
               {
                   "v": 1,
                    "key": {
                       "_id": 1
                   },
                   "ns": "test.test",
                   "name": "_id_"
               }
           ]
       },
       {
           "name": "uploads",
           "indexes": []
       }
   ]
}

Use this same data format for creating new tables, or updating and deleting existing tables. You can also use the ‘names’ url parameter in a delete request to delete existing tables.

Record Administration

Again, whether they are called “entities”, “documents”, or “items”, we collectively call them “records”. These are the groups of name-value pairs that make up your app’s data. The DSP REST API provides the following ways for your app to retrieve, create, update and delete data from the tables.

Retrieving Records

When it comes to making your NoSQL data available to your app, our REST API provides several different methods to get just the data sets you want, when you want it. Here are the available options for a GET request on a particular table.

table get request

The table name is sent as part of the url, while the other options are sent as url parameters. If url parameters are not your cup of tea, you can send most of them as posted data (using POST request with X-HTTP-METHOD = GET as mentioned earlier. Note that the “record” parameter can only be sent in this way.). Most of them are self explanatory, but a few may need clarification.

  • ids” – This is a comma delimited list of unique identifiers (think primary key) values for retrieving multiple records at a time. If passed as POST data, this can also be an array of ids. In the case where they is no static identifier field (DynamoDb), use  of the “id_field” parameter is necessary to indicate which field these values belong to.
  • filter” – This is where you use our simple SQL-like filter string. Comparison operators supported on every platform are ‘ = ‘, ‘ != ‘, ‘ > ‘, ‘ >= ‘, ‘ < ‘, ‘ <= ‘; or as their SQL short form ‘ eq ‘, ‘ ne ‘, ‘ gt ‘, ‘ ge ‘, ‘ lt ‘, ‘ le ‘. Depending on the db vendor, others such as ‘ contains ‘, ‘ like ‘, and ‘ begins_with ‘. Spaces surrounding the operator is required. String values on the right side of the comparison must be within single of double quotes. Depending on the db vendor, logical comparisons (AND, OR, NOT) are also supported. More on the differences in later blogs. The whole filter string must be url-encoded.
  • record” – Used only in POST data, this option allows you to send partial (minimally the identifier fields) or complete records back to the database to be “refreshed” with the latest values. This is also helpful in instances where the table has multiple indexes, i.e. DynamoDb and Azure Tables.

For example, if we want to find the first 3 records in the zipcodes table information that have a population of over 20,000 people, returning only the city, state and percentage over 40 years of age. If you are familiar with SQL, this would look like “SELECT city,state,PopulationOver40 from zipcodes WHERE pop > 20000;”. Here is what it looks like in our REST API.

https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes?filter=pop+%3E+20000&limit=3&fields=city%2Cstate%2CPopulationOver40

And this is what gets returned…

{
 "record": [
   {
     "_id": "28659",
     "city": "NORTH WILKESBORO",
     "state": "NC",
     "PopulationOver40": 9.19
   },
   {
     "_id": "31201",
     "city": "HUBER",
     "state": "GA",
     "PopulationOver40": 99.877
   },
   {
     "_id": "71291",
     "city": "WEST MONROE",
     "state": "LA",
     "PopulationOver40": 80.099
   }
 ]
}

Creating Records

Use this same data format above for creating new records. One could post an array of records, or a single record like this…

POST https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes
   {
     "_id": "29684",
     "city": "STARR",
     "state": "SC",
     "pop": 2889,
     "PopulationOver40": 82.4
   }

If the DB vendor does not automatically create the primary key, it must be included in the POST request. The applicable keys are always returned for successfully created records on a create request.

Updating and Merging Into Records

The same array of records or a single record format with changes is supported for updating records. Using the PUT HTTP verb as mentioned above will replace the whole record with the posted data if found by matching identifiers in the record.

PUT https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes
   {
     "_id": "29684",
     "city": "STARR",
     "state": "SC",
     "pop": 2890,
     "PopulationOver40": 79.4
   }

If you only want to merge changes into a record without having to reset everything, then use the MERGE or PATCH HTTP verb and send only the changes along with the identification fields.

PATCH https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes
   {
     "_id": "29684",
     "pop": 2890,
     "PopulationOver40": 79.4
   }

If you only want to update or merge data for one record, and it can be identified by a single key field, then you could also add the id to the end of the url, and pass only the fields that require change.

PATCH https://dsp-mydspname.cloud.dreamfactory.com/rest/mongo/zipcodes/29684
   {
     "pop": 2890,
     "PopulationOver40": 79.4
   }

Two other methods of merging data are by id list or filter (using “ids” or “filter” url parameters mentioned above for retrieving records). In these cases, if not natively supported, the server will query the table for the filtering results, merge in the record changes and push the changes back in an update request. This is an easy way of updating multiple records with the same field-value changes.

Deleting Records 

Deleting is similar to updating records but no posted data is required.

More to Come…

Stay tuned for a series of blogs on each of the supported NoSQL services and how to use them in your next awesome app.

Adding a Remote Web Service to Your DSP

 
ToddAppletonThe DreamFactory Services Platform (DSP) provides a flexible and powerful backend for your mobile apps. If you don’t have a DreamFactory account, sign up here. One of the key features is the ability to integrate any number of remote web services, and access them directly through the DreamFactory REST API. In this blog post we’ll walk through the process of setting up and invoking a remote web service. Continue reading “Adding a Remote Web Service to Your DSP”

SenchaCon, A Look Back

 
ToddAppletonLast week I had the pleasure of attending SenchaCon 2013 in Orlando, FL.  It was four days of fun learning including boot camps, new product announcements, and in-depth sessions on all aspects of Sencha and general web development.  Throughout the conference there were a few recurring themes that I wanted to touch on in this post, and in some cases connect the dots back to our DreamFactory Services Platform (DSP).

HTML5 for Mobile Apps

There was a lot of talk about the future of HTML5 and JavaScript for building mobile apps. The gap between what is possible with native apps vs HTML5 apps is quickly closing. Modern browsers are introducing new API’s and CSS capabilities further lessening the need for native app development. Jacky Nguyen gave a great talk on the making of Fastbook, an HTML5-based clone of the native Facebook app that uses some clever tricks to maximize performance, exceeding that of the native app in most cases. At DreamFactory we think the rich client is the way to go, and HTML5 and JavaScript will be the go-to tools.

REST and JSON Everywhere

This should come as no surprise, but pretty much everyone is expecting to exchange JSON data via a REST API.

Apps for the Enterprise

One of the big announcements last week was the introduction of Sencha Space. It’s intended to address feedback from enterprise customers that they need more than just cool apps. They need the ability to host and manage those apps. They need to control which users can access each app, and have the ability to grant or revoke access by a particular device.  They need the ability to report on usage – who logged in, when, and where. We agree!

Small Teams, Big Apps

I repeatedly saw examples of small, or even one person teams producing quality mobile apps using Sencha technologies. Sencha gave a preview of Architect 3.0 which provides a lot of drag and drop capability for building HTML5 apps, eliminating much of the repetitive code everyone hates to write. In the spirit of single engineer apps,  DreamFactory and Modus Create partnered for an unconference session to demo their Sencha-based address book app running on the DreamFactory backend. The app is a hybrid using either Ext JS or Sencha Touch based on device type.  It was built by a single engineer at Modus Create in just a few weeks with no server side development required. These are just a few of the take aways from SenchaCon 2013. It was a great conference and we’re looking forward to next year!

Four Big Problems DreamFactory Solves

 
BillAppletonThe DreamFactory Services Platform (DSP) solves four big problems that plague mobile application developers. First, we provide an easy to use REST API that covers a wide variety of backend services. Many server side assets like SQL databases don’t have a client addressable API. In cases like this we build out the entire API for the backend system. Other server side assets might have an API, but they don’t support JSON data, so we translate everything into JSON for ease of use. Lastly, different systems from different vendors tend to have wildly different or technically challenging network protocols and authentication methods. Go look at the interfaces for MongoDB, S3, Azure Tables, SimpleDB, SendGrid, MySQL, and CloudFiles. They all work differently. So we simplify all those systems into a single REST API that can be used from the client. Second, we hide the master credentials for all these different systems and expose their functionality through single sign-on with user roles and permissions. This means you never have to store or use a master credential from the client device where it can be stolen by a hacker. Instead you have a single user authenticate in to our service architecture where they are given limited capabilities to read, write, create, or update the various data sources. All of this is controlled by their user role. And if they lose their mobile device the system admin can instantly kill their session, protecting the backend systems. Third, we solve the originating host problem for HTML5 developers. Modern browsers limit web application communication to the originating server. This is a security measure designed to protect private networks behind the firewall. There are some exceptions to the rule, but in general HTML5 applications can always communicate with their originating host, while talking to other networks is problematic. So the DSP provides a single point of communication that combines data sources from other servers. By the way, your DSP also provides extensive application hosting options. Your CSS, HTML, and JavaScript files are stored right on the same platform as the other services for originating host access. If you need to use the REST API from another domain we can also do that with programmable CORs access that can be set up by the administrator. The DSP also makes life easier for native client technologies. When a native client attempts to communicate with a whole bunch of different servers bad things tend to happen. The client must open network communications with each backend system, handle a bunch of authentication protocols, load and manage all the data, handle network errors and latency, etc. This adds a lot of complexity to the application, especially on a mobile device with intermittent connectivity and limited bandwidth. But with a DSP your native client can communicate to a single endpoint and consume a single REST API that combines everything needed for application runtime. Lastly, the DSP provides administration for all of the different users, applications, and services that are needed. You can control which users see which applications, and which applications can use which services. You can use the DSP Admin Panel to create sophisticated user role and permission systems. For example, the sales, marketing, and executive personnel at a company might each log in and see a very different view of the applications and data. For a simpler scenario without a log in, you could create a “read only” user role that delivers all the various documents and database information that the application uses. The only actual user log in would be for the system administrator who curates the media stored on the server. Taken together, these four capabilities form the powerful backbone of the DreamFactory Services Platform. We are working hard to solve these backend problems so that you can focus on the front end. The result is high performance mobile applications written by much smaller development teams with reduced time to value.
http://www.dreamfactory.com

Try our Free Edition right now, or install our Apache License software package on your cloud or data center. Manage your applications with AWS, Azure, Rackspace, or any VM.

Great REST API’s You Can’t Use

 
BillAppletonThe Mongo database has a great REST API, but you can’t use it. They tell you this right in the docs: “Please note that this apiKey will give full access to all data within the databases belonging to your MongoLab account. If you do not have it secured on an app server, someone can gain access to it and your data.” In other words, you must include your API Key in the REST request to connect to your data. So this API cannot be used from a client. In a native application these credentials could be found in the binary source code or sniffed network traffic. In HTML5 someone could simply view the source and grab your master credentials. This API can only be used from a server. The Amazon S3 document storage system also has a REST API, but you can’t use that, either. Here’s how they explain it: “You then use your AWS secret access key to calculate the HMAC of that string… Finally, you add this signature as a parameter of the request by using the syntax described in this section.” In other words the secret key must be used to create each API request. If someone gains access to your secret key then they have access to all of your data. Not just your S3 files, I’m talking about ALL of your Amazon Web Services. Once again, you need a server to use S3 or other Amazon services. What about Windows Azure? Here is how they tell you to create an Azure BLOB or Azure Tables account: “The Create Storage Account operation creates a new storage account in Windows Azure. The Create Storage Account request may be specified as follows. Replace <subscription-id> with your subscription ID in the following URL…” So once again, you cannot use these services from HTML5 or from a native application without fear of having your master account exposed. So you must build a server to use these powerful services. Well, if you’re going to build a server, you might as well build a good one. Here are some of the issues you will probably want to think about: 1) Your server must hide the master credentials to all the remote services you need like MongoDB, S3, and Azure. This way your individual accounts with Rackspace, Amazon, Microsoft, etc. cannot be hacked. 2) Meanwhile, you must provide client access to these services. User management capabilities and single-sign on with password hashing and time sensitive session IDs are the best practice here. Now add powerful CRUD roles and permissions by user to safeguard data access. 3) You don’t want to rebuild your server over and over in the future. You need a comprehensive palette of services for all the modern capabilities you might need such as SQL data, NoSQL data, BLOB storage, application hosting, email, and external services. 4) While you are at it, might as well simplify the wildly different interfaces these guys use to talk to all their different services. Every single one of these beasts will need to be tamed with a simple, clean, easy to use REST interface that supports native JSON objects. 5) Make your service interface lightening fast with carefully optimized code, and then conduct some load testing in the real world with tens of thousands of users. Now you are ready for a security audit. 6) Be sure that you have solved the “originating host” problems for HTML5 users. They will need to access all these services from a single domain. A nice feature to add would be programmatic CORs access so that they could use your services from other domains if needed. So that’s the bad news. In order to use powerful services for your next application you are going to have to build a sophisticated server. But finally, some good news: we have already built this server just for you. The DreamFactory Services platform does all of this stuff. We’ve been working hard on this project for years. You can try it out at our website www.dreamfactory.com or install the open source software package on your favorite cloud for free. There is a REST API for everything you need. In fact you won’t need to do any server side programming at all. Just go write your application. So what are you waiting for?

Mobile App Development with CORs and Pub/Sub

 

JasonSykesThe ability to build apps from your desktop or local server is a huge plus for developers using their own debug tools, test suites, etc. With our latest Version 1.0.4 release, we now support these use cases plus the ability to run apps from outside of your DSP and still enjoy the flexibility of our REST API.

Simply log in to your DSP and click on the System Config menu item on the left hand side. On the System Config screen you’ll see the CORS Access options. Setting * will allow you to work from your desktop or anywhere. You may also use your localhost server, or another DSP or remote server you wish to have access to your data or service. Having the allowed verbs checkboxes enforces another layer of access control by locking down the entire DSP to use only the verbs checked for CORS. A future release will have CORS Access specifiable per application.

To use CORS without logging in, you’ll also have to allow for Guest Access, and create a Role for guest user access. This way you can limit any guest access to only database tables or services you see fit.

system config

If you’d like to pass authentication instead of having a guest role, consult our Swagger API docs here for the latest API call for logging in a user (see the /user/session POST request). That POST will return a session_id that you can pass as a new header called X-DreamFactory-Session-Token with all your future API requests. You will also need to include the API name you created with your application as X-DreamFactory-Application-Name if you haven’t already.

We’ve also added live data through a new publication / subscription framework.

We’re using Faye running on Node. Check here for the Faye documentation as well some good advice on security.

Need to see it working? Here’s a Plunker showing our Angular Todo Application using CORS and PubSub to talk to our demo server.

To see pub/sub working on it’s own, try this real time chat app on Plunker

Enjoy! And as always, if you have any questions, please contact us!

Shedding Development Pounds

 
BenBusseMuch has been written about how cloud computing, open source software, and programming frameworks have reduced software development costs exponentially in the last decade. As more than a few pundits have said, “anyone with a good idea and an Amazon Web Services account can now create the next Facebook!” Sure, the hyperbole is for dramatic effect, but the underlying point is true: anyone with programming skills can now create a web or mobile application faster and more cheaply than ever before. The benefits of these advances are clear in the world of startup companies and developers of consumer applications. But how do enterprises, with a far more complex set of requirements and stakeholders, benefit? How do enterprises leverage “agile” and “lean” software development practices, tools, and infrastructure to be more competitive and meet the increasingly demanding expectations of customers, business partners, and their own employees? And how does the technology shift to cloud infrastructure relate to the explosion of “bring your own device” mobile usage at work? There’s no single silver bullet answer to these questions, but several important trends are relevant to how developers of business apps should be thinking about the intersection of cloud technologies and mobile to make their projects successful.
  • HTML5 is ready for prime time as the best front-end standard for companies that need to build and deploy applications that work on desktop computers, tablets, and phones. HTML5 enables companies to write one application, instead of having to write entirely separate applications for the desktop, iOS, Android, Windows, and other mobile OS platforms.
  • Front-end mobile development toolkits like PhoneGap and Sencha and frameworks like jQuery Mobile and Angular.js will gain market influence since they complement platform- independent development across mobile and web. These toolkits enable developers to package a single application for installation on all mobile operating systems. Users can easily download and install these packaged apps onto their mobile devices just as they would a normal “native” mobile application from an app store.
  • Likewise, the mobile back-end stack for mobile applications will largely be open source,  standards-based, pre-configured, highly secure, and architected specifically for mobile usage. And the mobile back-end will not be subject to someone else’s control. You can install it on your own cloud infrastructure or data center. This is the area that we at DreamFactory are tackling. We’ve written more about this in a blog post about the evolution and design goals of the Dreamfactory Services Platform.
  • Developers will benefit enormously from both front-end and back-end simplification and standardization. Now an individual or small team of application developers can quickly create a prototype app, iterate on the design, create the application, and deliver it to end users in a production environment. Standardization also enables reusable design patterns (and reusable source code in many cases). We’ve written more about this in a blog post about the rise of the front-end developer.
  • The promise of these game changing trends is exciting but enterprises are by necessity risk averse and therefore reluctant to introduce new technologies that haven’t been battle tested in production environments. Enterprises need the flexibility and control to use the core back-end infrastructure that suits their application requirements, whether public cloud, hybrid public-private cloud, or on premise. Back-end service platforms like DreamFactory must be easily installed on the desired server infrastructure and must also integrate seamlessly with existing IT development, deployment, scaling, and security tools and practices. We’ll be writing a lot more about this topic in future posts. Stay tuned!

Getting Started: DSP Account Setup Screencast

 
BenBusseWe’ll be posting many screencasts to help you quickly learn how to use DreamFactory to build great apps. The first one to watch is how to sign up and get started building apps with your free trial of the DSP.

In our first ever Account Setup screencast, Todd Appleton, manager of software engineering, explains all you need to know to get up and running with DreamFactory. In just over 7 minutes you’ll learn how to:

  • Create your Dreamfactory account
  • Spin up a new DSP free trial instance
  • Launch applications on your free trial instance

describe the image

After watching Tom’s quick screencast, you’ll be ready to explore the platform on your own and begin building your first application.

Once you’ve signed up for a free trial, don’t forget to check out more screencasts, explore the REST API, and get familiar with our documentation.

We love feedback! Let us know what you think in the comments.