Logstash integration enhances DreamFactory logging capabilities

The Log service now supports integration with Logstash. Easily connect your DreamFactory instance to a Logstash service listening for input on UDP, TCP, or HTTP protocol. Once you create a DreamFactory Logstash service, you can utilize a unified REST API and start logging activities directly from your application or from DreamFactory using scripting services. Logstash is a native DreamFactory service and is supported by features such as role-service-access, lookup usage, live API documentation, and caching.

In this blog I will walk you through how to create a Logstash service and then use that service in a post-process event script to log user login activity and in a pre-process event script to log user logout activity.

Configuring a Logstash service is as simple as configuring any other service in the DreamFactory admin console. You start by logging into the admin console, head over to the ‘Services’ tab, and click on the ‘Create’ button on the left sidebar. From the ‘Service Type’ drop down menu, select Log -> Logstash, complete the service configuration form, and save it to create your service. Here’s how the Logstash configuration looks like.

Screen_Shot_2016-10-18_at_1.49.29_PM.png

Provide basic service information in the ‘Info’ tab – Name, Label, and Description. We are naming our service ‘logstash’ here.

Screen_Shot_2016-10-18_at_1.49.42_PM.png

Then provide configuration information in the ‘Config’ tab.

  • Host – String. Required. IP Address/Hostname of the machine running the Logstash service.
  • Port – Integer. Required. Port number that Logstash is listening on for inputs.
  • Protocol/Format – String. Required. Network protocol/format that Logstash input is configured for. Supported options are GELF (UDP), HTTP, TCP, UDP
    • GELF (UDP) – Choose this if your Logstash service is configured to accept GELF format – http://docs.graylog.org/en/2.1/pages/gelf.html.
    • HTTP – Choose this if your Logstash service is configured to listen on HTTP protocol. Data is sent using JSON format.
    • TCP – Choose this if your Logstash service is configured to listen on TCP protocol. Data is sent using JSON format.
    • UDP – Choose this if your Logstash service is configured to listen on UDP protocol. Data is sent using JSON format.

Once your Logstash service is configured, head over to the ‘API Docs’ tab and expand your newly created Logstash service to see all the available APIs ready for you to use.

Screen_Shot_2016-10-18_at_1.49.52_PM.png

Now we’ll use the POST /logstash API for our post-process event script that we’ll attach to the ‘user.session.post.post_process’ event to log user login activity. We’ll also attach a pre-process event script to ‘user.session.delete.pre_process’ event to log user logout activity.

Screen_Shot_2016-10-18_at_1.50.07_PM.png

Create the event scripts:

Head over to ‘Scripts’ tab and select:

user -> user.session -> post -> user.session.post.post_process.

Select ‘V8JS’ from script type/language drop down, check the ‘Active’ checkbox and put the following script in the script editor.

if(event.response.status_code === 200){

var email = '';

if(platform.session.user && platform.session.user.email){
    email = platform.session.user.email;
}

var log = {
    "level":"alert",
    "message":"Logging in [" + email + "]"
};

platform.api.post('logstash', log);
}

Save the script by clicking on the save button and then use the back button to go back to ‘user.session’ level and select the following.

user -> user.session -> delete -> user.session.delete.pre_process.

Select ‘V8JS’ from script type/language drop down, check the ‘Active’ checkbox and put the following script in the script editor.

var email = '';

if(platform.session.user && platform.session.user.email){
    email = platform.session.user.email;
}

var log = {
    "level":"alert",
    "message":"Logging out [" + email + "]"
};

platform.api.post('logstash', log);

In order for your scripts to be able to access the Logstash service you will need to create a role that allows ‘Script’ access to Logstash service. For this, we head over to the ‘Roles’ tab and create a role call ‘log_role’.

Screen_Shot_2016-10-18_at_1.50.23_PM-1.png

After you enter the basic role information, check the ‘Active’ checkbox and click on the ‘Access’ tab. Here we allow script access to our Logstash service for the method POST. Leave the ‘Component’ drop down as is (* selection).

Screen_Shot_2016-10-18_at_1.50.36_PM.png

Once we’ve created this role we’ll need to assign it to an app for which we want to log the user activity. Head over to the ‘Apps’ tab and select your application and then select your newly created role (log_role) in the ‘Assign a Default Role’ drop down. Now you are all set to start logging user login and logout activity in Logstash. Login to your app and then logout and checkout your log messages in your Logstash output.

Note: The DreamFactory Logstash service only interacts with Logstash. If you are using an entire ELK stack (Elasticsearch, Logstash, Kibana) for your logging, you need to set up and configure your ELK stack. To accept log messages from DreamFactory’s Logstash service, just configure your Logstash to listen on UDP, TCP, or HTTP port. See this link for installing and configuring Logstash.

DreamFactory’s Logstash service does not support any authentication. You can either set up your Logstash service locally on the same machine where your DreamFactory instance is running and then connect to it using ‘localhost’ or set up your Logstash service on a different machine in your internal network and connect to it using the internal IP. You can then configure Logstash to output to a local or remote Elasticsearch service including SaaS solutions like Elastic Cloud.

 

 



Get started with DreamFactory with a free hosted DreamFactory development environment. Or, download and run it on the server, cloud, or desktop of your choice.