Executive Summary
API observability is a critical operational requirement for production REST API platforms. DreamFactory, as an enterprise API generation and management platform, produces high-volume API traffic that demands robust logging, real-time analytics, and diagnostic capabilities. This guide demonstrates how to implement a complete observability stack using Logstash for log ingestion and processing, Elasticsearch for indexed storage and search, and Grafana for advanced visualization and alerting. This architecture provides engineering teams with the infrastructure needed for performance monitoring, security auditing, usage analytics, and rapid troubleshooting across all DreamFactory-managed APIs.
Modern API platforms like DreamFactory generate massive volumes of operational data. Every API request—whether against a MySQL database, MongoDB collection, AWS S3 bucket, or custom scripted endpoint—produces metadata that contains valuable insights:
Without proper observability infrastructure, this data is ephemeral—written to rotating log files and lost within hours or days. Engineering teams lose visibility into production behavior, making it difficult to:
The Logstash + Elasticsearch + Grafana stack solves these challenges by creating a durable, searchable, visualizable record of all API activity.
Logstash is an open-source data processing pipeline developed by Elastic. It ingests data from multiple sources, transforms it through filter plugins, and outputs it to various destinations. For API observability, Logstash serves as the critical bridge between raw application logs and structured, indexed data.
Logstash consumes DreamFactory's Laravel-formatted application logs, parses them into structured JSON documents, enriches them with contextual metadata (user details, service information, rate limit status), and delivers them to Elasticsearch for indexing. This transformation converts raw log lines like:
[2024-01-15 14:23:45] production.INFO: API Request {"method":"GET","endpoint":"/api/v2/mysql/_table/users","user_id":42,"duration_ms":156}
Into fully structured, queryable records with normalized timestamps, extracted dimensions, and calculated metrics.
Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. It stores documents in JSON format across sharded indices, providing millisecond query performance even across billions of records.
Elasticsearch stores every processed API log record in time-based indices (e.g., dreamfactory-logs-2024.01.15). It enables:
The combination of structured indexing and powerful query DSL makes Elasticsearch the analytical foundation of the observability stack.
Grafana is an open-source observability platform that creates interactive dashboards, visualizations, and alerts from multiple data sources. While originally designed for time-series metrics, modern Grafana excels at log analytics, trace visualization, and hybrid observability workflows.
Grafana transforms Elasticsearch query results into actionable insights through:
Unlike general-purpose databases, this stack is optimized for write-heavy, time-series workloads. Elasticsearch's inverted indices make substring searches and aggregations fast even across terabytes of log data. Logstash's pipeline architecture handles throughput spikes without data loss.
As DreamFactory's logging evolves—adding new fields, changing formats, or introducing new service types—Elasticsearch's dynamic mapping adapts automatically. No schema migrations or downtime required.
Logstash offers 200+ input, filter, and output plugins. Elasticsearch supports custom analyzers, scripting, and machine learning features. Grafana provides extensive visualization options and integrations with incident management platforms like PagerDuty and Slack.
The core stack is open source with no licensing costs. Organizations can run it on their own infrastructure, avoiding per-GB ingestion fees charged by commercial SaaS offerings. For teams already managing Kubernetes or VM infrastructure, this provides significant cost advantages at scale.
Elasticsearch's Query DSL enables complex analytical queries that would be cumbersome in SQL:
{
"aggs": {
"error_rate_by_service": {
"terms": {"field": "service_name"},
"aggs": {
"error_percentage": {
"bucket_script": {
"buckets_path": {
"errors": "status_codes.errors",
"total": "_count"
},
"script": "params.errors / params.total * 100"
}
}
}
}
}
}
This query calculates error rate by service in a single request—a pattern common in API observability.
Modern Elasticsearch deployments support not just logs but also Prometheus-style metrics and distributed traces. Grafana visualizes all three signal types in correlated dashboards, enabling true full-stack observability.
Running Elasticsearch in production requires expertise in:
Small teams may find this burden significant compared to managed alternatives like Datadog or New Relic.
Elasticsearch is memory-hungry. A production cluster typically requires:
High-volume DreamFactory deployments processing millions of API requests per day may need dedicated infrastructure just for observability.
While Elasticsearch offers machine learning features, they require manual model training and tuning. Unlike AI-powered SaaS tools that automatically detect anomalies, the LEG stack requires explicit threshold definitions and alerting rules.
Building effective Elasticsearch queries in Grafana requires understanding both Lucene query syntax and Elasticsearch aggregations. Business users may struggle to self-serve analytics without pre-built dashboards.
The following sections outline key configuration steps for implementing the LEG stack with DreamFactory. These configurations are platform-agnostic and can be deployed on-premises, in the cloud (AWS, Azure, GCP), using managed services (AWS OpenSearch, Elastic Cloud, Grafana Cloud), or in containerized environments (Docker, Kubernetes).
DreamFactory includes a built-in Logstash logging service that pushes API event data directly to Logstash via network protocols. This eliminates the need to access DreamFactory log files directly.
Service Configuration:
api-observability (or your preferred name)Logstash Connection Settings:
logstash.yourcompany.com or IP)12201 for GELF, 5000 for HTTP)Log Context Configuration:
Select which data elements to capture with each log entry:
Service Event Mapping:
Configure which DreamFactory events trigger log entries. Each event can have a custom log level and message. The event format is {service_name}.{event_type}.
Common event mappings:
Service: mysql.*
Log Level: INFO
Message: MySQL service activity
Service: mongodb.*
Log Level: INFO
Message: MongoDB service activity
Service: user.registered
Log Level: INFO
Message: New user registration
Service: system.admin.session.create
Log Level: WARNING
Message: Admin login detected
Event Pattern Examples:
mysql.post.post_process, mongodb.get.pre_processmysql.* (all events for MySQL service), user.* (all events for user service)For comprehensive API observability, create event mappings for all your DreamFactory services (database services, file storage services, custom scripted services, etc.)
Configure Logstash to receive data from the DreamFactory Logstash service. The input configuration must match the protocol you selected in DreamFactory.
The location and method for applying Logstash configuration depends on your deployment:
Self-Hosted Logstash:
/etc/logstash/conf.d/dreamfactory.confsudo systemctl restart logstashsudo /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/dreamfactory.confDocker/Docker Compose:
/usr/share/logstash/pipeline/dreamfactory.confdocker restart logstash or docker compose restart logstashKubernetes:
/usr/share/logstash/pipeline/kubectl rollout restart deployment/logstashManaged Services (AWS OpenSearch, Elastic Cloud):
input {
gelf {
port => 12201
type => "dreamfactory"
}
}
filter {
if [type] == "dreamfactory" {
# Parse DreamFactory event context
if [_platform][session][user] {
mutate {
add_field => {
"user_id" => "%{[_platform][session][user][id]}"
"user_email" => "%{[_platform][session][user][email]}"
}
}
}
# Extract request metadata
if [_event][request] {
mutate {
add_field => {
"endpoint" => "%{[_event][request][uri]}"
"method" => "%{[_event][request][method]}"
"service_name" => "%{[_event][request][service]}"
}
}
}
# Extract response metadata
if [_event][response] {
mutate {
add_field => {
"status_code" => "%{[_event][response][status_code]}"
}
}
}
# Classify response status
if [status_code] {
ruby {
code => '
status = event.get("status_code").to_i
if status >= 200 && status < 300
event.set("status_class", "success")
elsif status >= 400 && status < 500
event.set("status_class", "client_error")
elsif status >= 500
event.set("status_class", "server_error")
end
'
}
}
# Calculate performance tier if duration is available
# Note: You may need to calculate duration from timestamps
# Clean up nested structures for simpler querying
mutate {
remove_field => ["_platform", "_event"]
}
}
}
output {
elasticsearch {
hosts => ["https://your-elasticsearch-host:9200"]
index => "dreamfactory-api-%{+YYYY.MM.dd}"
}
}
input {
http {
port => 5000
type => "dreamfactory"
codec => json
}
}
filter {
# Use the same filter configuration as GELF above
}
output {
elasticsearch {
hosts => ["https://your-elasticsearch-host:9200"]
index => "dreamfactory-api-%{+YYYY.MM.dd}"
}
}
input {
tcp {
port => 5000
type => "dreamfactory"
codec => json_lines
}
}
filter {
# Use the same filter configuration as GELF above
}
output {
elasticsearch {
hosts => ["https://your-elasticsearch-host:9200"]
index => "dreamfactory-api-%{+YYYY.MM.dd}"
}
}
Once you have Elasticsearch deployed in your environment of choice, configure it to optimally store and index DreamFactory API logs.
Create an index template to optimize field mappings for DreamFactory API logs:
curl -X PUT "https://your-elasticsearch-host:9200/_index_template/dreamfactory-api-template" -H 'Content-Type: application/json' -d'
{
"index_patterns": ["dreamfactory-api-*"],
"template": {
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.codec": "best_compression"
},
"mappings": {
"properties": {
"@timestamp": {"type": "date"},
"endpoint": {"type": "keyword"},
"method": {"type": "keyword"},
"service_name": {"type": "keyword"},
"service_type": {"type": "keyword"},
"service_category": {"type": "keyword"},
"status_code": {"type": "short"},
"status_class": {"type": "keyword"},
"duration_ms": {"type": "float"},
"performance_tier": {"type": "keyword"},
"user_id": {"type": "integer"},
"user_email": {"type": "keyword"},
"ip_address": {"type": "ip"},
"geo": {
"properties": {
"location": {"type": "geo_point"}
}
}
}
}
}
}
'
Automate data retention and lifecycle management with an ILM policy:
curl -X PUT "https://your-elasticsearch-host:9200/_ilm/policy/dreamfactory-api-policy" -H 'Content-Type: application/json' -d'
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "50GB",
"max_age": "1d"
}
}
},
"warm": {
"min_age": "7d",
"actions": {
"shrink": {"number_of_shards": 1},
"forcemerge": {"max_num_segments": 1}
}
},
"delete": {
"min_age": "90d",
"actions": {"delete": {}}
}
}
}
}
'
This policy:
Once Grafana is deployed in your environment, connect it to your Elasticsearch instance and build dashboards for visualizing DreamFactory API metrics.
In Grafana, navigate to Configuration → Data Sources → Add data source → Elasticsearch and configure:
Basic Settings:
https://your-elasticsearch-host:9200)Production Elasticsearch deployments should have authentication enabled. Choose the appropriate method:
elastic or custom user)# Set password for elastic user
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
Create an API key in Elasticsearch:
curl -X POST "https://your-elasticsearch-host:9200/_security/api_key" \
-u elastic:your-password \
-H "Content-Type: application/json" \
-d '{
"name": "grafana-dreamfactory-api-logs",
"role_descriptors": {
"grafana_reader": {
"cluster": ["monitor"],
"indices": [
{
"names": ["dreamfactory-api-*"],
"privileges": ["read", "view_index_metadata"]
}
]
}
}
}'
Authorization with value: ApiKey <base64-encoded-api-key>dreamfactory-api-*dreamfactory-api-YYYY.MM.dd index format)@timestampmessagelevelDreamFactory's Laravel foundation makes it naturally compatible with this observability stack. The platform's service-oriented architecture—where each database, file storage, or external API is a distinct service—maps perfectly to Elasticsearch's structured indexing. Every DreamFactory service generates consistent, well-formatted logs that Logstash can process without complex parsing logic.
One of DreamFactory's core value propositions is automatic API generation from existing data sources. This creates a challenge: how do you monitor APIs you didn't manually code? The LEG stack solves this by capturing every auto-generated endpoint's performance, regardless of whether it's a table operation, stored procedure call, or complex join query. Engineering teams gain the same observability for generated APIs as they would for hand-written code.
DreamFactory often powers multi-tenant SaaS applications where a single instance serves multiple customers. The LEG stack enables per-tenant performance monitoring by indexing user_id, api_key, or custom tenant identifiers. Operations teams can:
DreamFactory includes basic request logging and rate limiting, but lacks deep analytics and long-term trend analysis. The LEG stack extends these capabilities without replacing them:
This separation of concerns keeps DreamFactory focused on API generation and management while delegating observability to specialized tools.
When DreamFactory deployments encounter issues—slow queries, authentication failures, or integration breakages—the LEG stack dramatically reduces diagnostic time. Instead of SSH-ing into containers and grepping log files, engineers query Elasticsearch for precise time ranges, error patterns, and correlated events. A problem that might take hours to diagnose through manual log review can be isolated in minutes with targeted Grafana dashboards.
Organizations in healthcare, finance, and government often face strict audit logging requirements. DreamFactory's API logs, when indexed in Elasticsearch with appropriate retention policies, provide:
The LEG stack transforms DreamFactory from a simple API platform into a compliance-ready enterprise solution.
This architecture is ideal when:
Consider managed alternatives when:
The LEG stack represents a middle path: more powerful than simple log aggregation, more cost-effective than enterprise APM platforms, and more controllable than fully-managed SaaS offerings. For organizations running DreamFactory at scale, it provides the observability foundation needed to operate confidently in production.
The DreamFactory Logstash service provides several advantages over file-based logging:
Real-time data: Events are pushed to Logstash as they occur, enabling immediate visibility into API activity rather than waiting for batch log processing.
Structured data by default: DreamFactory sends fully structured event data with user context, session information, and request/response metadata directly to Logstash. No complex parsing or regex patterns needed.
Works everywhere: Whether DreamFactory runs in Docker containers, Kubernetes, serverless functions, or traditional VMs, the network-based approach works identically. File-based logging struggles with ephemeral containers and distributed deployments.
Security: Network transmission supports encryption (TLS) and authentication. Log files on disk require careful permissions management and create potential security risks.
The Logstash service is designed for minimal performance impact:
Asynchronous delivery: Log events are sent to Logstash after the API response is returned to the client. Users experience no latency from logging operations.
Efficient protocols: GELF (UDP) adds less than 1ms overhead per request. Even HTTP-based logging adds only 2-5ms on average.
Selective logging: Use service event mappings to log only what matters. For example, log data modification operations (mysql.post, mongodb.delete) but skip high-frequency read operations to reduce overhead.
In production deployments handling 10,000+ requests per minute, the Logstash service typically adds less than 0.5% CPU overhead and has no measurable impact on API throughput.
Yes. The DreamFactory Logstash service works with both self-hosted and managed observability platforms:
Managed Elasticsearch options:
Managed Grafana options:
Fully managed alternatives:
The DreamFactory Logstash service can send data to commercial observability platforms like Datadog, New Relic, or Splunk via their HTTP endpoints. Simply configure the service to use HTTP protocol and point it to the platform's ingestion API.
The choice between self-hosted and managed services depends on your team's operational expertise, compliance requirements, and preference for control versus convenience. The DreamFactory Logstash service configuration is identical regardless—just point it to your Logstash endpoint.
Implementing Logstash, Elasticsearch, and Grafana for DreamFactory API observability creates a production-grade monitoring system that scales with your platform's growth. This architecture transforms raw API logs into actionable insights, enabling engineering teams to detect issues proactively, investigate problems efficiently, and optimize performance systematically.
By following the implementation patterns outlined in this guide—structured logging in DreamFactory, enriched processing in Logstash, indexed storage in Elasticsearch, and visual analytics in Grafana—you build an observability stack that evolves with your needs. The flexibility of open-source tools combined with DreamFactory's extensible architecture provides a foundation for long-term operational excellence.
For DreamFactory users committed to enterprise-scale API management, the LEG stack is not just a monitoring solution—it's a strategic investment in reliability, security, and data-driven decision making.