Skip to main content
LogFleet’s log-to-metric extraction lets you derive metrics from your logs at the edge, reducing cloud bandwidth by 100-1000x compared to shipping raw logs.

​
How It Works

Instead of shipping every log line to the cloud: 
❌ Traditional: 1M logs/hour × 500 bytes = 500 MB/hour
LogFleet extracts metrics at the edge: 
✅ LogFleet: 1M logs/hour → 100 metric points = 10 KB/hour

​
Creating a Metric Config

​
Via API

curl -X POST https://api.logfleet.io/api/v1/metric-configs \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nginx_metrics",
    "description": "Extract metrics from nginx access logs",
    "log_source": "nginx",
    "enabled": true,
    "metrics": [
      {
        "name": "http_requests_total",
        "type": "counter",
        "description": "Total HTTP requests",
        "pattern": "\"(GET|POST|PUT|DELETE|PATCH)\\s",
        "labels": {
          "method": "$1"
        }
      },
      {
        "name": "http_response_status",
        "type": "counter",
        "description": "HTTP response status codes",
        "pattern": "HTTP/\\d\\.\\d\"\\s(\\d{3})",
        "labels": {
          "status": "$1"
        }
      },
      {
        "name": "request_duration_seconds",
        "type": "histogram",
        "description": "Request duration in seconds",
        "pattern": "request_time=(\\d+\\.\\d+)",
        "value_capture": "$1",
        "buckets": [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
      }
    ]
  }'

​
Metric Types

​
Counter

Counts occurrences of a pattern. Useful for counting events. 
{
  "name": "error_count",
  "type": "counter",
  "pattern": "level=(ERROR|FATAL)",
  "labels": {
    "level": "$1"
  }
}
Use cases:
  • HTTP request counts
  • Error counts by type
  • Event occurrences

​
Gauge

Captures the current value. Useful for point-in-time measurements. 
{
  "name": "queue_size",
  "type": "gauge",
  "pattern": "queue_size=(\\d+)",
  "value_capture": "$1"
}
Use cases:
  • Queue depths
  • Connection counts
  • Temperature readings
  • Memory usage

​
Histogram

Tracks value distributions. Useful for latencies and sizes. 
{
  "name": "response_time_seconds",
  "type": "histogram",
  "pattern": "response_time_ms=(\\d+)",
  "value_capture": "$1",
  "value_transform": "divide_by_1000",
  "buckets": [0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
}
Use cases:
  • Response times
  • Payload sizes
  • Processing durations

​
Pattern Syntax

Patterns use regular expressions with capture groups:
PatternMatchesCapture
level=(\\w+)level=ERRORERROR
status=(\\d{3})status=200200
duration=([\\d.]+)msduration=123.45ms123.45

​
Common Patterns

{
  "metrics": [
    {
      "name": "http_requests_total",
      "type": "counter",
      "pattern": "\"(GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS)\\s([^\\s]+)\\sHTTP",
      "labels": {
        "method": "$1",
        "path": "$2"
      }
    },
    {
      "name": "http_status_total",
      "type": "counter",
      "pattern": "HTTP/\\d\\.\\d\"\\s(\\d{3})",
      "labels": {
        "status": "$1",
        "status_class": "${1:0:1}xx"
      }
    },
    {
      "name": "http_bytes_total",
      "type": "counter",
      "pattern": "HTTP/\\d\\.\\d\"\\s\\d{3}\\s(\\d+)",
      "value_capture": "$1"
    }
  ]
}

{
  "metrics": [
    {
      "name": "log_messages_total",
      "type": "counter",
      "pattern": "\"level\":\\s*\"(DEBUG|INFO|WARN|ERROR|FATAL)\"",
      "labels": {
        "level": "$1"
      }
    },
    {
      "name": "api_latency_seconds",
      "type": "histogram",
      "pattern": "\"duration_ms\":\\s*(\\d+)",
      "value_capture": "$1",
      "value_transform": "divide_by_1000",
      "buckets": [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
    }
  ]
}

{
  "metrics": [
    {
      "name": "syslog_messages_total",
      "type": "counter",
      "pattern": "<(\\d+)>",
      "labels": {
        "priority": "$1"
      }
    },
    {
      "name": "auth_failures_total",
      "type": "counter",
      "pattern": "(Failed password|authentication failure)",
      "labels": {
        "type": "auth_failure"
      }
    }
  ]
}

​
Labels

Labels add dimensions to your metrics, enabling powerful queries: 
{
  "name": "http_requests_total",
  "labels": {
    "method": "$1",
    "status": "$2",
    "service": "api-gateway"
  }
}

​
Static Labels

Fixed values that apply to all matches: 
"labels": {
  "environment": "production",
  "datacenter": "us-east-1"
}

​
Dynamic Labels

Captured from the log line: 
"labels": {
  "method": "$1",
  "path": "$2"
}

​
Cardinality Warning

High-cardinality labels (like user IDs or request IDs) can cause performance issues. Stick to labels with bounded values (methods, status codes, environments).

​
Value Transforms

Transform captured values before recording:
TransformDescriptionExample
divide_by_1000Convert ms to seconds1234 → 1.234
divide_by_1000000Convert μs to seconds1234567 → 1.234567
multiply_by_1000Convert seconds to ms1.234 → 1234

​
Testing Patterns

Test your patterns before deploying: 
# Sample log line
echo '192.168.1.1 - - [15/Jan/2024:10:00:00 +0000] "GET /api/users HTTP/1.1" 200 1234 0.045' | \
  grep -oP '"(GET|POST|PUT|DELETE)\s'

​
Deployment

Metric configs are automatically synced to agents:
  1. Create/update config via API
  2. Agents sync every 60 seconds
  3. New metrics start appearing immediately

​
Version Control

Configs use optimistic locking. When updating: 
# Include the current version
curl -X PUT https://api.logfleet.io/api/v1/metric-configs/{id} \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "version": 3,
    "name": "nginx_metrics",
    ...
  }'
If the version doesn’t match, you’ll get 409 Conflict.

​
Viewing Metrics

Query metrics from the LogFleet dashboard or via API: 
curl -X GET "https://api.logfleet.io/api/v1/metric-configs/{id}/metrics?start=1h-ago" \
  -H "Authorization: Bearer $TOKEN"

​
Best Practices

Start Simple

Begin with a few key metrics and expand based on needs.

Use Histograms for Latency

Histograms provide percentiles; counters only give averages.

Limit Label Cardinality

Avoid high-cardinality labels that create millions of time series.

Test Patterns

Validate regex patterns against sample logs before deployment.