> ## Documentation Index
> Fetch the complete documentation index at: https://docs.logfleet.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Metrics

> Extract metrics from logs at the edge

LogFleet's log-to-metric extraction lets you derive metrics from your logs at the edge, reducing cloud bandwidth by **100-1000x** compared to shipping raw logs.

## How It Works

Instead of shipping every log line to the cloud:

```
❌ Traditional: 1M logs/hour × 500 bytes = 500 MB/hour
```

LogFleet extracts metrics at the edge:

```
✅ LogFleet: 1M logs/hour → 100 metric points = 10 KB/hour
```

## Creating a Metric Config

### Via API

```bash theme={null}
curl -X POST https://api.logfleet.io/api/v1/metric-configs \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nginx_metrics",
    "description": "Extract metrics from nginx access logs",
    "log_source": "nginx",
    "enabled": true,
    "metrics": [
      {
        "name": "http_requests_total",
        "type": "counter",
        "description": "Total HTTP requests",
        "pattern": "\"(GET|POST|PUT|DELETE|PATCH)\\s",
        "labels": {
          "method": "$1"
        }
      },
      {
        "name": "http_response_status",
        "type": "counter",
        "description": "HTTP response status codes",
        "pattern": "HTTP/\\d\\.\\d\"\\s(\\d{3})",
        "labels": {
          "status": "$1"
        }
      },
      {
        "name": "request_duration_seconds",
        "type": "histogram",
        "description": "Request duration in seconds",
        "pattern": "request_time=(\\d+\\.\\d+)",
        "value_capture": "$1",
        "buckets": [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
      }
    ]
  }'
```

## Metric Types

### Counter

Counts occurrences of a pattern. Useful for counting events.

```json theme={null}
{
  "name": "error_count",
  "type": "counter",
  "pattern": "level=(ERROR|FATAL)",
  "labels": {
    "level": "$1"
  }
}
```

**Use cases:**

* HTTP request counts
* Error counts by type
* Event occurrences

### Gauge

Captures the current value. Useful for point-in-time measurements.

```json theme={null}
{
  "name": "queue_size",
  "type": "gauge",
  "pattern": "queue_size=(\\d+)",
  "value_capture": "$1"
}
```

**Use cases:**

* Queue depths
* Connection counts
* Temperature readings
* Memory usage

### Histogram

Tracks value distributions. Useful for latencies and sizes.

```json theme={null}
{
  "name": "response_time_seconds",
  "type": "histogram",
  "pattern": "response_time_ms=(\\d+)",
  "value_capture": "$1",
  "value_transform": "divide_by_1000",
  "buckets": [0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
}
```

**Use cases:**

* Response times
* Payload sizes
* Processing durations

## Pattern Syntax

Patterns use regular expressions with capture groups:

| Pattern                | Matches             | Capture  |
| ---------------------- | ------------------- | -------- |
| `level=(\\w+)`         | `level=ERROR`       | `ERROR`  |
| `status=(\\d{3})`      | `status=200`        | `200`    |
| `duration=([\\d.]+)ms` | `duration=123.45ms` | `123.45` |

### Common Patterns

<Accordion title="HTTP Logs (Apache/Nginx Combined Format)">
  ```json theme={null}
  {
    "metrics": [
      {
        "name": "http_requests_total",
        "type": "counter",
        "pattern": "\"(GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS)\\s([^\\s]+)\\sHTTP",
        "labels": {
          "method": "$1",
          "path": "$2"
        }
      },
      {
        "name": "http_status_total",
        "type": "counter",
        "pattern": "HTTP/\\d\\.\\d\"\\s(\\d{3})",
        "labels": {
          "status": "$1",
          "status_class": "${1:0:1}xx"
        }
      },
      {
        "name": "http_bytes_total",
        "type": "counter",
        "pattern": "HTTP/\\d\\.\\d\"\\s\\d{3}\\s(\\d+)",
        "value_capture": "$1"
      }
    ]
  }
  ```
</Accordion>

<Accordion title="Application Logs (JSON Format)">
  ```json theme={null}
  {
    "metrics": [
      {
        "name": "log_messages_total",
        "type": "counter",
        "pattern": "\"level\":\\s*\"(DEBUG|INFO|WARN|ERROR|FATAL)\"",
        "labels": {
          "level": "$1"
        }
      },
      {
        "name": "api_latency_seconds",
        "type": "histogram",
        "pattern": "\"duration_ms\":\\s*(\\d+)",
        "value_capture": "$1",
        "value_transform": "divide_by_1000",
        "buckets": [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
      }
    ]
  }
  ```
</Accordion>

<Accordion title="Syslog Messages">
  ```json theme={null}
  {
    "metrics": [
      {
        "name": "syslog_messages_total",
        "type": "counter",
        "pattern": "<(\\d+)>",
        "labels": {
          "priority": "$1"
        }
      },
      {
        "name": "auth_failures_total",
        "type": "counter",
        "pattern": "(Failed password|authentication failure)",
        "labels": {
          "type": "auth_failure"
        }
      }
    ]
  }
  ```
</Accordion>

## Labels

Labels add dimensions to your metrics, enabling powerful queries:

```json theme={null}
{
  "name": "http_requests_total",
  "labels": {
    "method": "$1",
    "status": "$2",
    "service": "api-gateway"
  }
}
```

### Static Labels

Fixed values that apply to all matches:

```json theme={null}
"labels": {
  "environment": "production",
  "datacenter": "us-east-1"
}
```

### Dynamic Labels

Captured from the log line:

```json theme={null}
"labels": {
  "method": "$1",
  "path": "$2"
}
```

### Cardinality Warning

<Warning>
  High-cardinality labels (like user IDs or request IDs) can cause performance issues.
  Stick to labels with bounded values (methods, status codes, environments).
</Warning>

## Value Transforms

Transform captured values before recording:

| Transform           | Description           | Example                |
| ------------------- | --------------------- | ---------------------- |
| `divide_by_1000`    | Convert ms to seconds | `1234` → `1.234`       |
| `divide_by_1000000` | Convert μs to seconds | `1234567` → `1.234567` |
| `multiply_by_1000`  | Convert seconds to ms | `1.234` → `1234`       |

## Testing Patterns

Test your patterns before deploying:

```bash theme={null}
# Sample log line
echo '192.168.1.1 - - [15/Jan/2024:10:00:00 +0000] "GET /api/users HTTP/1.1" 200 1234 0.045' | \
  grep -oP '"(GET|POST|PUT|DELETE)\s'
```

## Deployment

Metric configs are automatically synced to agents:

1. Create/update config via API
2. Agents sync every 60 seconds
3. New metrics start appearing immediately

### Version Control

Configs use optimistic locking. When updating:

```bash theme={null}
# Include the current version
curl -X PUT https://api.logfleet.io/api/v1/metric-configs/{id} \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "version": 3,
    "name": "nginx_metrics",
    ...
  }'
```

If the version doesn't match, you'll get `409 Conflict`.

## Viewing Metrics

Query metrics from the LogFleet dashboard or via API:

```bash theme={null}
curl -X GET "https://api.logfleet.io/api/v1/metric-configs/{id}/metrics?start=1h-ago" \
  -H "Authorization: Bearer $TOKEN"
```

## Best Practices

<CardGroup cols={2}>
  <Card title="Start Simple" icon="play">
    Begin with a few key metrics and expand based on needs.
  </Card>

  <Card title="Use Histograms for Latency" icon="chart-histogram">
    Histograms provide percentiles; counters only give averages.
  </Card>

  <Card title="Limit Label Cardinality" icon="tags">
    Avoid high-cardinality labels that create millions of time series.
  </Card>

  <Card title="Test Patterns" icon="vial">
    Validate regex patterns against sample logs before deployment.
  </Card>
</CardGroup>