Vector Configuration

Overview

LogFleet uses Vector for log collection at the edge. The default configuration provides:

HTTP JSON input on port 9880
Syslog UDP input on port 514
Automatic enrichment with edge metadata
Metric extraction for counters and gauges
Local Loki storage with configurable retention

Default Sources

HTTP JSON (Primary)

Send JSON logs via HTTP POST:

curl -X POST http://localhost:9880 \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Order completed",
    "level": "info",
    "service": "pos",
    "order_id": "12345"
  }'

Optional Headers:

X-Device-ID: Device identifier
X-Source-Type: Device type (e.g., “pos”, “kiosk”)
X-Service: Service name

Syslog UDP

Send standard syslog messages:

echo "<14>Jan 15 10:30:00 pos-terminal app: Order completed" | nc -u localhost 514

Syslog severity is automatically mapped to log levels:

0-3: error
4: warn
5-6: info
7: debug

Automatic Enrichment

All logs are automatically enriched with:

Field	Source	Example
`source_type`	Input type	`http`, `syslog`
`edge_agent`	`AGENT_NAME` env	`store-001`
`location_id`	`LOCATION_ID` env	`nyc-01`
`region`	`REGION` env	`us-east`
`env`	`ENVIRONMENT` env	`production`
`timestamp`	Log or current time	`2024-01-15T10:30:00Z`
`level`	Normalized	`info`, `error`, etc.

Metric Extraction

Vector extracts metrics from logs automatically, reducing bandwidth by 100-1000x.

Automatic Metrics

Metric	Type	Tags	Trigger
`edge.logs`	counter	level, location_id	All logs
`edge.errors`	counter	location_id	`level=error/fatal/critical`
`edge.events`	counter	event, location_id	Logs with `.event` field
`edge.latency_ms`	gauge	location_id	Logs with `.latency_ms` field
`edge.order_value`	gauge	location_id	Logs with `.amount` field

Emit Business Events

Include an event field in your logs to create business metrics:

curl -X POST http://localhost:9880 \
  -H "Content-Type: application/json" \
  -d '{
    "event": "order_completed",
    "amount": 24.99,
    "latency_ms": 150,
    "service": "pos"
  }'

This creates three metrics:

edge.events{event="order_completed", location_id="..."} (counter)
edge.order_value{location_id="..."} (gauge: 24.99)
edge.latency_ms{location_id="..."} (gauge: 150)

Loki Labels

Logs are stored in Loki with these labels:

{source_type="http", edge_agent="store-001", location_id="nyc-01", level="info"}

Keep label cardinality low. High cardinality labels (like user IDs) can overwhelm Loki.

Customizing Vector Config

Override with Volume Mount

Mount a custom config file:

docker run -d \
  --name logfleet-agent \
  -e API_KEY=$API_KEY \
  -e CLOUD_URL=https://api.logfleet.io \
  -v /path/to/custom-vector.toml:/etc/vector/vector.toml:ro \
  ghcr.io/sadhiappan/logfleet-agent:latest

Add File Tailing

To collect from log files:

[sources.app_logs]
type = "file"
include = ["/var/log/app/*.log"]
read_from = "beginning"

[transforms.enrich_app_logs]
type = "remap"
inputs = ["app_logs"]
source = '''
.source_type = "file"
.edge_agent = get_env_var("AGENT_NAME") ?? "unknown"
.location_id = get_env_var("LOCATION_ID") ?? "unknown"
'''

Add Custom Metrics

Extract additional metrics from specific log patterns:

[transforms.custom_metrics]
type = "log_to_metric"
inputs = ["all_logs"]

[[transforms.custom_metrics.metrics]]
type = "counter"
field = "payment_method"
name = "payments"
namespace = "edge"
tags.method = "{{payment_method}}"
tags.location_id = "{{location_id}}"

Datadog Integration

By default, metrics are sent to Datadog (if DATADOG_API_KEY is set):

docker run -d \
  --name logfleet-agent \
  -e API_KEY=$API_KEY \
  -e CLOUD_URL=https://api.logfleet.io \
  -e DATADOG_API_KEY=your_datadog_api_key \
  ghcr.io/sadhiappan/logfleet-agent:latest

Cost Optimization:

Metrics are aggregated over 5-minute windows (99%+ reduction)
Only business metrics are sent (no internal Vector metrics)
Counters preferred over histograms (12x fewer data points)

Prometheus Metrics

A Prometheus exporter runs on port 9598 for local monitoring:

curl http://localhost:9598/metrics

Debugging

Enable Console Output

Uncomment the console sink in your Vector config:

[sinks.console]
type = "console"
inputs = ["all_logs"]
encoding.codec = "json"

Validate Configuration

vector validate --config /path/to/vector.toml

Check Vector API

curl http://localhost:8686/health

Environment Variables

Variable	Default	Purpose
`AGENT_NAME`	`edge-agent-1`	Agent identifier in logs
`LOCATION_ID`	`unknown`	Location identifier
`REGION`	`unknown`	Geographic region
`ENVIRONMENT`	`production`	Environment tag
`DATADOG_API_KEY`	(none)	Enable Datadog metrics

Full Default Configuration

The complete default Vector configuration is available at: github.com/sadhiappan/logfleet/blob/main/agent/configs/vector/vector.toml

Deployment

Operations

Support

Vector Configuration

Overview

Default Sources

HTTP JSON (Primary)

Syslog UDP

Automatic Enrichment

Metric Extraction

Automatic Metrics

Emit Business Events

Loki Labels

Customizing Vector Config

Override with Volume Mount

Add File Tailing

Add Custom Metrics

Datadog Integration

Prometheus Metrics

Debugging

Enable Console Output

Validate Configuration

Check Vector API

Environment Variables

Full Default Configuration

​Overview

​Default Sources

​HTTP JSON (Primary)

​Syslog UDP

​Automatic Enrichment

​Metric Extraction

​Automatic Metrics

​Emit Business Events

​Loki Labels

​Customizing Vector Config

​Override with Volume Mount

​Add File Tailing

​Add Custom Metrics

​Datadog Integration

​Prometheus Metrics

​Debugging

​Enable Console Output

​Validate Configuration

​Check Vector API

​Environment Variables

​Full Default Configuration

Overview

Default Sources

HTTP JSON (Primary)

Syslog UDP

Automatic Enrichment

Metric Extraction

Automatic Metrics

Emit Business Events

Loki Labels

Customizing Vector Config

Override with Volume Mount

Add File Tailing

Add Custom Metrics

Datadog Integration

Prometheus Metrics

Debugging

Enable Console Output

Validate Configuration

Check Vector API

Environment Variables

Full Default Configuration