Overview
LogFleet uses Vector for log collection at the edge. The default configuration provides:
- HTTP JSON input on port 9880
- Syslog UDP input on port 514
- Automatic enrichment with edge metadata
- Metric extraction for counters and gauges
- Local Loki storage with configurable retention
Default Sources
HTTP JSON (Primary)
Send JSON logs via HTTP POST:
curl -X POST http://localhost:9880 \
-H "Content-Type: application/json" \
-d '{
"message": "Order completed",
"level": "info",
"service": "pos",
"order_id": "12345"
}'
X-Device-ID: Device identifierX-Source-Type: Device type (e.g., “pos”, “kiosk”)X-Service: Service name
Syslog UDP
Send standard syslog messages:
echo "<14>Jan 15 10:30:00 pos-terminal app: Order completed" | nc -u localhost 514
- 0-3: error
- 4: warn
- 5-6: info
- 7: debug
Automatic Enrichment
All logs are automatically enriched with:
| Field | Source | Example |
|---|
source_type | Input type | http, syslog |
edge_agent | AGENT_NAME env | store-001 |
location_id | LOCATION_ID env | nyc-01 |
region | REGION env | us-east |
env | ENVIRONMENT env | production |
timestamp | Log or current time | 2024-01-15T10:30:00Z |
level | Normalized | info, error, etc. |
Automatic Metrics
| Metric | Type | Tags | Trigger |
|---|
edge.logs | counter | level, location_id | All logs |
edge.errors | counter | location_id | level=error/fatal/critical |
edge.events | counter | event, location_id | Logs with .event field |
edge.latency_ms | gauge | location_id | Logs with .latency_ms field |
edge.order_value | gauge | location_id | Logs with .amount field |
Emit Business Events
Include an event field in your logs to create business metrics:
curl -X POST http://localhost:9880 \
-H "Content-Type: application/json" \
-d '{
"event": "order_completed",
"amount": 24.99,
"latency_ms": 150,
"service": "pos"
}'
edge.events{event="order_completed", location_id="..."} (counter)edge.order_value{location_id="..."} (gauge: 24.99)edge.latency_ms{location_id="..."} (gauge: 150)
Loki Labels
Logs are stored in Loki with these labels:
{source_type="http", edge_agent="store-001", location_id="nyc-01", level="info"}
Keep label cardinality low. High cardinality labels (like user IDs) can overwhelm Loki.
Customizing Vector Config
Override with Volume Mount
Mount a custom config file:
docker run -d \
--name logfleet-agent \
-e API_KEY=$API_KEY \
-e CLOUD_URL=https://api.logfleet.io \
-v /path/to/custom-vector.toml:/etc/vector/vector.toml:ro \
ghcr.io/sadhiappan/logfleet-agent:latest
Add File Tailing
To collect from log files:
[sources.app_logs]
type = "file"
include = ["/var/log/app/*.log"]
read_from = "beginning"
[transforms.enrich_app_logs]
type = "remap"
inputs = ["app_logs"]
source = '''
.source_type = "file"
.edge_agent = get_env_var("AGENT_NAME") ?? "unknown"
.location_id = get_env_var("LOCATION_ID") ?? "unknown"
'''
Add Custom Metrics
Extract additional metrics from specific log patterns:
[transforms.custom_metrics]
type = "log_to_metric"
inputs = ["all_logs"]
[[transforms.custom_metrics.metrics]]
type = "counter"
field = "payment_method"
name = "payments"
namespace = "edge"
tags.method = "{{payment_method}}"
tags.location_id = "{{location_id}}"
Datadog Integration
By default, metrics are sent to Datadog (if DATADOG_API_KEY is set):
docker run -d \
--name logfleet-agent \
-e API_KEY=$API_KEY \
-e CLOUD_URL=https://api.logfleet.io \
-e DATADOG_API_KEY=your_datadog_api_key \
ghcr.io/sadhiappan/logfleet-agent:latest
- Metrics are aggregated over 5-minute windows (99%+ reduction)
- Only business metrics are sent (no internal Vector metrics)
- Counters preferred over histograms (12x fewer data points)
Prometheus Metrics
A Prometheus exporter runs on port 9598 for local monitoring:
curl http://localhost:9598/metrics
Debugging
Enable Console Output
Uncomment the console sink in your Vector config:
[sinks.console]
type = "console"
inputs = ["all_logs"]
encoding.codec = "json"
Validate Configuration
vector validate --config /path/to/vector.toml
Check Vector API
curl http://localhost:8686/health
Environment Variables
| Variable | Default | Purpose |
|---|
AGENT_NAME | edge-agent-1 | Agent identifier in logs |
LOCATION_ID | unknown | Location identifier |
REGION | unknown | Geographic region |
ENVIRONMENT | production | Environment tag |
DATADOG_API_KEY | (none) | Enable Datadog metrics |
Full Default Configuration
The complete default Vector configuration is available at:
github.com/sadhiappan/logfleet/blob/main/agent/configs/vector/vector.toml
