Troubleshooting

This guide covers common issues and their solutions. For architecture-level failure handling, see Resilience.

Quick Diagnostics

Run this command on your edge device to get a quick health check:

# Check all LogFleet containers
docker-compose ps

# View recent logs
docker-compose logs --tail=50

# Check Vector health
curl -s http://localhost:8686/health | jq

Common Issues

Agent Not Appearing in Dashboard

Symptoms: Edge agent deployed but not visible in LogFleet dashboard. Diagnostic steps:

# 1. Verify agent is running
docker ps | grep logfleet

# 2. Check registration logs
docker logs logfleet-agent 2>&1 | grep -i "register"

# 3. Test API connectivity
curl -I https://api.logfleet.io/health

Common causes:

Cause	Solution
Invalid API key	Regenerate key with `edge:register` permission
Network blocked	Ensure outbound HTTPS (443) is allowed
DNS resolution	Verify `api.logfleet.io` resolves correctly
Clock skew	Sync NTP - JWT validation requires accurate time

Fix:

# Verify API key is set
echo $LOGFLEET_API_KEY

# Test registration manually
curl -X POST https://api.logfleet.io/api/v1/edge/register \
  -H "X-API-Key: $LOGFLEET_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "test-agent", "location": "test"}'

Logs Not Being Collected

Symptoms: Agent running but no logs appearing in local Loki. Diagnostic steps:

# 1. Check Vector is receiving data
curl -s http://localhost:8686/metrics | grep "events_in_total"

# 2. Verify Vector config
docker exec logfleet-vector vector validate /etc/vector/vector.yaml

# 3. Check Loki ingestion
curl -s "http://localhost:3100/loki/api/v1/labels" | jq

Common causes:

Cause	Solution
Wrong log path	Update `file` source paths in Vector config
Permission denied	Run Vector with appropriate user/group
Loki not ready	Wait for Loki to initialize (check `loki_ready` metric)
Log format mismatch	Verify transform regex matches your log format

Fix for file permissions:

# Check log file permissions
ls -la /var/log/your-app/

# Add Vector user to log group
usermod -aG syslog vector

Metrics Not Reaching Cloud

Symptoms: Local logs work but metrics don’t appear in cloud dashboards. Diagnostic steps:

# 1. Check Vector sink status
curl -s http://localhost:8686/metrics | grep "sink"

# 2. Verify outbound connectivity
curl -I https://api.logfleet.io/api/v1/edge/metrics

# 3. Check for backpressure
docker logs logfleet-vector 2>&1 | grep -i "backpressure\|retry"

Common causes:

Cause	Solution
Rate limited	Check Rate Limits quotas
API key expired	Regenerate API key
Payload too large	Reduce batch size in Vector config
Network intermittent	Metrics buffer locally and retry

Fix rate limiting:

# vector.yaml - reduce batch size
sinks:
  logfleet_metrics:
    type: http
    batch:
      max_events: 100  # Reduce from default
      timeout_secs: 5

High Memory Usage

Symptoms: Loki or Vector consuming excessive memory. Diagnostic steps:

# Check container memory
docker stats --no-stream

# Loki memory breakdown
curl -s http://localhost:3100/metrics | grep "loki_ingester_memory"

# Vector memory
curl -s http://localhost:8686/metrics | grep "memory"

Common causes:

Cause	Solution
High log volume	Increase memory limits or add sampling
Retention too long	Reduce `retention_period` in Loki config
Too many streams	Reduce label cardinality
Buffer overflow	Increase disk buffer, reduce memory buffer

Fix - reduce Loki memory:

# loki-config.yaml
limits_config:
  ingestion_rate_mb: 4
  ingestion_burst_size_mb: 8
  max_streams_per_user: 1000

chunk_store_config:
  max_look_back_period: 168h  # 7 days instead of 30

Disk Filling Up

Symptoms: Edge device running out of storage. Diagnostic steps:

# Check disk usage
df -h

# Find large Loki chunks
du -sh /var/lib/loki/*

# Check retention is working
curl -s http://localhost:3100/metrics | grep "compactor"

Common causes:

Cause	Solution
Compactor not running	Verify Loki compactor is enabled
Retention too long	Reduce `retention_period`
High ingestion rate	Add sampling or filtering
WAL not flushing	Check Loki WAL health

Fix - force compaction:

# Trigger manual compaction
curl -X POST http://localhost:3100/compactor/ring/forget

# Or restart with compaction
docker-compose restart loki

Log Streaming Not Working

Symptoms: Cannot stream logs on-demand from cloud. Diagnostic steps:

# 1. Check streaming is enabled
echo $STREAM_ENABLED

# 2. Verify WebSocket connectivity
curl -I https://api.logfleet.io/api/v1/streaming/ws

# 3. Check agent streaming status
curl -s http://localhost:8080/status | jq '.streaming'

Common causes:

Cause	Solution
Streaming disabled	Enable via dashboard or API
Firewall blocking WS	Allow WebSocket upgrade on port 443
Agent offline	Check agent heartbeat status
Session expired	Streaming auto-disables after timeout

Fix - enable streaming:

# Enable streaming via API
curl -X POST https://api.logfleet.io/api/v1/streaming/start \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"agent_id": "your-agent-id", "duration_minutes": 30}'

Debug Mode

Enable verbose logging for deeper diagnostics:

# Vector debug mode
docker run -e VECTOR_LOG=debug logfleet/vector

# Loki debug mode
docker run -e LOKI_LOG_LEVEL=debug grafana/loki

Getting Help

If these steps don’t resolve your issue:

Check the logs - Most issues are visible in container logs
GitHub Issues - Search or file at github.com/sadhiappan/logfleet
Community - Join our Discord for real-time help

When reporting issues, include:

LogFleet version (docker images | grep logfleet)
Edge device OS and architecture
Relevant container logs
Output from diagnostic commands above

Deployment

Operations

Support

Quick Diagnostics

Common Issues

Agent Not Appearing in Dashboard

Logs Not Being Collected

Metrics Not Reaching Cloud

High Memory Usage

Disk Filling Up

Log Streaming Not Working

Debug Mode

Getting Help

​Quick Diagnostics

​Common Issues

​Agent Not Appearing in Dashboard

​Logs Not Being Collected

​Metrics Not Reaching Cloud

​High Memory Usage

​Disk Filling Up

​Log Streaming Not Working

​Debug Mode

​Getting Help

Quick Diagnostics

Common Issues

Agent Not Appearing in Dashboard

Logs Not Being Collected

Metrics Not Reaching Cloud

High Memory Usage

Disk Filling Up

Log Streaming Not Working

Debug Mode

Getting Help