Log Streaming

LogFleet provides on-demand log streaming for troubleshooting. Instead of always shipping logs to the cloud, you stream them only when needed.

How It Works

Normal Operation:
Edge Agent → Metrics Only → Cloud

Streaming Enabled:
Edge Agent → Metrics + Logs → Cloud (temporary)

Streaming sessions have automatic timeouts to prevent forgotten streams from running indefinitely.

Starting a Stream

Via API

curl -X POST https://api.logfleet.io/api/v1/stream/start \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "timeout_seconds": 300,
    "filters": {
      "level": ["ERROR", "WARN"],
      "source": "nginx"
    }
  }'

Response:

{
  "session": {
    "id": "sess_abc123",
    "agent_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "status": "active",
    "started_at": "2024-01-15T10:00:00Z",
    "expires_at": "2024-01-15T10:05:00Z",
    "websocket_url": "wss://api.logfleet.io/api/v1/stream/ws/sess_abc123"
  }
}

Connecting to WebSocket

const ws = new WebSocket(
  `wss://api.logfleet.io/api/v1/stream/ws/sess_abc123?token=${token}`
);

ws.onmessage = (event) => {
  const log = JSON.parse(event.data);
  console.log(`[${log.level}] ${log.source}: ${log.message}`);
};

Filtering Logs

Apply filters to reduce noise and focus on relevant logs:

Level Filter

{
  "filters": {
    "level": ["ERROR", "WARN", "FATAL"]
  }
}

Source Filter

{
  "filters": {
    "source": "nginx"
  }
}

Pattern Filter

{
  "filters": {
    "pattern": "user_id=12345"
  }
}

Combined Filters

{
  "filters": {
    "level": ["ERROR"],
    "source": "api-gateway",
    "pattern": "timeout"
  }
}

Session Timeouts

All streaming sessions have mandatory timeouts:

Setting	Value
Default timeout	5 minutes
Maximum timeout	1 hour
Warning before end	30 seconds

Timeout Warning

30 seconds before a session ends, you’ll receive a control message:

{
  "type": "control",
  "event": "session_ending",
  "reason": "timeout",
  "seconds_remaining": 30
}

Extending a Session

To continue streaming, start a new session before the current one ends:

curl -X POST https://api.logfleet.io/api/v1/stream/start \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "timeout_seconds": 300
  }'

Stopping a Stream

Manually stop streaming when done:

curl -X POST https://api.logfleet.io/api/v1/stream/stop \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "sess_abc123"
  }'

Viewing Active Sessions

List all active streaming sessions:

curl -X GET https://api.logfleet.io/api/v1/stream/sessions \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "sessions": [
    {
      "id": "sess_abc123",
      "agent_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "agent_name": "edge-location-01",
      "status": "active",
      "started_at": "2024-01-15T10:00:00Z",
      "expires_at": "2024-01-15T10:05:00Z"
    }
  ],
  "total": 1
}

Log Message Format

Each log message contains:

{
  "type": "log",
  "timestamp": "2024-01-15T10:00:00.123Z",
  "level": "ERROR",
  "source": "nginx",
  "message": "upstream timed out (110: Connection timed out)",
  "metadata": {
    "host": "edge-location-01",
    "file": "/var/log/nginx/error.log",
    "line": 12345
  }
}

Use Cases

Incident Response

Stream logs from affected agents during an outage to identify root cause.

Debugging

Watch real-time logs while reproducing an issue in production.

Deployment Verification

Monitor logs during deployments to catch errors immediately.

Security Investigation

Stream logs to investigate suspicious activity or security alerts.

Best Practices

Use Filters

Always apply filters to reduce bandwidth and focus on relevant logs. Streaming everything can overwhelm both the network and your analysis.

Set Appropriate Timeouts

Use the shortest timeout that meets your needs. 5 minutes is usually sufficient for quick debugging sessions.

Stop When Done

Manually stop sessions when you’re done troubleshooting. Don’t rely on timeouts for cleanup.

Monitor Usage

Keep an eye on active sessions across your organization to avoid excessive bandwidth usage.

Bandwidth Considerations

Streaming full logs consumes significantly more bandwidth than metrics:

Mode	Bandwidth (est.)
Metrics only	~10 KB/hour
Streaming (filtered)	~1-10 MB/hour
Streaming (all logs)	~100+ MB/hour

Use streaming sparingly and with appropriate filters.

Client Libraries

JavaScript/TypeScript

import { LogFleetClient } from '@logfleet/client';

const client = new LogFleetClient({ token: 'your-jwt-token' });

const stream = await client.startStream({
  agentId: 'agent-uuid',
  timeout: 300,
  filters: { level: ['ERROR'] }
});

stream.on('log', (log) => {
  console.log(`[${log.level}] ${log.message}`);
});

stream.on('ending', (secondsRemaining) => {
  console.log(`Stream ending in ${secondsRemaining}s`);
});

// When done
await stream.stop();

Python

from logfleet import Client

client = Client(token="your-jwt-token")

with client.stream(
    agent_id="agent-uuid",
    timeout=300,
    filters={"level": ["ERROR"]}
) as stream:
    for log in stream:
        print(f"[{log.level}] {log.message}")

Client libraries are coming soon. See the WebSocket documentation for implementing streaming with raw WebSockets.

Deployment

Operations

Support

How It Works

Starting a Stream

Via API

Connecting to WebSocket

Filtering Logs

Level Filter

Source Filter

Pattern Filter

Combined Filters

Session Timeouts

Timeout Warning

Extending a Session

Stopping a Stream

Viewing Active Sessions

Log Message Format

Use Cases

Incident Response

Debugging

Deployment Verification

Security Investigation

Best Practices

Bandwidth Considerations

Client Libraries

JavaScript/TypeScript

Python

​How It Works

​Starting a Stream

​Via API

​Connecting to WebSocket

​Filtering Logs

​Level Filter

​Source Filter

​Pattern Filter

​Combined Filters

​Session Timeouts

​Timeout Warning

​Extending a Session

​Stopping a Stream

​Viewing Active Sessions

​Log Message Format

​Use Cases

Incident Response

Debugging

Deployment Verification

Security Investigation

​Best Practices

​Bandwidth Considerations

​Client Libraries

​JavaScript/TypeScript

​Python

How It Works

Starting a Stream

Via API

Connecting to WebSocket

Filtering Logs

Level Filter

Source Filter

Pattern Filter

Combined Filters

Session Timeouts

Timeout Warning

Extending a Session

Stopping a Stream

Viewing Active Sessions

Log Message Format

Use Cases

Best Practices

Bandwidth Considerations

Client Libraries

JavaScript/TypeScript

Python