2 min to read

What is Loki?

A comprehensive guide to Grafana Loki and its components

Featured image

Image Reference


Overview

Let’s explore Loki, a log aggregation system developed by Grafana Labs, designed specifically for cloud-native environments.


What is Loki?

Loki is a horizontally scalable, highly available log aggregation system designed by Grafana Labs. Unlike traditional logging systems, Loki indexes metadata (labels) rather than log content, making it highly efficient and cost-effective for cloud-native environments.

Key Features of Loki
  1. Easy Integration
    • Supports various log formats and sources
    • Simple setup and configuration
  2. Scalable Storage
    • Petabyte-scale capability
    • 100% persistence with object storage
  3. Advanced Functionality
    • Metrics generation from logs
    • Real-time log tailing
    • Native integration with Prometheus and Grafana


Loki Components

Core Components

1. Promtail
  • Log collection agent
  • Runs on each node as a DaemonSet
  • Collects and forwards logs to Loki
2. Loki Server
  • Receives and processes log data
  • Indexes metadata while keeping log content unindexed
3. Distributor
  • Distributes incoming logs across ingesters
  • Handles load balancing and replication
4. Ingester
  • Writes log data to storage
  • Manages in-memory caching
  • Handles data compression
5. Querier
  • Processes log queries
  • Handles both real-time and historical data

Optional Components

Additional Services
  1. Query Frontend
    • Optimizes and caches queries
    • Reduces load on queriers
  2. Compactor
    • Optimizes stored data
    • Improves query performance
  3. Storage Backends
    • Supports multiple storage options
    • Including file system, S3, GCS


How Promtail Works

Installation and Configuration

Promtail is typically deployed as a DaemonSet in Kubernetes clusters, ensuring log collection from all nodes. Configuration includes:

Log Discovery and Collection Process

Collection Workflow
  1. File Discovery
    • Automatically finds log files in configured paths
    • Typically monitors /var/log/pods/ in Kubernetes
  2. Metadata Handling
    • Extracts metadata from Kubernetes API
    • Adds labels like pod name, namespace, node name
  3. Log Transmission
    • Sends logs to Loki server
    • Supports local buffering for reliability

Reliability Features


References