Logging Architecture

Architecture Overview

ZCP collects logs from all registered Data Plane Clusters and stores them centrally in OpenSearch within the Control Plane Cluster. The platform enables cluster-wide log queries through OpenSearch Dashboards and provides role-based access for log visualization per project.

The logging service consists of four key components: Fluent Bit, Fluentd, OpenSearch, and OpenSearch Dashboards, deployed across both Data Plane Clusters and the Control Plane Cluster:

• Fluent Bit runs on all worker nodes in the Data Plane Cluster as a DaemonSet, collecting logs from each node.
• Fluentd processes and forwards collected logs to Elasticsearch.

In the Control Plane Cluster, OpenSearch is deployed on three dedicated worker nodes to ensure high availability and efficient log storage. These nodes store and manage logs sent from Fluent Bit and Fluentd within the Data Plane Cluster. An additional Fluentd instance in the Control Plane Cluster organizes the log pipeline between clusters and performs pre- and post-processing of logs.

Log Query & Visualization

Logs stored in OpenSearch can be queried and visualized using OpenSearch Dashboards.

To centrally manage logs from all Data Plane Clusters, OpenSearch is utilized. This implementation supports:

• OIDC authentication for secure access.
• Index Policy management for log retention and lifecycle control.
• Role and permission management for security compliance.

Components and Roles

Control Plane Components

Component	Roles
OpenSearch Dashboards	Visualizations for data stored in OpenSearch
OpenSearch	Store collected logs and provide a search and analytics engine for them
Fluentd	Collect logs of user application containers and all system components running on each node

Data Plane Components

Component	Roles
Fluentd	Preliminary processing of logs collected by fluent-bit and passing them to fluentd in Control Plane
Fluent Bit	Collect logs of user application containers and all system components running on each node