Skip to content

POC: use OTELcol instead of vector #3334

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pavolloffay wants to merge 1 commit into
base: master
Choose a base branch
from
Open

POC: use OTELcol instead of vector #3334

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ...ects/-home-ploffay-projects-openshift-cluster-logging-operator/memory/MEMORY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- [CLO OTEL Migration](project_clo_otel_migration.md) — Migrating CLO from Vector to OTEL collector, multi-milestone proposal
17 changes: 17 additions & 0 deletions ...rojects-openshift-cluster-logging-operator/memory/project_clo_otel_migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
name: clo-otel-migration
description: Working on proposal to migrate CLO from Vector collector to OpenTelemetry collector - milestone 1 replaces Vector binary with OTEL collector
metadata:
type: project
---

User is working on a migration proposal to replace Vector with the OpenTelemetry collector in the Cluster Logging Operator (CLO).

**Why:** CLO currently uses Vector (Datadog) as the log collection/forwarding engine. The goal is to migrate to the OTEL collector to align with OpenTelemetry standards and potentially consolidate with the OpenTelemetry Operator.

**How to apply:** When working on CLO code or the migration proposal, understand that:
- Milestone 1: Replace Vector binary with OTEL collector in CLO-managed pods, generate OTEL collector YAML config instead of Vector TOML
- Milestone 2: Use OTEL collector CR instead of CLO-managed deployment
- Milestone 3: Full migration to OpenTelemetry Operator
- The ClusterLogForwarder CRD API stays the same in milestone 1
- Key files: api/observability/v1/ (CRD types), internal/generator/vector/ (Vector config generation)
8 changes: 8 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ export NAMESPACE?=openshift-logging
export LOKI_OPERATOR_CHANNEL?=stable-6.4

IMAGE_LOGGING_VECTOR?=quay.io/openshift-logging/vector:v0.54.0
IMAGE_OTEL_COLLECTOR?=ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.127.0
IMAGE_LOGFILEMETRICEXPORTER?=quay.io/openshift-logging/log-file-metric-exporter:latest
IMAGE_LOGGING_EVENTROUTER?=quay.io/openshift-logging/eventrouter:v0.5.0
IMAGE_TLS_SCANNER?=quay.io/openshift/tls-scanner:latest
Expand Down Expand Up @@ -129,6 +130,7 @@ run:
@mkdir -p $(CURDIR)/tmp
LOG_LEVEL=$(LOG_LEVEL) \
RELATED_IMAGE_VECTOR=$(IMAGE_LOGGING_VECTOR) \
RELATED_IMAGE_OTEL_COLLECTOR=$(IMAGE_OTEL_COLLECTOR) \
RELATED_IMAGE_LOG_FILE_METRIC_EXPORTER=$(IMAGE_LOGFILEMETRICEXPORTER) \
OPERATOR_NAME=$(OPERATOR_NAME) \
WATCH_NAMESPACE="" \
Expand Down Expand Up @@ -230,12 +232,14 @@ deploy-catalog:
test-env: ## Echo test environment, useful for running tests outside of the Makefile.
@echo \
RELATED_IMAGE_VECTOR=$(IMAGE_LOGGING_VECTOR) \
RELATED_IMAGE_OTEL_COLLECTOR=$(IMAGE_OTEL_COLLECTOR) \
RELATED_IMAGE_LOG_FILE_METRIC_EXPORTER=$(IMAGE_LOGFILEMETRICEXPORTER) \
IMAGE_TLS_SCANNER=$(IMAGE_TLS_SCANNER) \

.PHONY: test-functional
test-functional: test-functional-benchmarker-vector
RELATED_IMAGE_VECTOR=$(IMAGE_LOGGING_VECTOR) \
RELATED_IMAGE_OTEL_COLLECTOR=$(IMAGE_OTEL_COLLECTOR) \
RELATED_IMAGE_LOG_FILE_METRIC_EXPORTER=$(IMAGE_LOGFILEMETRICEXPORTER) \
go test -race \
./test/functional/... \
Expand All @@ -258,6 +262,7 @@ test-functional-benchmarker-vector: bin/functional-benchmarker
.PHONY: test-unit
test-unit: test-forwarder-generator test-unit-api
RELATED_IMAGE_VECTOR=$(IMAGE_LOGGING_VECTOR) \
RELATED_IMAGE_OTEL_COLLECTOR=$(IMAGE_OTEL_COLLECTOR) \
RELATED_IMAGE_LOG_FILE_METRIC_EXPORTER=$(IMAGE_LOGFILEMETRICEXPORTER) \
go test -coverprofile=test.cov -race ./api/... ./internal/... `go list ./test/... | grep -Ev 'test/(e2e|functional|framework|client|helpers)'`

Expand Down Expand Up @@ -319,13 +324,15 @@ apply: namespace $(OPERATOR_SDK) ## Install kustomized resources directly to the
.PHONY: test-upgrade
test-upgrade: $(JUNITREPORT)
RELATED_IMAGE_VECTOR=$(IMAGE_LOGGING_VECTOR) \
RELATED_IMAGE_OTEL_COLLECTOR=$(IMAGE_OTEL_COLLECTOR) \
RELATED_IMAGE_LOG_FILE_METRIC_EXPORTER=$(IMAGE_LOGFILEMETRICEXPORTER) \
IMAGE_LOGGING_EVENTROUTER=$(IMAGE_LOGGING_EVENTROUTER) \
exit 0

.PHONY: test-e2e
test-e2e: $(JUNITREPORT)
RELATED_IMAGE_VECTOR=$(IMAGE_LOGGING_VECTOR) \
RELATED_IMAGE_OTEL_COLLECTOR=$(IMAGE_OTEL_COLLECTOR) \
RELATED_IMAGE_LOG_FILE_METRIC_EXPORTER=$(IMAGE_LOGFILEMETRICEXPORTER) \
IMAGE_LOGGING_EVENTROUTER=$(IMAGE_LOGGING_EVENTROUTER) \
IMAGE_TLS_SCANNER=$(IMAGE_TLS_SCANNER) \
Expand All @@ -336,6 +343,7 @@ test-e2e-local: $(JUNITREPORT) deploy-image
LOG_LEVEL=3 \
LOKI_OPERATOR_CHANNEL=$(LOKI_OPERATOR_CHANNEL) \
RELATED_IMAGE_VECTOR=$(IMAGE_LOGGING_VECTOR) \
RELATED_IMAGE_OTEL_COLLECTOR=$(IMAGE_OTEL_COLLECTOR) \
RELATED_IMAGE_LOG_FILE_METRIC_EXPORTER=$(IMAGE_LOGFILEMETRICEXPORTER) \
IMAGE_LOGGING_EVENTROUTER=$(IMAGE_LOGGING_EVENTROUTER) \
IMAGE_TLS_SCANNER=$(IMAGE_TLS_SCANNER) \
Expand Down
28 changes: 20 additions & 8 deletions internal/collector/collector.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
configv1 "github.com/openshift/api/config/v1"
obs "github.com/openshift/cluster-logging-operator/api/observability/v1"
internalobs "github.com/openshift/cluster-logging-operator/internal/api/observability"
"github.com/openshift/cluster-logging-operator/internal/collector/otel"
"github.com/openshift/cluster-logging-operator/internal/collector/vector"
"github.com/openshift/cluster-logging-operator/internal/constants"
"github.com/openshift/cluster-logging-operator/internal/factory"
Expand Down Expand Up @@ -106,37 +107,48 @@ func New(confHash, clusterID string, collectorSpec *obs.CollectorSpec, secrets i
if collectorSpec == nil {
collectorSpec = &obs.CollectorSpec{}
}
factory := &Factory{

imageName := constants.VectorName
visit := Visitor(vector.CollectorVisitor)
podLabelVisitor := PodLabelVisitor(vector.PodLogExcludeLabel)

if annotations[constants.AnnotationCollectorType] == constants.OTELCollectorName {
imageName = constants.OTELCollectorName
visit = otel.CollectorVisitor
podLabelVisitor = otel.PodLogExcludeLabel
}

f := &Factory{
ClusterID: clusterID,
ConfigHash: confHash,
CollectorSpec: *collectorSpec,
ImageName: constants.VectorName,
Visit: vector.CollectorVisitor,
ImageName: imageName,
Visit: visit,
ConfigMaps: configMaps,
Secrets: secrets,
ForwarderSpec: forwarderSpec,
CommonLabelInitializer: func(o runtime.Object) {
runtime.SetCommonLabels(o, constants.VectorName, resNames.ForwarderName, constants.CollectorName)
runtime.SetCommonLabels(o, imageName, resNames.ForwarderName, constants.CollectorName)
},
ResourceNames: resNames,
PodLabelVisitor: vector.PodLogExcludeLabel,
PodLabelVisitor: podLabelVisitor,
isDaemonset: isDaemonset,
annotations: annotations,
}
return factory
return f
}

func (f *Factory) NewDaemonSet(namespace, name string, trustedCABundle *v1.ConfigMap, tlsProfileSpec configv1.TLSProfileSpec) *apps.DaemonSet {
podSpec := f.NewPodSpec(trustedCABundle, f.ForwarderSpec, f.ClusterID, tlsProfileSpec, namespace)
ds := factory.NewDaemonSet(namespace, name, name, constants.CollectorName, constants.VectorName, f.MaxUnavailable(), *podSpec, f.CommonLabelInitializer, f.PodLabelVisitor)
ds := factory.NewDaemonSet(namespace, name, name, constants.CollectorName, f.ImageName, f.MaxUnavailable(), *podSpec, f.CommonLabelInitializer, f.PodLabelVisitor)
ds.Spec.Template.Annotations[constants.AnnotationSecretHash] = f.Secrets.Hash64a()
ds.Spec.Template.Annotations[constants.AnnotationConfigMapHash] = f.ConfigMaps.Hash64a()
return ds
}

func (f *Factory) NewDeployment(namespace, name string, trustedCABundle *v1.ConfigMap, tlsProfileSpec configv1.TLSProfileSpec) *apps.Deployment {
podSpec := f.NewPodSpec(trustedCABundle, f.ForwarderSpec, f.ClusterID, tlsProfileSpec, namespace)
dpl := factory.NewDeployment(namespace, name, constants.CollectorName, constants.VectorName, 2, *podSpec, f.CommonLabelInitializer, f.PodLabelVisitor)
dpl := factory.NewDeployment(namespace, name, constants.CollectorName, f.ImageName, 2, *podSpec, f.CommonLabelInitializer, f.PodLabelVisitor)
dpl.Spec.Template.Annotations[constants.AnnotationSecretHash] = f.Secrets.Hash64a()
dpl.Spec.Template.Annotations[constants.AnnotationConfigMapHash] = f.ConfigMaps.Hash64a()
return dpl
Expand Down
20 changes: 16 additions & 4 deletions internal/collector/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,11 @@ package collector

import (
"fmt"

log "github.com/ViaQ/logerr/v2/log/static"
"github.com/openshift/cluster-logging-operator/internal/collector/otel"
"github.com/openshift/cluster-logging-operator/internal/collector/vector"
"github.com/openshift/cluster-logging-operator/internal/constants"
"github.com/openshift/cluster-logging-operator/internal/reconcile"
"github.com/openshift/cluster-logging-operator/internal/runtime"
"github.com/openshift/cluster-logging-operator/internal/utils"
Expand All @@ -12,16 +15,25 @@ import (
"sigs.k8s.io/controller-runtime/pkg/client"
)

func (f *Factory) configFileData(namespace, collectorConfig string) map[string]string {
if f.ImageName == constants.OTELCollectorName {
return map[string]string{
otel.ConfigFile: collectorConfig,
}
}
return map[string]string{
vector.ConfigFile: collectorConfig,
vector.RunVectorFile: fmt.Sprintf(vector.RunVectorScript, vector.GetDataPath(namespace, f.ResourceNames.ForwarderName)),
}
}

// ReconcileCollectorConfig reconciles a collector config specifically for the collector defined by the factory
func (f *Factory) ReconcileCollectorConfig(k8sClient client.Client, reader client.Reader, namespace, collectorConfig string, owner metav1.OwnerReference) error {
log.V(3).Info("Updating ConfigMap and Secrets")
configMap := runtime.NewConfigMap(
namespace,
f.ResourceNames.ConfigMap,
map[string]string{
vector.ConfigFile: collectorConfig,
vector.RunVectorFile: fmt.Sprintf(vector.RunVectorScript, vector.GetDataPath(namespace, f.ResourceNames.ForwarderName)),
},
f.configFileData(namespace, collectorConfig),
f.CommonLabelInitializer)

utils.AddOwnerRefToObject(configMap, owner)
Expand Down
20 changes: 20 additions & 0 deletions internal/collector/otel/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
package otel

import (
"path"

"github.com/openshift/cluster-logging-operator/internal/constants"
)

const (
ConfigFile = "config.yaml"
DefaultDataPath = "/var/lib/otelcol"
configPath = "/etc/otelcol"
)

func GetDataPath(namespace, forwarderName string) string {
if constants.OpenshiftNS == namespace && constants.SingletonName == forwarderName {
return DefaultDataPath
}
return path.Join(DefaultDataPath, namespace, forwarderName)
}
33 changes: 33 additions & 0 deletions internal/collector/otel/visitors.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
package otel

import (
"github.com/openshift/cluster-logging-operator/internal/collector/common"
"github.com/openshift/cluster-logging-operator/internal/factory"
"github.com/openshift/cluster-logging-operator/internal/runtime"
corev1 "k8s.io/api/core/v1"
)

func CollectorVisitor(collectorContainer *corev1.Container, podSpec *corev1.PodSpec, resNames *factory.ForwarderResourceNames, namespace, logLevel string) {
collectorContainer.Env = append(collectorContainer.Env,
corev1.EnvVar{Name: "OTEL_LOG_LEVEL", Value: logLevel},
)

dataPath := GetDataPath(namespace, resNames.ForwarderName)
collectorContainer.VolumeMounts = append(collectorContainer.VolumeMounts,
corev1.VolumeMount{Name: common.ConfigVolumeName, ReadOnly: true, MountPath: configPath},
corev1.VolumeMount{Name: common.DataDir, ReadOnly: false, MountPath: dataPath},
)

collectorContainer.Command = []string{"/otelcol-contrib"}
collectorContainer.Args = []string{"--config=" + configPath + "/" + ConfigFile}

podSpec.Volumes = append(podSpec.Volumes,
corev1.Volume{Name: common.ConfigVolumeName, VolumeSource: corev1.VolumeSource{ConfigMap: &corev1.ConfigMapVolumeSource{LocalObjectReference: corev1.LocalObjectReference{Name: resNames.ConfigMap}}}},
corev1.Volume{Name: common.DataDir, VolumeSource: corev1.VolumeSource{HostPath: &corev1.HostPathVolumeSource{Path: dataPath}}},
)
}

func PodLogExcludeLabel(o runtime.Object) {
// OTEL collector's filelog receiver uses exclude patterns in config rather than pod labels.
// No pod label needed for self-exclusion.
}
3 changes: 3 additions & 0 deletions internal/constants/annotations.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,7 @@ const (
// AnnotationMaxUnavailable (Deprecated) configures the maximum number of DaemonSet pods that can be unavailable during a rolling update.
// This can be an absolute number (e.g., 1) or a percentage (e.g., 10%). Default is 100%.
AnnotationMaxUnavailable = "observability.openshift.io/max-unavailable-rollout"

// AnnotationCollectorType selects the collector implementation. Supported values: "vector" (default), "otelcol".
AnnotationCollectorType = "logging.openshift.io/dev-preview-collector-type"
)
4 changes: 3 additions & 1 deletion internal/constants/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ const (
TrustedCABundleMountDir = "/etc/pki/ca-trust/extracted/pem/"
ElasticsearchName = "elasticsearch"
VectorName = "vector"
OTELCollectorName = "otelcol"
KibanaName = "kibana"
LogfilesmetricexporterName = "logfilesmetricexporter"
LogfilesmetricexporterPort = int32(2112)
Expand All @@ -63,7 +64,8 @@ const (
CollectorServiceAccountName = "logcollector"
CollectorTrustedCAName = "collector-trusted-ca-bundle"

VectorImageEnvVar = "RELATED_IMAGE_VECTOR"
VectorImageEnvVar = "RELATED_IMAGE_VECTOR"
OTELCollectorImageEnvVar = "RELATED_IMAGE_OTEL_COLLECTOR"
LogfilesmetricImageEnvVar = "RELATED_IMAGE_LOG_FILE_METRIC_EXPORTER"

ContainerLogDir = "/var/log/containers"
Expand Down
39 changes: 39 additions & 0 deletions internal/controller/observability/collector.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,11 @@ func GenerateConfig(k8Client client.Client, clf obs.ClusterLogForwarder, resourc
tlsProfile, _ := tls.FetchAPIServerTlsProfile(k8Client)
op[framework.ClusterTLSProfileSpec] = tls.GetClusterTLSProfileSpec(tlsProfile)
EvaluateAnnotationsForEnabledCapabilities(clf.Annotations, op)

if clf.Annotations[constants.AnnotationCollectorType] == constants.OTELCollectorName {
return generateOTELConfig(clf.Spec)
}

g := forwardergenerator.New()
generatedConfig, err := g.GenerateConf(secrets, clf.Spec, clf.Namespace, clf.Name, resourceNames, op)

Expand All @@ -191,6 +196,40 @@ func GenerateConfig(k8Client client.Client, clf obs.ClusterLogForwarder, resourc
return generatedConfig, err
}

// generateOTELConfig returns a minimal OTEL collector configuration.
// TODO: Replace with full config generation from CLF spec.
func generateOTELConfig(_ obs.ClusterLogForwarderSpec) (string, error) {
return `receivers:
filelog:
include:
- /var/log/pods/*/*/*.log
exclude:
- /var/log/pods/openshift-logging_collector-*/*/*.log
start_at: end
include_file_path: true
include_file_name: false
operators:
- type: container
id: container-parser

processors:
batch:
send_batch_size: 8192
timeout: 2s

exporters:
debug:
verbosity: basic

service:
pipelines:
logs:
receivers: [filelog]
processors: [batch]
exporters: [debug]
`, nil
}

Comment on lines +199 to +232

@coderabbitai coderabbitai Bot Jul 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Map relevant files and inspect the collector config generation plus the functional test namespace setup.
git ls-files | rg '^(internal/controller/observability/collector\.go|test/framework/functional/framework\.go|test/framework/functional/otel/deploy\.go|test/framework/functional/.*\.go)$'

echo '--- collector.go outline ---'
ast-grep outline internal/controller/observability/collector.go --view expanded || true

echo '--- framework.go outline ---'
ast-grep outline test/framework/functional/framework.go --view expanded || true

echo '--- otel deploy.go outline ---'
ast-grep outline test/framework/functional/otel/deploy.go --view expanded || true

echo '--- search for collector namespace/exclude/start_at references ---'
rg -n --context 3 'openshift-logging_collector-|start_at: end|generateOTELConfig|ClusterLogForwarderSpec|Debug exporter|collector' internal/controller/observability/collector.go test/framework/functional test/framework -g '!**/vendor/**'

Repository: openshift/cluster-logging-operator

Length of output: 50390

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Read the smallest relevant slices around the config generator and functional namespace creation.
sed -n '180,260p' internal/controller/observability/collector.go

echo '--- framework.go relevant slice ---'
rg -n 'type .*Namespace|t\.NS|Namespace|CreateNamespace|openshift-logging' test/framework/functional/framework.go -A 4 -B 4

echo '--- otel deploy.go relevant slice ---'
sed -n '1,240p' test/framework/functional/otel/deploy.go

Repository: openshift/cluster-logging-operator

Length of output: 10626

Make the self-exclusion namespace dynamic. The exclude path only matches openshift-logging, but the functional framework deploys the collector in t.NS.Name, so its own pod logs won’t be excluded there. Derive this from the forwarder namespace instead of hardcoding it.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/controller/observability/collector.go` around lines 199 - 232, The
OTEL collector self-exclusion is hardcoded to openshift-logging, so collector
pods in other namespaces are not excluded. Update generateOTELConfig to derive
the exclude path from the ClusterLogForwarder spec’s namespace (the namespace
used by the forwarder/collector) instead of a fixed string, and wire that value
into the returned config so the collector excludes its own pod logs regardless
of deployment namespace.

// EvaluateAnnotationsForEnabledCapabilities populates generator options with capabilities enabled by the ClusterLogForwarder
func EvaluateAnnotationsForEnabledCapabilities(annotations map[string]string, options framework.Options) {
if annotations == nil {
Expand Down
1 change: 1 addition & 0 deletions internal/utils/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ var (
// COMPONENT_IMAGES are keys based on the "container name" + "-{image,version}"
var COMPONENT_IMAGES = map[string]string{
constants.VectorName: constants.VectorImageEnvVar,
constants.OTELCollectorName: constants.OTELCollectorImageEnvVar,
constants.LogfilesmetricexporterName: constants.LogfilesmetricImageEnvVar,
}

Expand Down
Loading
Loading