The OpenTelemetry Operator: instrumenting Kubernetes without SDKs

Every language-specific APM agent in your cluster is a ticket to another team: import the SDK and configure the exporter, then cut a release and repeat for every service in every language. The OpenTelemetry Operator, a Kubernetes controller that extends the Kubernetes API with custom resources, removes that coordination loop. It registers a mutating admission webhook, a hook the API server calls to modify a pod spec as the pod is created, and that webhook injects auto-instrumentation into pods at admission time, which means you can instrument Kubernetes without SDK imports, code changes, or a dev-team dependency.

This guide covers installing the OpenTelemetry Operator, configuring automatic instrumentation, verifying telemetry, and troubleshooting injection failures. It also examines manual instrumentation trade-offs, managing configuration with GitOps, and how eBPF and service mesh telemetry complement OpenTelemetry.

What It Means to Instrument Kubernetes Without an SDK

Zero-code instrumentation lets you collect telemetry from supported applications without modifying their source code or rebuilding container images. The OpenTelemetry Operator uses a mutating admission webhook to inject instrumentation into newly created pods, allowing language-specific agents to collect telemetry automatically.

To use the Operator, install it with webhook TLS configured and create an Instrumentation resource pointing to a reachable OTLP endpoint. Add a language-specific annotation to your workload’s pod template or namespace, then roll out new pods. Verify that instrumentation was injected and telemetry reaches your Collector or backend.

Zero-code instrumentation captures operations from supported libraries and frameworks, including HTTP requests and database calls. However, application-specific business logic isn’t automatically instrumented. You’ll still need manual SDK instrumentation for custom business spans, attributes, and events.

How the OpenTelemetry Operator Injects Agents

A Kubernetes operator extends the Kubernetes API with custom resources and runs controllers that reconcile the desired state declared in those resources with the actual cluster state for a particular application or service. The OpenTelemetry Operator applies this model through controllers that manage Collector resources (OpenTelemetryCollector). It also runs a mutating admission webhook that adds automatic instrumentation to matching pods when they are created, by adding an init container and environment variables to the pod spec.

Mutating Admission Webhook and Init Containers

The operator registers a MutatingWebhookConfiguration whose rules send matching Pod creation requests to its admission webhook at the /mutate-v1-pod endpoint. The webhook checks the instrumentation annotations and modifies eligible pod specs before the pods are created. The injection flow runs in three steps:

1. You add an injection annotation, such as instrumentation.opentelemetry.io/inject-java: "opentelemetry/default-instrumentation", to the pod template of a Deployment. This selects the Instrumentation resource in the opentelemetry namespace.When the pod is created, the API server sends the admission request to the operator’s webhook.
2. The webhook mutates the pod spec: it adds an init container that copies the language agent into a shared volume. For Java, the webhook adds an init container that copies the Java agent into a shared volume. It mounts that volume in the application container and sets JAVA_TOOL_OPTIONS so the JVM loads the agent at startup.
3. The application container starts, the runtime picks up the agent from the injected env var, and spans flow to the exporter endpoint that the Instrumentation resource defines.

The Pod-injection webhook defaults to failurePolicy: Ignore, while the Operator’s other admission webhooks can use Fail. When the Pod-injection webhook can’t be called, matching pods are admitted and start without instrumentation, because injection happens only during admission. The Operator does not reinstrument those pods after the webhook recovers, so recreate the affected pods, for example with a rollout, to trigger injection.

The Instrumentation Custom Resource

An Instrumentation custom resource defines exporter settings, sampling, propagation, and language-specific options, but it does not configure any workload on its own. Workloads opt in through injection annotations, and the annotation value selects the resource. A value of "true" selects the default-named resource in the workload’s own namespace, while a qualified value such as opentelemetry/default-instrumentation selects the resource defined below from another namespace:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default-instrumentation
  namespace: opentelemetry
spec:
  exporter:
    endpoint: http://otel-collector.opentelemetry.svc.cluster.local:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  nodejs:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://otel-collector.opentelemetry.svc.cluster.local:4317

Create the Instrumentation resource before starting annotated application pods. The Operator injects agents during pod creation, so existing pods need a rollout to pick up the configuration. The injected agent volume has a default size limit of 200Mi. You can adjust volume limits and init-container resource requirements through the language-specific settings in the Instrumentation resource.

cert-manager and the Webhook TLS Dependency

The Kubernetes API server communicates with the Operator’s webhook over TLS, so the webhook needs a trusted serving certificate and a matching caBundle. The Operator Helm chart uses cert-manager by default to manage this certificate. Install cert-manager before the Operator when using this configuration. Operator v0.162.0 supports cert-manager v1.

You can also use Helm-generated self-signed certificates by setting admissionWebhooks.certManager.enabled=false and admissionWebhooks.autoGenerateCert.enabled=true. The two approaches have different maintenance requirements:

• cert-manager (default): Issues and renews webhook certificates automatically. Install cert-manager before the Operator and ensure that certificate issuance and CA injection remain healthy.
• Helm auto-generated: Removes the cert-manager dependency but requires you to manage certificate renewal. With the default chart settings, certificates are valid for 365 days and regenerated on helm upgrade because recreate is enabled. Repeated GitOps rendering can generate different CA material and cause webhook TLS failures, including x509: certificate signed by unknown authority. For production, use cert-manager or another approach that renews certificates and maintains consistent webhook trust.

Installing the Operator and Collector

The following procedure installs the OpenTelemetry Operator, deploys an Operator-managed Collector, and enables automatic instrumentation for a Java workload. It uses Operator Helm chart 0.128.0 (Operator 0.162.0) and cert-manager v1.21.1. Before installing, check your Kubernetes version against the compatibility matrix.

The Collector provides an OTLP endpoint for instrumented applications, but it’s optional. You can export to another compatible OTLP destination instead. The example uses cert-manager for webhook TLS; the Helm-generated certificate alternative is described in the preceding section.

First, cert-manager:

helm install \
  cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.21.1 \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

Second, the Operator:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
  --namespace opentelemetry-operator-system \
  --create-namespace \
  --version 0.128.0

Third, create an Operator-managed Collector. The following OpenTelemetryCollector resource deploys a Collector that accepts OTLP/gRPC on port 4317 and OTLP/HTTP on port 4318. It uses a debug exporter to confirm that traces arrive during testing.

kubectl create namespace opentelemetry --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f - <<'EOF'
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: opentelemetry
spec:
  mode: deployment
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch: {}
    exporters:
      debug:
        verbosity: basic
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [debug]
EOF

The Operator creates a Collector Deployment and a Service named otel-collector in the opentelemetry namespace. This matches the otel-collector.opentelemetry.svc.cluster.local address used in the earlier Instrumentation example. The Collector receives traffic on ports 4317 and 4318 and logs a summary of received traces through its debug exporter. For production, replace or supplement the debug exporter with one that sends telemetry to your observability backend.

Fourth, verify that the Operator and its managed Collector are ready:

kubectl get pods -n opentelemetry-operator-system
kubectl get opentelemetrycollectors -n opentelemetry
kubectl get deployments -n opentelemetry
kubectl get pods -n opentelemetry
kubectl get svc -n opentelemetry
kubectl get mutatingwebhookconfigurations

Check four things:

• The OpenTelemetry Operator pod is Ready.
• The OpenTelemetryCollector resource named otel exists, and its managed Collector Deployment is Ready.
• The otel-collector Service exists in the opentelemetry namespace and exposes the configured OTLP ports.
• The Operator’s mutating admission webhook is registered.

Unexpected webhook configurations can reference missing Services and disrupt admission. Check their ownership labels, Service references, and selectors before changing or removing them. Ready Operator and Collector pods don’t confirm successful telemetry export, so verify delivery after enabling instrumentation.

Fifth, save the Instrumentation YAML shown earlier as instrumentation.yaml and apply it:

kubectl apply -f instrumentation.yaml
kubectl get instrumentation default-instrumentation -n opentelemetry

Sixth, annotate an existing Java Deployment. Replace the placeholders with your workload’s name and namespace:

kubectl patch deployment/<deployment-name> \
  -n <app-namespace> --type merge \
  -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"opentelemetry/default-instrumentation"}}}}}'

kubectl rollout status deployment/<deployment-name> \
  -n <app-namespace>

The patch changes the pod template, triggering a rollout. The Operator injects the Java agent into newly created pods.

Finally, send test requests to an instrumented application endpoint and verify injection and telemetry delivery:

kubectl get pods -n <app-namespace>
kubectl describe pod <new-pod-name> -n <app-namespace>
kubectl logs deployment/otel-collector -n opentelemetry

Check the new pod for the instrumentation init container and injected environment variables. The Collector’s debug exporter should report received spans. Because the example samples 25% of new traces, send several requests before concluding that telemetry is missing.

Collector Deployment Topology

The OpenTelemetry Operator supports four Collector deployment modes: deployment, daemonset, statefulset, and sidecar, with deployment as the default for an OpenTelemetryCollector resource (deployment modes). You can also deploy the Collector independently using the standalone Helm chart (open-telemetry/opentelemetry-collector). This chart supports deployment, daemonset, and statefulset, but not sidecar. Unlike the Operator resource, the standalone chart requires you to set mode explicitly.

The exporter endpoint in an Instrumentation resource depends on two things: where the Collector runs, and which OTLP protocol the instrumented agent uses. The table below shows the endpoint pattern for each mode:

If you enable tail sampling across multiple Collector gateways, all spans belonging to the same trace must reach the same instance. You can use a load-balancing exporter configured to route by trace ID.

OTLP/gRPC uses port 4317 by default, while OTLP/HTTP protobuf uses 4318. These ports are configurable, so the endpoint must match the Collector’s receiver configuration. Java, .NET, Go, Python, and Deno use OTLP/HTTP by default, while Node.js defaults to OTLP/gRPC. Each agent must use a protocol and endpoint supported by the receiving Collector.

The default-instrumentation resource sends telemetry from Java, .NET, Go, and Python agents to the Collector’s OTLP/HTTP receiver on port 4318. Its nodejs.env override directs Node.js telemetry to the OTLP/gRPC receiver on port 4317. To use OTLP/HTTP for Node.js instead, set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf and remove the Node.js-specific endpoint override so it inherits the shared 4318 endpoint.

RBAC for the Operator

The OpenTelemetry Operator and its managed Collectors use separate Kubernetes ServiceAccounts with different RBAC requirements. The Operator’s ServiceAccount needs permissions to manage Collector workloads, Services, ConfigMaps, ServiceAccounts, opentelemetry.io custom resources, and leader-election leases. The Operator Helm chart’s ClusterRole defines these permissions, including additional rules controlled by chart settings.

The Collector’s ServiceAccount needs Kubernetes API permissions only when its configured components query Kubernetes resources. For example, the Kubernetes Attributes processor retrieves pod, namespace, and ReplicaSet metadata, requiring read access to those resources. Add the processor to the OpenTelemetryCollector configuration and grant its ServiceAccount the necessary permissions. Other Kubernetes receivers, including those collecting cluster metrics or events, have their own RBAC requirements. The OTLP receiver and debug exporter used earlier don’t require these additional Kubernetes API permissions.

The manager.createRbacPermissions setting is disabled by default. When enabled, the Helm chart grants the Operator additional permissions to create and manage ClusterRole and ClusterRoleBinding resources for its Collectors. Leave it disabled when you manage Collector RBAC separately, so the Operator doesn’t receive unnecessary cluster-wide permissions.

Upgrading and Uninstalling the Operator

Before upgrading, review the release notes and the UPGRADING.md upgrade instructions in the chart documentation, and check Kubernetes compatibility for the target release in the compatibility matrix. Pin the target chart version with --version so the upgrade is repeatable. Next, back up your existing Instrumentation and OpenTelemetryCollector resources:

kubectl get instrumentations,opentelemetrycollectors -A -o yaml > otel-backup.yaml

Older chart installations can require a one-time CRD ownership migration before Helm can manage existing CRDs, so check the upgrade instructions for whether yours does. Changes to instrumentation settings don’t replace agents in running application pods, so plan a rollout of affected workloads after the upgrade.

Uninstalling the Operator’s Helm release removes its chart-managed components, but the Operator CRDs are retained by default. Before removing those CRDs manually, check for remaining OpenTelemetryCollector, Instrumentation, and other custom resources. Deleting a CRD also deletes every custom resource stored under it, so remove CRDs only after confirming they are no longer needed.

Triggering Injection With Annotations

You enable automatic instrumentation through annotations on a namespace or on an individual workload’s pod template. Namespace annotations set a default for matching workloads, while pod-template annotations give per-workload control. Injection takes effect when eligible pods are created, not retroactively on running pods, and it still depends on a supported runtime and a valid Instrumentation resource. Each language has its own annotation key:

The annotation value selects which Instrumentation resource the operator uses for injection:

• "true": the default-named resource in the workload’s own namespace.
• A resource name, such as "my-instrumentation": the resource with that name in the workload’s own namespace.
• "my-namespace/my-instrumentation": a resource in another namespace.
• "false": disables injection.

The Instrumentation resource created earlier is named default-instrumentation and lives in the opentelemetry namespace, so "true" does not select it for workloads in other namespaces. Reference it explicitly with the namespace/name form. The following fragment belongs inside an existing Deployment manifest:

spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "opentelemetry/default-instrumentation"

For a Deployment, place instrumentation annotations under spec.template.metadata.annotations, not the Deployment’s top-level metadata.annotations. The webhook reads annotations on newly created pods, so annotating the Deployment object alone won’t activate injection. Ensure the referenced Instrumentation resource exists before rolling out replacement pods.

When both namespace and pod annotations are present, the Operator applies these precedence rules:

1. A pod annotation with a specific `Instrumentation` resource name or `“false”` overrides the namespace annotation.
2. A pod annotation of `“true”` uses the namespace’s resource name if one is specified. If the namespace annotation is `“false”`, the pod’s `“true”` takes precedence and enables injection.
3. If the annotation is absent or empty at either level, the value from the other level applies.

For multi-container pods, use instrumentation.opentelemetry.io/container-names to select application containers. When instrumenting different languages within the same pod, use language-specific annotations such as instrumentation.opentelemetry.io/java-container-names and instrumentation.opentelemetry.io/python-container-names. Multi-instrumentation is enabled by default in Operator v0.162.0. Go also supports container selection through instrumentation.opentelemetry.io/go-container-names, but only one target container can be specified for Go injection.

Supported Language Runtimes

Operator v0.162.0 provides automatic instrumentation for Java, .NET, Node.js, Python, and Go, and adds PHP and Ruby auto-instrumentation, both disabled by default. Each language hooks the runtime through a different injected environment variable or runtime configuration, and some, such as Go, PHP and Ruby, need explicit enablement. Moving Node.js instrumentation across version 0.79.0 requires an explicit image update because of semantic convention changes.

• Java: The Operator injects the instrumentation agent and adds it to JAVA_TOOL_OPTIONS so the JVM loads it at startup. The Operator appends to values set in the pod spec, but values set inside the image or through some ConfigMap-based configurations can be overridden, so check existing JVM options for conflicts.
• Node.js: The Operator sets NODE_OPTIONS to preload the instrumentation module when the runtime starts. Existing startup options and module-loading behavior affect instrumentation.
• Python: The Operator sets PYTHONPATH to load the instrumentation libraries and exports over OTLP/HTTP protobuf by default. For Alpine or other musl-based images, set instrumentation.opentelemetry.io/otel-python-platform: "musl".
• .NET: The Operator injects the environment variables that load the OpenTelemetry CLR profiler and startup hook, for example CORECLR_ENABLE_PROFILING and DOTNET_STARTUP_HOOKS. The correct native profiler depends on the container’s C library, covered below.

.NET auto-instrumentation uses a native CLR profiler that must match the container’s C library. The instrumentation.opentelemetry.io/otel-dotnet-auto-runtime annotation selects linux-x64 (glibc, the default) or linux-musl-x64 (Alpine and other musl-based images). Set it to "linux-musl-x64" for musl-based containers. Using the wrong runtime identifier can prevent the profiler from loading and produce errors such as System.DllNotFoundException.

Go Runtime Constraints

Go injection works differently from the other runtimes because the Go runtime offers no startup hook for loading an agent the way Java, Node.js, Python, and .NET do. The Go agent instead uses eBPF uprobes to observe the running Go process from outside, which is why the injected sidecar needs elevated kernel access.

First, coverage. The agent traces supported operations in net/http, database/sql, google.golang.org/grpc, and github.com/segmentio/kafka-go (Go compatibility). You get automatic spans for HTTP, SQL, gRPC, and Kafka calls made through those libraries, and nothing for other operations.

Second, signals and containers. This agent produces traces only, not metrics or logs (signal limitations). The Operator can target only one application container for Go injection, even if the pod contains other containers. These limitations apply to this auto-instrumentation setup, not the OpenTelemetry Go SDK, so other signals and unsupported operations require SDK instrumentation or another collection method.

Third, configuration. Each piece of the setup has a specific job:

• Go instrumentation setting: Set manager.autoInstrumentation.go.enabled=true in the Operator Helm chart to enable Go injection. It’s disabled by default, and the chart passes --enable-go-instrumentation to the Operator when enabled.
• The instrumentation.opentelemetry.io/otel-go-auto-target-exe annotation: it identifies the executable to instrument.
• shareProcessNamespace: true: it lets the injected sidecar see the application’s process.
• Sidecar security context: The Go eBPF sidecar uses privileged: true and runAsUser: 0 by default, unless a custom security context is supplied through the Instrumentation resource.

This is a partial Deployment manifest showing the pod template settings:

spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-go: "opentelemetry/default-instrumentation"
        instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/app/server"
    spec:
      shareProcessNamespace: true

Replace /app/server with the actual path to your Go executable. The annotations alone are not enough: the feature gate, a supported Go runtime and libraries, the correct executable path, the sidecar’s permissions, and a reachable telemetry endpoint must all line up.

The Go auto-instrumentation agent has its own kernel requirements, separate from OBI. Its README targets Linux kernels above 4.4, and production compatibility also depends on the kernel’s available eBPF features and security settings.

The security difference matters in clusters with restrictive policies. Unlike the runtime-level agents for Java, Node.js, Python, and .NET, which need no extra kernel capabilities, Go injection adds a sidecar that requires privileged: true and runAsUser: 0. Clusters enforcing restrictive Pod Security standards reject that sidecar unless the namespace allows privileged pods.

Troubleshooting Injection Failures

When traces are missing, check the path from injection to export: the Operator and its admission webhook, the Instrumentation resource and annotations, the injected agent in the new pod, the application’s exporter endpoint and protocol, the Collector receiver, and the backend.

1. Webhook and Operator health. First, confirm that the Operator’s mutating admission webhook is registered and that the manager container is running without admission errors. Webhook configurations are cluster-scoped, so the first command needs no namespace flag. kubectl get mutatingwebhookconfigurations kubectl logs -l app.kubernetes.io/name=opentelemetry-operator --container manager \ -n opentelemetry-operator-system --follow

• context deadline exceeded when calling mpod.kb.io indicates that the admission webhook didn’t respond within the timeout. Check the webhook Service, ready endpoints, Operator logs, and network connectivity from the Kubernetes API server.
• no endpoints available for service indicates that the webhook Service has no ready backing endpoints. Check the Operator pod’s readiness and the Service selector.
• x509: certificate signed by unknown authority indicates a certificate trust failure. Check the webhook’s TLS configuration in step 4.

1. Annotation and the selected Instrumentation resource. Next, connect the workload’s annotation to the exact Instrumentation resource it selects, because confirming that some Instrumentation exists is not enough when the annotation references a resource in another namespace. kubectl get instrumentation -n opentelemetry kubectl describe instrumentation default-instrumentation -n opentelemetry kubectl get deployment <deployment-name> -n <app-namespace> -o yaml
2. Injected agent in the pod. Third, check the pod specification before anything else. kubectl describe pod <pod-name> -n <app-namespace> kubectl get pod <pod-name> -n <app-namespace> \ -o jsonpath='{.spec.initContainers[*].name}' kubectl get events -n <app-namespace>
   • cp: can't preserve ownership ... Operation not permitted means filesystem permissions or the security context block copying the agent files.
   • A runAsNonRoot error means the pod’s user restriction conflicts with an instrumentation image that runs as root.
   • OOMKilled means the init container exhausted its memory limit. Raise the limit through the Instrumentation resource’s per-language resource requirements.
   • Evicted means Kubernetes removed the pod under node resource pressure.
   • ImagePullBackOff requires checking the image reference, registry access, and pull credentials, which come from the pod’s imagePullSecrets or its ServiceAccount.
3. Webhook TLS. Fourth, identify the Operator’s own webhook before inspecting its certificate, because cert-manager-webhook belongs to cert-manager and not to Pod injection. kubectl get mutatingwebhookconfigurations kubectl describe mutatingwebhookconfiguration <operator-webhook-name> kubectl get certificates -n opentelemetry-operator-system
   • Expired certificate: Check the webhook certificate’s validity period. If cert-manager manages it, inspect the Certificate resource’s readiness and renewal status.
   • CA mismatch: The webhook’s caBundle must trust the certificate presented by the Operator’s webhook Service. A mismatch can cause x509: certificate signed by unknown authority errors, including after certificate regeneration.
   • Missing CA bundle: With cert-manager, check that the webhook configuration has the appropriate CA injection annotation and that cainjector is running without reconciliation or RBAC errors.
   • Helm-generated certificates: If admissionWebhooks.certManager.enabled=false, inspect the webhook TLS Secret and the CA bundle generated by the Helm chart. This setup doesn’t depend on cert-manager’s Certificate resources or cainjector.
4. Telemetry delivery. Finally, confirm that a correctly injected application delivers spans to the backend. Start by confirming that the injected application starts, then send a test request through an endpoint that uses an instrumented library. Compare the pod’s OTEL_EXPORTER_OTLP_ENDPOINT and protocol (gRPC on 4317 or HTTP/protobuf on 4318) against the Collector receiver, and confirm that the Collector has the otlp receiver and a traces pipeline. The example samples 25% of new traces (parentbased_traceidratio at 0.25), so send several requests before concluding that spans are missing. kubectl logs deployment/otel-collector -n opentelemetry

The Operator creates the otel-collector Deployment from the OpenTelemetryCollector resource named otel. Check its logs for receiver and exporter errors. The example uses the debug exporter with basic verbosity, which logs the number of spans received and confirms that telemetry has reached the Collector.

To inspect individual spans and resource attributes, change spec.config.exporters.debug.verbosity to normal or detailed in the OpenTelemetryCollector resource and reapply the configuration. The debug exporter doesn’t forward traces to a production backend. To verify end-to-end delivery, configure an exporter for your tracing backend and confirm that the traces appear there.

Zero-Code vs. Manual Instrumentation Trade-Offs

Automatic instrumentation captures spans for supported frameworks and libraries, such as HTTP requests, database calls, and messaging interactions, without changing application code. Coverage depends on the runtime, the supported libraries, and configuration, and agents do record exceptions in supported frameworks. Application-specific operations fall outside that coverage: tracing business workflows, attaching custom attributes, or recording exceptions the agent doesn’t capture automatically needs code-level instrumentation. The two approaches complement each other, with automatic instrumentation supplying the baseline telemetry and manual instrumentation adding business context.

A hybrid approach uses Operator-managed instrumentation for supported framework and library operations and adds manual spans where business context matters. Platform teams manage shared instrumentation settings centrally, so application teams don’t repeat SDK setup per service. New pods still have to be created for injection, and each workload still needs compatible runtime and exporter configuration plus telemetry verification.

Overhead depends on the agent version, application, runtime, and configuration, which is why the OpenTelemetry project gives no single estimate. Benchmark your own workload under production-like limits before estimating CPU, memory, or latency cost. The measurements below show what each test covers and where its scope ends.

• Application-level latency and startup benchmark: The Java instrumentation project’s benchmark harness measures application latency and startup overhead under controlled load. In its November 2, 2026 release run, request p95 latency increased from 8.59 ms without instrumentation to 8.79 ms with the latest stable agent, while startup time increased from 12,332 ms to 13,287 ms. The linked summary is updated as new runs complete, so these figures are a dated snapshot rather than a fixed benchmark result. Test your own workload before using them to estimate overhead.
• Method-level microbenchmark: An ICPE ‘26 preprint using MooBench (2,000,000 executions, 10 JVM starts, call-tree depth 10) measured Java agent 1.56.0 at 0.315 µs for each instrumented call on x86_64. The figure isolates the cost the agent adds to each instrumented call, separate from application work.
• Application startup overhead: In one Java agent deployment report, average startup time increased from 61.4 seconds without an agent to 81.7 seconds with agent 1.10 and 84.9 seconds with agent 1.26. Each configuration was tested five times. This is a workload-specific example rather than a general startup penalty. Compare startup times under the same resource limits, review startup and readiness probes, and disable unnecessary instrumentation libraries where possible.
• Instrumentation method and sampling study: A 2024 study of a five-service Spring microservice system found that automatic tracing used roughly twice the CPU of manual tracing under its test conditions. Automatic tracing had median overheads of approximately 42% CPU, 22% memory, and 24% request p95 latency. In a separate experiment using manually instrumented traces, reducing the head-based sampling rate lowered median CPU overhead from 17.8% to 3.6%, while memory overhead showed no consistent downward trend. These results demonstrate how instrumentation method and sampling affect overhead under the tested conditions, not what every Kubernetes workload will experience.

The operator removes code changes for supported workloads, but automatic instrumentation still depends on runtime compatibility, the available instrumentation libraries, and deployment permissions. Runtimes the operator cannot inject still have options: SDK instrumentation or a different collection approach. Manual instrumentation gives the most control over application-specific spans.

Alternatives to the Operator: eBPF and Service Mesh

Two other mechanisms produce telemetry without SDKs: kernel-level eBPF capture and service mesh proxies. Each observes from a different vantage point, and each trades differently against the operator.

Grafana Beyla / OpenTelemetry eBPF Instrumentation (OBI)

OpenTelemetry eBPF Instrumentation (OBI), which originated from Grafana Beyla, uses eBPF plus runtime-specific instrumentation to collect application traces and metrics without source-code changes. It reached v1.0.0 with a stable core compatibility contract. Unlike Operator injection, it instruments supported processes that are already running, without recreating their pods, and coverage depends on the runtime, protocol, and enabled features.

OBI supports workloads written in Java, .NET, Go, Python, Node.js, Ruby, C, C++, and Rust, although instrumentation capabilities and runtime requirements differ (support matrix). It requires Linux 5.8+ with BTF, except on supported RHEL-derived 4.18 kernels that carry the required eBPF backports (platform requirements). It supports capability-based deployments as an alternative to privileged: true, and current releases no longer need CAP_SYS_ADMIN for most probe attachment, though some features still do (security docs).

OBI collects application and network telemetry, runtime metrics for supported languages, and distributed traces, though collection depth varies by runtime and protocol. Trace-context propagation has its own limits: network-level trace-context propagation is disabled by default and requires OTEL_EBPF_BPF_CONTEXT_PROPAGATION=all. For generic HTTPS traffic, propagation works only between OBI-instrumented services and can be disrupted by L7 proxies. Go library-level instrumentation also supports propagation for HTTP/2 and gRPC.

Pixie

Pixie deploys Pixie Edge Modules (PEMs) as a DaemonSet that capture data through eBPF probes, with kprobes on networking syscalls and uprobes on TLS libraries, covering protocols such as HTTP, HTTP/2 and gRPC, DNS, MySQL, PostgreSQL, Redis, and Kafka, without modifying application code. Pixie stores telemetry in memory inside the monitored cluster, with retention typically measured in hours and capped at about 24 hours. Longer retention means exporting to an external storage or observability system, which Pixie supports through its OpenTelemetry integrations.

eBPF collection needs elevated node-level access. The shipped PEM manifest at the linked commit runs privileged with host PID and host network (pem_daemonset.yaml), and the exact capabilities depend on the Pixie version and deployment configuration. Pixie requires Linux kernel 4.14+, and the current installation requirements list the rest.

groundcover eBPF sensor

groundcover’s eBPF sensor deploys as a single DaemonSet, one pod per node, and captures logs, metrics, traces, Kubernetes events, network calls, and HTTP requests directly from the Linux kernel, with no SDKs, no per-service agents, and no application restarts. It requires Kubernetes 1.21+ and Linux kernel 4.16+, supports EKS, AKS, and GKE, and does not support AWS Fargate or Docker Desktop. Existing OpenTelemetry instrumentation coexists with the sensor’s data in the same interface, so operator-injected SDK spans and kernel-captured telemetry are not an either-or.

Storage runs under BYOC (Bring Your Own Cloud, where your data plane runs in your own cloud account). ClickHouse stores logs, traces, and Kubernetes events, and VictoriaMetrics stores metrics, both inside your environment. Retained telemetry therefore lives in storage you control rather than in node memory.

If you want to validate the architecture on your own cluster, the free plan includes BYOC and requires no credit card. Deploy the eBPF sensor on one cluster and evaluate full-cluster visibility within hours.

Service Mesh (Istio, Linkerd)

Istio and Linkerd proxies produce request rates, error rates, latency, and network-level metrics as part of routing traffic, with no instrumentation in application code. Istio generates istio_requests_total, istio_request_duration_milliseconds, and TCP metrics through its metrics configuration, and Linkerd’s proxies report success rate, request rate, and p50/p95/p99 latency without developer work. Proxies do not provide the application-specific span detail that language agents or manual instrumentation produce.

Proxies emit spans, but connecting them into end-to-end traces requires applications to propagate trace-context headers from inbound to outbound requests. Existing tracing libraries or Operator-injected agents already handle that propagation; without them, you add the forwarding through configuration or code changes. Sampling is a separate setting: Istio’s examples use a 1% default sampling rate that you change through the Telemetry API’s randomSamplingPercentage, while header propagation decides whether each sampled trace is complete.

Trace-context mismatches split application and proxy spans into separate traces. In one CNCF example, an OpenTelemetry-instrumented application and the Envoy sidecar produced two disconnected traces for the same request because their propagation configuration wasn’t aligned. Configuring compatible tracing providers and propagation formats joined the spans into one trace, so the failure came from configuration, not from combining a mesh with OpenTelemetry.

Choosing between them comes down to what you already run and what you can restart. Use the Operator when you control rollouts: it injects language-specific agents for Java, Node.js, Python, and .NET, and covers Go through a privileged eBPF sidecar limited to four library families. Use standalone eBPF tools when you cannot restart pods or need Go coverage beyond those libraries, since they observe workloads that are already running. Multi-signal collection from one sensor is specific to groundcover’s eBPF sensor, whose node-level DaemonSet collects logs, metrics, traces, and Kubernetes events, and other eBPF tools do not necessarily offer it. If you already run a service mesh, use its request metrics and proxy-level tracing to complement application instrumentation. Configure compatible trace-context propagation when you need end-to-end traces.

GitOps and Drift Prevention

GitOps suits Instrumentation resources, Collector configuration (OpenTelemetryCollector CRs), and workload annotations, but webhook certificates need attention. When cert-manager manages webhook trust, its cainjector populates the caBundle field in the webhook configurations and CRDs it is configured for. If Argo CD also tracks this controller-managed field, the webhook resource can remain OutOfSync because its live caBundle differs from the rendered manifest. You can prevent this false drift using Argo CD’s diff configuration.

First, the Argo CD fix: identify which webhook resources have a controller-managed caBundle, then scope an ignoreDifferences rule to them. This Application-level example targets one webhook configuration:

spec:
  ignoreDifferences:
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      name: <operator-webhook-name>
      jqPathExpressions:
        - .webhooks[]?.clientConfig.caBundle

Replace <operator-webhook-name> with the Operator’s webhook configuration name from the output of this command:

kubectl get mutatingwebhookconfigurations

When centralized management is intentional, apply the rule system-wide in argocd-cm instead:

resource.customizations.ignoreDifferences.admissionregistration.k8s.io_MutatingWebhookConfiguration: |
  jqPathExpressions:
    - '.webhooks[]?.clientConfig.caBundle'

This system-wide rule ignores caBundle on every MutatingWebhookConfiguration that Argo CD manages. Ignore the field only when a trusted controller maintains it, never to hide an unexplained certificate mismatch or a broken trust configuration.

Second, RespectIgnoreDifferences=true: ignoreDifferences only affects the diff. Add RespectIgnoreDifferences=true to syncPolicy.syncOptions when syncs would otherwise overwrite the injected caBundle, because it makes syncs of existing resources leave the ignored field alone (issue #9071). Enable it for the resources that need it, not as a blanket default.

Next, Flux: the setting depends on the controller that applies the resource. A Kustomization uses spec.ignore rules targeting the caBundle path. A HelmRelease uses its own drift detection settings.

The final point is a separate problem, Helm-generated certificates. Argo CD renders Helm without cluster lookups, so lookup returns empty and the chart regenerates certificate material on every sync. Ignoring caBundle (the chart’s documented workaround) hides only the diff, so the certificate Secret, the serving certificate, and the caBundle must still match, or the webhook fails TLS verification. cert-manager avoids this regeneration.

Ordering is the second GitOps concern. The Instrumentation resource must exist before the pods it instruments are created, so sequence the installation explicitly:

1. Install the webhook certificate-management dependencies where applicable (cert-manager).
2. Install the Operator CRDs and the Operator controller.
3. Verify the webhook is ready, then create the Instrumentation resources.
4. Deploy or roll out the annotated application workloads.

For Argo CD, sync waves order resources within one Application, for example cert-manager and CRDs at wave -1 or 0 and Instrumentation at wave 1. When these components live in separate Applications, wave numbers do not order them, so you need readiness checks or dependency orchestration between Applications.

For Flux, Kustomization.spec.dependsOn orders Kustomizations, while HelmRelease.spec.dependsOn waits for other HelmReleases to be Ready and does not wait on arbitrary Kustomizations.

CRD upgrades need separate care. The OpenTelemetry Operator chart installs its CRDs as templates, and older installations need a CRD ownership migration described in the chart’s upgrade instructions. Flux’s upgrade.crds: CreateReplace governs CRDs in a chart’s CRD directory, so it is not the fix for templated CRDs.

Three Operator behaviors affect GitOps workflows even with correct drift rules:

1. 
   Instrumentation changes do not rewrite running pods. The Operator injects configuration when pods are created, so changes to Instrumentation resources don’t automatically update running pods. Trigger a controlled rollout through the workload’s GitOps process, for example by committing a pod-template annotation change to Git rather than making an untracked manual change. Then check the replacement pods for the expected instrumentation configuration and verify telemetry.
2. 
   Readiness signals differ by resource. Recent OpenTelemetryCollector implementations expose status.observedGeneration and conditions, while Instrumentation resources do not provide equivalent reconciliation status, so check what your installed Operator release exposes. For Instrumentation, confirm webhook readiness and successful injection into newly created pods instead of treating the resource’s existence as proof that telemetry works.
3. 
   Generated RBAC can conflict with GitOps tracking. Some Operator versions may generate ClusterRole and ClusterRoleBinding resources even when an OpenTelemetryCollector references an existing ServiceAccount. These resources can inherit Argo CD tracking labels and appear to belong to the wrong Application. If unexpected RBAC resources show OutOfSync, inspect their labels, ownership, and the installed Operator version before changing tracking configuration. Don’t broadly exclude ClusterRoles or ClusterRoleBindings, because that hides legitimate RBAC drift.