Datadog for Golang: APM setup, dd-trace-go, and the eBPF alternative

Your Go services run on Kubernetes, and before the first trace shows up someone has to decide how much instrumentation work the platform team will own. Instrumenting Golang services with Datadog means adding dd-trace-go v2 to every binary and running the Datadog Agent on every node; you also choose between Orchestrion's compile-time rewriting and hand-wired contrib packages; the top results for golang datadog are Datadog's own docs, which list the steps and skip the cost of owning them. First, the instrumentation components and dd-trace-go setup; second, tracer coverage; third, SDK-per-language maintenance and billing; fourth, the eBPF alternative; fifth, Go observability's next steps; and sixth, common questions.

What Datadog for Golang instrumentation involves

Datadog's Golang APM has two required parts, the tracer and the Agent, plus an optional compile-time tool, Orchestrion:

dd-trace-go v2: The tracing library compiled into your Go binary. The latest stable release is v2.11.2, published October 13, 2026, under the module path github.com/DataDog/dd-trace-go/v2. The v2.11.0 release notes raised the minimum Go version to 1.26.0 and dropped support for Go 1.25 and earlier; Datadog's tracing page still says Go 1.18+, and the docs page has not caught up. Plan on the release note, which is newer and more specific. The Datadog Agent: A per-node process that receives spans from the tracer and forwards them to Datadog. The Go tracing page says to install the Agent first, version 5.21.1 or later (vendor-stated). Orchestrion: Datadog's compile-time auto-instrumentation tool, latest stable v1.14.0, also published October 13, 2026. Its go.mod pins github.com/DataDog/dd-trace-go/v2 v2.11.2, so upgrading Orchestrion also moves your tracer version.

Those three components define the setup path, and the path starts with the tracer in main.

How dd-trace-go works

dd-trace-go runs inside your process: it records spans in memory and sends them over HTTP to an Agent on the same node. Setup covers starting the tracer, configuring service tags, connecting the Agent and choosing Orchestrion or contrib packages.

Installing dd-trace-go and starting the tracer

The repo README gives the install line as go get github.com/DataDog/dd-trace-go/v2/ddtrace/tracer (README); Datadog's Go tracing page gives no go get command. Start the tracer before any instrumented code runs, because tracer.StartSpan is a no-op when the tracer is not started (pkg.go.dev):

package main

import (
	"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func main() {
	tracer.Start(
		tracer.WithService("checkout"),
		tracer.WithEnv("prod"),
		tracer.WithServiceVersion("1.4.2"),
	)
	defer tracer.Stop()
	// ... start your server
}

The three With* options map to the unified service tags, which you can also set through environment variables.

Unified service tagging and env config

Unified service tagging means every trace, metric, log and profile carries matching env and service values, along with the same version. Datadog's Go library configuration page documents these defaults:

DD_TRACE_AGENT_URL defaults to http://localhost:8126, accepts unix:// paths, and overrides host and port when set. That URL is also how you choose the transport to the Agent.

Running the Datadog Agent

Ordinary APM traces always go through the Agent; option.go sends ordinary trace submissions to the Agent URL. The only documented paths that skip a local Agent are AWS Lambda through the Datadog Lambda Extension (Lambda docs), Google Cloud Run and Azure container platforms through serverless-init with DD_API_KEY and DD_SITE (Cloud Run docs), CI test reporting, and LLM Observability. In Kubernetes, every node that runs Go pods needs an Agent the pods can reach.

For a local docker-compose setup, run the Agent as a second service with DD_API_KEY set and pick one of two transports (Docker APM docs):

• TCP: Set DD_AGENT_HOST to the Agent container name and DD_TRACE_AGENT_PORT=8126 on the app.
• UDS: Mount /var/run/datadog/ into both containers and set DD_TRACE_AGENT_URL=unix:///var/run/datadog/apm.socket on the app. On the Agent side, DD_APM_RECEIVER_SOCKET takes priority over host and port.

Datadog recommends UDS for Kubernetes (Kubernetes APM docs), and the Helm chart's values.yaml defaults datadog.apm.socketEnabled to true and portEnabled to false. If you use TCP instead, DD_AGENT_HOST must come from status.hostIP so each pod talks to the Agent on its own node. DogStatsD metrics follow the same pattern: UDP 8125 or the /var/run/datadog/dsd.socket socket.

With the Agent reachable, the remaining decision is how spans get into your code in the first place.

Orchestrion vs contrib packages

Orchestrion rewrites your code at compile time; contrib packages require you to wrap each framework and client by hand. Orchestrion uses the Go toolchain's -toolexec hook to intercept compilation units and modify the AST, which lets it instrument application code and its dependencies, including the standard library (Orchestrion docs). Install it with go install github.com/DataDog/orchestrion@latest, run orchestrion pin (and commit go.mod and go.sum, along with orchestrion.tool.go), then build with orchestrion go build . or go build -toolexec="orchestrion toolexec" . (automatic instrumentation docs).

Two directives control what it touches. //dd:span above a function wraps it in a span, accepts literal key:value tags such as //dd:span tag-name:foo, and attaches a returned error to the span automatically. //orchestrion:ignore disables instrumentation for the annotated syntax tree (custom-trace docs), but callee-side instrumentation added inside a dependency cannot be opted out locally (automatic instrumentation docs). Mixing Orchestrion with manual contrib imports produces duplicate spans; discussion #566 reports two spans per request when chi or net/http is instrumented both ways.

The trade-offs line up as follows:

Orchestrion is incompatible with Bazel/rules_go (AAP setup docs). Datadog's 2024 blog says it needs a build-process change and redeploy, which makes it slightly less automated than eBPF (Datadog blog).

Key features and capabilities

The Go tracer's coverage is defined by its contrib modules, and in v2 every one of them is a nested Go module with its own go.mod (contrib README). Coverage includes framework integrations and custom spans, metrics and log correlation, profiling and runtime metrics, plus application security.

Framework, database and client integrations

Datadog's compatibility page lists the v2 import paths, all prefixed github.com/DataDog/dd-trace-go/contrib/:

• Gin: .../gin-gonic/gin/v2
• Echo v4: .../labstack/echo.v4/v2 (an echo.v5/v2 module was added in v2.10.0)
• Chi: .../go-chi/chi/v2 and .../go-chi/chi.v5/v2
• net/http: .../net/http/v2
• database/sql: .../database/sql/v2
• GORM: .../gorm.io/gorm.v1/v2
• go-redis: .../go-redis/redis/v2 and .../redis/go-redis.v9/v2
• gRPC: .../google.golang.org/grpc/v2
• Kafka: .../IBM/sarama/v2, plus confluent-kafka-go and kafka-go modules

One warning on the Kafka row: the compatibility page shows .../confluentinc/confluent-kafka-go/v2, while the go.mod files on main use .../confluent-kafka-go/kafka.v2/v2 and .../confluent-kafka-go/kafka/v2. Check pkg.go.dev before you pin.

Custom spans and context propagation

Datadog recommends passing context.Context explicitly everywhere a span needs a parent (custom instrumentation docs). tracer.StartSpanFromContext(ctx, "checkout.reserve") returns a *Span and a new context; pass that context into a goroutine and call StartSpanFromContext again inside it, with defer span.Finish(), and the child attaches to the right parent. Record errors with span.Finish(tracer.WithError(err)) and attributes with span.SetTag.

Across services, inject with tracer.Inject(span.Context(), tracer.HTTPHeadersCarrier(req.Header)) and extract with tracer.Extract(tracer.HTTPHeadersCarrier(r.Header)). The default propagation style is datadog,tracecontext,baggage, so W3C traceparent headers from OpenTelemetry-instrumented neighbours are read, and Datadog's propagation docs say the Datadog format wins on extraction when both are present.

Custom metrics and log correlation

Custom metrics go through DogStatsD using github.com/DataDog/datadog-go/v5/statsd (v5.9.1), which exposes Gauge, Count, Incr, Histogram and Distribution and accepts WithNamespace and WithTags options (pkg.go.dev). Log correlation needs the dd.trace_id and dd.span_id fields in each line (log correlation docs). For a log/slog service, github.com/DataDog/dd-trace-go/contrib/log/slog/v2 provides WrapHandler and NewJSONHandler, and injects the trace and span ID fields (the ext.LogKeyTraceID and ext.LogKeySpanID keys) when you log with the span's context via logger.Log(ctx, ...) (pkg.go.dev). A zap module, .../contrib/go.uber.org/zap/v2, arrived in v2.11.0 with TraceFields(ctx), which returns trace-correlation fields you spread into each log call. Under Orchestrion v1.11.0+, Orchestrion rewrites those zap calls automatically (PR #4729).

Continuous Profiler and runtime metrics

The profiler lives at github.com/DataDog/dd-trace-go/v2/profiler and collects only CPU and heap profiles by default; block, mutex and goroutine profiles are opt-in through profiler.WithProfileTypes (profiler docs). Those opt-in types map to Go failure modes that matter in production: mutex and block profiles expose lock contention, while the goroutine profile exposes goroutine leaks, and GC pauses appear in the v2 runtime metrics, which dd-trace-go v2.2.0+ submits automatically under names like runtime.go.metrics.sched_pauses_total_gc.seconds (runtime metrics v2 docs). The profile types page warns that block profiling causes noticeable overhead in production, and that goroutine profiling needs stop-the-world pauses of about 1 µs per goroutine per 60-second period, which matters above 10,000 goroutines; for reading these profiles without a vendor, see groundcover's Go profiling guide.

Datadog publishes no numeric CPU or memory overhead figure for the Go profiler or tracer. The closest figures are Nick Ripley's personal measurements (Go 1.17.6, GC disabled) showing under 1% latency and CPU increase except a roughly 2% rise for a single-goroutine HTTP workload (repo), and Felix Geisendörfer's note that the Go 1.21 execution tracer "should provide less than 1% overhead for most applications" (blog). Neither is a Datadog benchmark.

Application security

AppSec is built into the tracer rather than shipped as a separate package, and you enable it with DD_APPSEC_ENABLED=true plus a redeploy (AppSec setup). Under Orchestrion the build becomes orchestrion go build -tags=appsec my-program. With CGO_ENABLED=0 the binary is no longer fully static, because the in-app WAF loads libc.so.6 and libpthread.so.0 at runtime, along with libdl.so.2. HTTP monitoring and blocking cover Gin, Echo, Chi, Gorilla Mux, Fiber and net/http, blocking needs Remote Configuration, and IAST is not supported for Go (compatibility). The WAF budget per request is DD_APPSEC_WAF_TIMEOUT, default 2ms, after which checks are skipped.

With each feature above, you maintain more instrumentation in each service, adding to the cost of the SDK-per-language model.

The maintenance and cost reality of the SDK-per-language model

Owning dd-trace-go means owning a Go version policy, a dependency tree, a migration schedule and a billing meter, per service.

Version pinning follows Go's own release cadence. Datadog supports the two latest Go releases, with the third in maintenance (compatibility policy), and v2.11.0 dropped Go 1.25 eight months after Go 1.26 shipped.

Because each contrib is a separate module, pkg.go.dev shows different versions for different integrations: gin and chi.v5 are at v2.10.1, while sarama is at v2.8.2. kafka.v2 is at v2.11.1 while the core sits at v2.11.2, and no page states that contribs release in lockstep.

Orchestrion adds its own pin on top; every orchestrion pin upgrade is a dependency bump across every Go service in your cluster.

v1 to v2 migration is already past its deadline. Datadog's compatibility table puts v1 end-of-life at 2025-12-31, and v1.74.7 was announced as the last minor v1 release, then retracted, so v1 users should take the latest v1.74.x (v1.74.7).

The move rewrites gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer to github.com/DataDog/dd-trace-go/v2/ddtrace/tracer, splits every contrib into its own module, replaces the ddtrace interfaces with structs, and removes tracer.WithServiceName (now WithService), tracer.WithPrioritySampling, tracer.WithHTTPRoundTripper and the old sampling-rule helpers (MIGRATING.md).

Trace IDs change from uint64 to string for 128-bit support.

The v2fix tool (go install github.com/DataDog/dd-trace-go/tools/v2fix@latest, then v2fix -fix .) automates part of it; its README says some changes cannot be fully automated (v2fix README).

Sampling rules are the lever between visibility and the bill. DD_TRACE_SAMPLE_RATE is deprecated in favour of DD_TRACE_SAMPLING_RULES, head-based sampling decides at the root span, the tracer rate-limits at 100 traces per second per service instance, and the Agent targets 10 traces per second by default (ingestion mechanisms).

PointFive names a sampling rule left at 100% after an incident investigation as a common cause of span overages (PointFive).

Billing has two meters stacked on a per-host base:

• Datadog's price list (vendor-stated, USD) shows APM at $31 per APM host per month billed annually and APM Pro at $35; APM Enterprise costs $40. Month-to-month APM costs $36, while APM Pro and APM Enterprise cost $42 and $48 respectively.
• Each host includes 1 million indexed spans and 150 GB of ingested spans; overage is $0.10 per GB ingested and $1.70 per million indexed spans at the default 15-day retention (APM billing docs). Datadog's own worked example puts 5 hosts with 30M indexed spans and 900 GB ingested at $212.50 per month.
• For Kubernetes, APM is priced by node rather than pod, and non-Fargate containers bill against the host running the Agent.
• The Continuous Profiler is a separate $19 per profiled host unless you take the APM Enterprise bundle. The standalone APM page shows an annual APM rate of $36, with APM Pro and APM Enterprise at $41 and $47 respectively. These figures apply when APM is bought on its own rather than attached to Infrastructure Monitoring; the docs do not reconcile the two lists beyond that (APM pricing).

The pattern across all four items is the same: cost and toil scale with services and languages, while span volume adds to the bill. The eBPF model removes those per-service instrumentation and volume-billing inputs.

The eBPF alternative: how Datadog for Golang compares to groundcover

groundcover's eBPF sensor captures Go traffic from the Linux kernel instead of from inside your binary, so the instrumentation decision moves from every service to one node-level deployment. The sensor runs as a Kubernetes DaemonSet, one pod per node (architecture), and attaches to syscalls, kprobes, tracepoints and uprobes to read protocol traffic as it crosses the socket boundary (eBPF vs traditional APM agents guide). From that vantage point it parses HTTP, gRPC, MySQL, PostgreSQL, Redis, DNS, Kafka, MongoDB, AMQP, GraphQL, AWS S3 and AWS SQS (supported technologies), which covers the same request, query and queue hops that the contrib list above instruments one import at a time.

Zero application instrumentation is the operational consequence. No go get, no tracer.Start, no Orchestrion rebuild, no restart; groundcover says a Go service deployed before the sensor appears in traces once the sensor runs on its node (vendor-stated).

The sensor needs no per-language SDK, so the same DaemonSet captures Java and Python services in your cluster at the protocol level; TLS capture still depends on language-specific hooks, as the Go case below shows. groundcover ingests OpenTelemetry (OTLP) spans as a first-class source alongside the eBPF signals, so services you already instrumented keep their traces in the same interface.

groundcover charges a flat per-node rate based on the monthly average of monitored Kubernetes nodes, with no per-GB ingestion, per-span, per-metric or per-user charges, so span volume and sampling rules stop being a billing input (pricing, vendor-stated).

Go has quirks that any eBPF sensor must handle, and groundcover documents its own answers. Go uses crypto/tls rather than libssl, so the sensor hooks (*Conn).Read and (*Conn).Write by binary symbol offset (eBPF auto-instrumentation guide); binaries built with stripped symbols lose TLS visibility, so keep -ldflags="-s" out of your production builds. Go stacks grow and move at runtime, which corrupts the return-address rewrite that uretprobes depend on, so function-level probing has to be done differently from C. The sensor needs privileged containers and a BTF-enabled kernel (kernel requirements), and it runs on Linux nodes only, not Fargate.

In-process waterfall spans still need OTel instrumentation:

• With kernel-level collection, you can trace a request entering your handler and a query leaving it, but not the function calls or worker pools in between. You also lack visibility into channel hand-offs. OpenTelemetry's own guidance says eBPF instrumentation "Cannot easily see internal function calls or custom business logic" (OpenTelemetry docs).
• groundcover's guide notes that most eBPF sensors cannot inject W3C traceparent headers outside a narrow opt-in path, which integrity kernel lockdown disables.

For the handful of services where you need that depth, add OpenTelemetry Go SDK spans and send them over OTLP; the sensor covers protocol-level traffic for the rest without code changes.

That split is the strategic choice: Datadog puts instrumentation work on every Go service and bills per host plus volume; the eBPF model puts collection on the node and leaves SDK work to the services that need in-process detail.

If you want to test the architecture on your own cluster, the free plan includes BYOC (Bring Your Own Cloud, where your data plane runs in your own cloud account) and requires no credit card. Deploy groundcover's eBPF sensor on one cluster and evaluate full-cluster visibility within hours under the documented deployment conditions.

What's next for Go observability

Orchestrion remains a moving target: its docs say it is still under active development and APIs may change (Orchestrion docs), and the Datadog-Alibaba collaboration it started has produced a separate OpenTelemetry Go compile-time instrumentation v1 (July 2026) that supports net/http, database/sql, gRPC, Redis and runtime metrics with an otelc go build command (OpenTelemetry blog). That announcement frames the choice the same way this article does: compile-time instrumentation for teams that can rebuild, eBPF where the binary cannot be rebuilt.

For a Go platform team, the durable investment is OTLP-shaped data: it survives a change of backend, and both compile-time and eBPF collectors now speak it. The OpenTelemetry Go SDK reached v1.47.0 on October 2, 2026 with the first stable Logs API and SDK (release), and the CNCF 2025 annual survey (628 respondents) found OpenTelemetry in production at 49% with 26% evaluating (CNCF). Datadog's position is that you should not run its SDK and the OpenTelemetry SDK in the same application, "as this leads to undefined behavior" (Datadog OTel compatibility); its bridge package github.com/DataDog/dd-trace-go/v2/ddtrace/opentelemetry exposes the OTel API on top of the Datadog tracer but does not support OTel SpanProcessors or OTLP exporters. On the kernel side, OpenTelemetry eBPF Instrumentation (OBI, formerly Grafana Beyla) shipped its first alpha in November 2025 (OpenTelemetry blog), which puts a vendor-neutral eBPF collector next to the proprietary ones.