Instrumenting Go applications with Datadog APM

Go took over the layer of the stack where observability is hardest: high-concurrency network services where a goroutine leak or a GC pause degrades every request in flight. Request logs won’t show you which of 40,000 goroutines is blocked on a mutex, and CPU graphs won’t tell you which handler allocated the heap that triggered the pause. This guide covers the full go/datadog integration path from tracer setup to framework integrations and node-level alternatives.

What is Datadog APM for Go

Datadog delivers Application Performance Monitoring (APM) for Go through dd-trace-go, the official Go tracing library. The library provides distributed tracing and profiling. Adjacent packages add runtime telemetry and application security monitoring. It sends data to a locally running Datadog Agent, which forwards it to Datadog’s backend. Datadog lists Go as a supported APM language in Datadog’s APM library matrix.

Teams already running the Datadog Agent use it for code-level trace visibility in their Go services. The tracing UI is the part practitioners usually care about first because it connects request paths, spans, errors, and latency in one view.

Two instrumentation paths exist. Manual instrumentation means importing dd-trace-go packages and editing source; Orchestrion instruments at compile time with no source changes. The sections below cover both.

How Datadog APM for Go works

The architecture has three hops:

1. dd-trace-go runs inside your application process and generates spans.
2. The Datadog Agent runs as a host daemon or Kubernetes DaemonSet and receives those spans over TCP port 8126 or a Unix Domain Socket.
3. The Agent forwards them to Datadog’s backend.

In Kubernetes, Datadog recommends the UDS path at /var/run/datadog/apm.socket, with the application pod talking to the Agent DaemonSet pod on the same node.

Sampling happens at the head by default. Without custom rules, the Agent targets 10 traces per second per service, and the SDK enforces a rate limit of 100 traces per second per service instance. That matters for billing later: host-based APM charges, span ingestion, and indexed-span volume make sampling configuration cost configuration too.

Prerequisites and environment setup

You need a Datadog Agent version 5.21.1 or newer running and reachable from your application. That is the documented floor, not the practical target: Datadog’s Go compatibility page now lists 7.76.1 as the minimum recommended Trace Agent version, because earlier Agents don’t automatically obfuscate authentication parameters in trace metadata.

For the Go toolchain, dd-trace-go follows the official Go release policy of supporting the two latest releases. Note a documentation quirk: the setup page still says “The Go Tracer requires Go 1.18+”, but the v2.9.1 go.mod declares go 1.25.0, and the go.mod directive is what your build enforces. Treat Go 1.25 as the real floor for current releases.

The tracer reads its Agent connection from environment variables:

DD_TRACE_AGENT_URL takes precedence over DD_AGENT_HOST and DD_TRACE_AGENT_PORT when set. A related variable, DD_DOGSTATSD_URL, belongs to the standalone DogStatsD client rather than the tracer configuration; it accepts udp://localhost:8125 or unix:///var/run/datadog/dsd.socket.

Installing dd-trace-go

Install the v2 module, which is the current major version. v2 reached general availability in June 2025, and Datadog recommends it for all users:

go get github.com/DataDog/dd-trace-go/v2/ddtrace/tracer
go get github.com/DataDog/dd-trace-go/v2/profiler

The latest stable tag is v2.9.1, published 2026-06-26. Pin it explicitly if your builds require reproducible versions: go get github.com/DataDog/dd-trace-go/v2/ddtrace/[email protected].

The v1 module (gopkg.in/DataDog/dd-trace-go.v1) is no longer developed. Datadog’s position is that every v1 release from v1.74.0 onward is a transitional release: it preserves the v1 API but uses v2 under the hood, and it carries no new features. That transitional line is the only one that allows both v1 and v2 imports in the same service, which is what makes a gradual, service-by-service migration possible. The last v1 publish was October 2025. New features land in v2 only.

Manual instrumentation and custom spans

Start the tracer once at application boot, then create spans around the operations you care about. In v2, Span is a struct referenced by pointer rather than an interface:

import "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"

func main() {
    tracer.Start(tracer.WithService("checkout-service"))
    defer tracer.Stop()

    span := tracer.StartSpan("process.order")
    // ... work ...
    span.Finish()

    child := span.StartChild("charge.card")
    child.Finish()
}

The migration guide describes the replacement: parent.StartChild("op") replaced v1’s tracer.StartSpan("op", tracer.ChildOf(parent)). For hot loops, v2 adds NewStartSpanConfig, WithStartSpanConfig, NewFinishConfig, and WithFinishConfig to reuse span configuration and cut functional-option allocation overhead.

Automatic instrumentation with Orchestrion

Orchestrion is Datadog’s compile-time instrumentation tool for Go: it rewrites your program at the AST level during compilation, so instrumented code passes through the normal compiler optimization pipeline and your source files stay untouched. It has been GA since v1.0.0, and the latest release is v1.11.0, published 2026-06-25. Setup is three steps:

go install github.com/DataDog/orchestrion@latest
orchestrion pin
orchestrion go build .

orchestrion pin writes an orchestrion.tool.go file that imports the Datadog integrations; commit it alongside go.mod and go.sum. Datadog’s docs and the Orchestrion README disagree slightly on which package that import points at — the tracer root (github.com/DataDog/dd-trace-go/v2) or the aggregate integration module (github.com/DataDog/dd-trace-go/orchestrion/all/v2) — so check what pin actually writes in your version rather than assuming. Either way you can replace it with imports of only the integrations you want, which trims transitive dependencies.

If prepending orchestrion doesn’t fit your build system, pass -toolexec 'orchestrion toolexec' to go build directly or set it in GOFLAGS. Two annotations control coverage: //dd:span creates a custom span around a function, and //orchestrion:ignore excludes a block from instrumentation. Orchestrion is not compatible with Bazel and rules_go; those builds need manual instrumentation.

For Docker builds, install the Orchestrion binary in your build stage and prepend it to the go build command there; the runtime image needs no change unless you enable AppSec, which adds shared-library requirements covered below.

Unified service tagging

Set three environment variables alongside tracer initialization. Datadog uses them to stamp telemetry across signals:

• DD_SERVICE: The service name; defaults to the process name if unset.
• DD_ENV: A global env tag applied to all telemetry from the process.
• DD_VERSION: The application version, which lets you compare error rates across deploys.

DD_TAGS accepts additional comma-separated key:value pairs. These same three values feed log correlation later, so set them once in your deployment manifest rather than per-signal.

Key features and capabilities

Beyond span collection, dd-trace-go bundles several capabilities:

• Framework integrations
• Runtime telemetry
• Profiling
• Log correlation
• Security monitoring
• A custom-metrics client

Each ships as a separate package or flag, and several bill separately.

Supported frameworks and contrib integrations

The contrib tree covers 50+ libraries, each following the import pattern github.com/DataDog/dd-trace-go/contrib/<package path>/v2. The integrations most Go services need:

import (
    httptrace "github.com/DataDog/dd-trace-go/contrib/net/http/v2"
    gintrace  "github.com/DataDog/dd-trace-go/contrib/gin-gonic/gin/v2"
    muxtrace  "github.com/DataDog/dd-trace-go/contrib/gorilla/mux/v2"
    sqltrace  "github.com/DataDog/dd-trace-go/contrib/database/sql/v2"
    redistrace "github.com/DataDog/dd-trace-go/contrib/redis/go-redis.v9/v2"
)

Beyond these, the contrib directory includes chi, Echo, Fiber, fasthttp, gRPC, four Kafka clients (sarama, confluent-kafka-go, segmentio/kafka-go, and twmb/franz-go, added in v2.9.0), pgx v5, GORM, MongoDB, gqlgen, and the AWS SDK. Orchestrion auto-instruments all of these, so the manual imports are only needed on the manual path.

Go runtime metrics

dd-trace-go v2 ships two runtime metrics systems, and the distinction matters if you have existing dashboards. The current system, runtime metrics v2, is on by default with tracer.Start() from v2.2.0 onward and reports under the runtime.go.metrics.* namespace: five goroutine-state metrics (live, runnable, running, waiting, not-in-Go), heap live bytes, heap goal, gc_heap_objects.objects, and a distribution of GC stop-the-world pause latencies in sched_pauses_total_gc.seconds. Turn it off with DD_RUNTIME_METRICS_V2_ENABLED=false.

The legacy system reports under runtime.go.*. Datadog disables it by default. Enable it with DD_RUNTIME_METRICS_ENABLED=true or tracer.WithRuntimeMetrics(), which collects every 10 seconds: num_goroutine, the mem_stats heap family, and GC pause quantiles in nanoseconds. If your alerts reference runtime.go.mem_stats.heap_alloc or the pause quantiles, keep legacy enabled until you migrate them to the v2 names.

Interpretation is where these earn their keep. A climbing sched_goroutines_waiting.goroutines count with flat request volume points at a leak or a blocked dependency; a rising gc_heap_live.bytes against a fixed heap goal predicts pause pressure before latency graphs show it.

Continuous Profiler for Go

The profiler is one import and one call:

import "github.com/DataDog/dd-trace-go/v2/profiler"

err := profiler.Start(
    profiler.WithService("checkout-service"),
    profiler.WithEnv("prod"),
    profiler.WithVersion("1.4.2"),
)
if err != nil {
    log.Fatal(err)
}
defer profiler.Stop()

The profiler enables CPU and heap profiles by default. The runtime execution trace is also on for amd64 and arm64. Enable block, mutex, and goroutine profiles with profiler.WithProfileTypes; the profiler package exposes the profile constants.

Two changes in v2.8.0 are worth knowing before you upgrade: the goroutine wait profile type was removed, which breaks any code that named it explicitly, and an experimental goroutine leak profile was added as an opt-in.

These map directly to the failure modes runtime metrics can only hint at:

• Heap profiles find the allocation site behind a memory leak.
• Mutex profiles quantify lock contention.
• The execution trace exposes scheduler behavior around GC pauses.

The profiler bills separately on top of the APM host charge.

Log and trace correlation

Datadog correlates a log line to a trace when the log carries five string fields: dd.trace_id, dd.span_id, dd.service, dd.env, and dd.version. For Go’s standard log/slog, the contrib/log/slog/v2 package injects them automatically:

import slogtrace "github.com/DataDog/dd-trace-go/contrib/log/slog/v2"

logger := slog.New(slogtrace.NewJSONHandler(os.Stdout, nil))
logger.InfoContext(ctx, "order processed")

WrapHandler wraps an existing handler instead of replacing it. One requirement trips people up: the handler pulls IDs via tracer.SpanFromContext(ctx), so logs must be emitted with a context carrying the active span. logger.Info(...) without a context produces uncorrelated logs. The handler also handles 128-bit trace IDs, which v2 generates by default as 32-character hex strings for W3C compatibility.

Application security (AppSec)

A common misconception: setting DD_APPSEC_ENABLED=true turns on AppSec. It doesn’t, on its own. Go AppSec requires the application to be compiled with Orchestrion (or instrumented manually with the AppSec SDK), and at runtime DD_APPSEC_ENABLED=true initializes the in-app WAF, header and body collection, and AppSec telemetry.

The appsec build tag is conditional, not mandatory. CGO hasn’t been required since tracer v1.53.0, but the WAF needs libc.so.6, libpthread.so.0, and libdl.so.2 at runtime. When CGO is enabled — the default on glibc-based images — those come along and no build tag is needed:

orchestrion go build -o=main .

When you build with CGO_ENABLED=0, Go would normally produce a static binary, which the WAF can’t use. That’s when the tag is required:

CGO_ENABLED=0 orchestrion go build -tags=appsec -o=main .

On Alpine, a glibc-built binary runs with apk add libc6-compat; a CGO-disabled Alpine build needs the appsec tag instead. The datadog.no_waf build tag disables AppSec entirely if these requirements are a problem. Supported frameworks include Gin, Echo, Chi, Gorilla Mux, Fiber, and net/http, and request blocking requires enabled Remote Configuration.

DogStatsD custom metrics

Custom metrics go through the separate datadog-go client, currently v5.9.0 (released 2026-06-24). statsd.New accepts three transports:

• UDP (127.0.0.1:8125)
• Unix Domain Socket (unix:///path/to/socket, Agent v6+, not available on Windows)
• Windows named pipes

With an empty address, the client resolves DD_DOGSTATSD_URL first, then falls back to DD_AGENT_HOST.

Client-side aggregation reduces packet volume before metrics leave the process. Since v5.0.0, basic aggregation is on by default for gauge, count, and set types with a 2-second flush interval; WithExtendedClientSideAggregation() extends it to histograms, distributions, and timings, which requires Agent 6.25.0+ or 7.25.0+. That option matters because metric volume and cardinality affect custom-metrics billing.

How Datadog APM for Go fits in

dd-trace-go is the agent-per-language model: every Go service imports the SDK, every Java service imports a different one, and each carries its own upgrade cycle, version compatibility matrix, and configuration surface. Datadog does use eBPF, but only in scoped products (Universal Service Monitoring, Network Monitoring, Cloud Security), and USM’s system-probe delivers RED metrics without an SDK rather than full traces.

OpenTelemetry offers one exit from per-vendor SDKs. dd-trace-go ships a native implementation of the OTel API at github.com/DataDog/dd-trace-go/v2/ddtrace/opentelemetry, so code written against go.opentelemetry.io/otel interfaces works with the Datadog tracer underneath. Datadog’s docs are direct about the constraint: “You should not install the official OpenTelemetry SDK or any OTLP Exporter packages. The Datadog SDK provides this functionality. Installing both can lead to runtime conflicts and duplicate data.”

Alternatively, run the OTel SDK itself and ship OTLP to the Agent (gRPC on 4317, HTTP on 4318, off by default) or to Datadog’s direct OTLP intake endpoint, which went GA on 2026-07-20. Since v2.8.0 the Datadog tracer can also export OTLP directly, bypassing the Agent entirely.

The third model skips per-service work entirely: node-level eBPF collection. groundcover’s Flora eBPF sensor deploys as a Kubernetes DaemonSet, one pod per node, and captures telemetry at the Linux kernel level.

• Deployment model: Flora runs once per node. No SDK import, no recompile, no restart.
• Captured protocols: It reconstructs HTTP requests and responses, SQL queries, and gRPC calls from network traffic.
• TLS visibility: Uprobes on SSL_write and SSL_read observe TLS payloads before encryption.
• Language coverage: The approach is language-agnostic for Linux services, so workloads become visible when they receive traffic.
• Benchmark caveat: In groundcover’s own benchmark against a Go 1.19 HTTP server at 3,000 req/s, Flora added 9% application CPU overhead and zero memory overhead, and consumed 73% less total CPU than the Datadog Agent. These are vendor-reported figures on a now-dated Go release; treat them as directional and benchmark against your own workload.

groundcover states the boundary directly: full waterfall traces still require instrumentation, “that’s the one thing eBPF doesn’t deliver on its own.” Kernel-level capture sees every network transaction but not internal code branches or business logic, which is why groundcover recommends layering eBPF coverage with OTel SDKs for the services where in-process spans matter, and ingests OTLP as a first-class data source alongside Flora’s signals. The economic structure differs too: Datadog bills per APM host plus span ingestion and indexing volume, while groundcover prices flat per node, so instrumenting more services or ingesting more spans doesn’t change the bill. If you want to test the node-level model against your own cluster, groundcover’s free tier includes the full BYOC (Bring Your Own Cloud) architecture, meaning your data plane runs in your own cloud account, with full-cluster visibility within hours of deploying Flora and no credit card required.

What’s next for Go observability

If you’re still on dd-trace-go v1, migration is overdue: the line is frozen, and every release since v1.74.0 exists only to bridge you to v2. The v2 breaking changes are mechanical but numerous:

• Module path: gopkg.in/DataDog/dd-trace-go.v1 becomes github.com/DataDog/dd-trace-go/v2, and contrib packages follow github.com/DataDog/dd-trace-go/contrib/<PACKAGE_DIR>/<PACKAGE_NAME>/v2.
• Types: Span and SpanContext are structs referenced by pointer, moved from ddtrace to ddtrace/tracer.
• Trace IDs: Now strings to support 128-bit IDs; TraceIDLower() returns the old uint64 form.
• Renamed options: WithServiceName becomes WithService, ChildOf becomes StartChild, and the sampling rule API collapses into SpanSamplingRules and TraceSamplingRules.

The v2fix tool (go install github.com/DataDog/dd-trace-go/tools/v2fix@latest, then v2fix -fix .) automates the import rewrites and deprecated-option replacements.

Two larger currents are reshaping the field. Datadog’s OTel convergence accelerated through 2025-2026: the DDOT Collector reached GA in 2025, and the direct OTLP intake now powers APM without Datadog-specific instrumentation. Zero-code instrumentation is arriving from two directions at once: compile-time rewriting (Orchestrion) inside the SDK model and kernel-level eBPF capture (Flora, plus scoped tools like Grafana’s Beyla) outside it. Platform teams now choose where instrumentation work happens: in the compiler with Orchestrion, or at the kernel with eBPF capture.