Understanding the Datadog Agent: architecture, deployment, and operation

Telemetry collected by the Datadog Agent passes through an on-host daemon before reaching Datadog. If you deploy the Datadog Agent, you operate this daemon on every monitored host or Kubernetes node, which means its architecture and resource footprint are your operational problem. So are its failure modes. First, define the Agent and trace its internals. Next, install and configure it across Linux, Windows, macOS, Docker, and Kubernetes. Then evaluate its operational features and alternatives, including groundcover’s Flora eBPF sensor. The article closes with troubleshooting and the FAQ.

What is the Datadog Agent

The Datadog Agent is an on-host daemon that collects metrics, logs, traces, and events from the host it runs on and forwards them to the Datadog backend over HTTPS. It is the primary instrumentation mechanism for the Datadog platform: one Agent per host or Kubernetes node, running alongside your workloads.

The Agent is open source. The DataDog/datadog-agent repository on GitHub carries the full source under the Apache License 2.0, with one exception: Datadog licenses the BPF code under GNU GPL v2, which prohibits incorporation into proprietary programs.

Datadog ships new Agent releases multiple times per month. The stable version at the time of this research was 7.82.2, released 2026-08-19; Datadog built it with Go 1.26.6 and embedded Python 3.13.15. Check the changelog for the current release before pinning a version.

How the Datadog Agent works

The Agent runs a check-and-forward loop. Integration checks execute on a 15-second default collection interval, the Agent aggregates the results locally, and an HTTP forwarder ships payloads to Datadog’s intake endpoints. When intake is unreachable, the forwarder queues payloads in memory and retries with exponential backoff: base 2 seconds, maximum 64 seconds, reached after 6 retries. The forwarder immediately drops transactions that receive HTTP 400 or 413 responses; 5xx and other 4xx responses trigger rescheduling, as detailed in the Agent retry documentation.

The Agent’s behavior depends on the signals and processes involved. The deployment platform determines how those processes run.

Telemetry it collects

Each signal type the Agent collects feeds a distinct Datadog product with its own pricing meter:

• Metrics: System and integration metrics feed Infrastructure Monitoring, which covers host maps, container maps, and the Orchestrator Explorer. Custom metrics arrive through DogStatsD, the Agent’s local StatsD server, which aggregates on a 10-second flush interval: gauges send the latest datapoint, counts send the sum. Datadog documents this behavior in its DogStatsD aggregation documentation.
• Traces: APM traces are SDK-based.
• Logs: The Logs Agent tails files and listens on TCP/UDP, feeding Log Management under Datadog’s “Logging without Limits” model of decoupled ingestion and indexing.
• Events: Kubernetes and integration events feed event streams and correlate with the other signals.

One architectural point matters for anyone evaluating the Agent against eBPF-native tools: Datadog uses eBPF in Universal Service Monitoring. Network and Security products also use it, but eBPF is not a universal collection mechanism. Application-level traces still come from language SDKs.

Internal architecture and sub-processes

“The Agent” consists of several processes. On a typical host or node you run:

• Core Agent: Runs integration checks, DogStatsD, the Logs Agent, and the forwarder.
• Trace Agent: Receives spans from application SDKs. In non-Kubernetes environments it defaults to a maximum of 0.5 CPU cores, and the Agent kills the APM process when memory hits 150% of the configured max_memory.
• Process Agent: Collects process and container-level data for container maps and the Orchestrator Explorer.
• System Probe: Runs the eBPF-based components for network and security monitoring.

On Kubernetes, a fifth component changes the topology. The Cluster Agent runs as a Deployment, not a DaemonSet, and solves an API-server load problem. Without it, every node Agent queries the Kubernetes API server independently, which Datadog found “put increasing load on the API server and etcd as the size of the cluster increased.”

With the Cluster Agent in place, node Agents cannot talk to the API server at all, according to Datadog’s Cluster Agent setup. The Cluster Agent proxies cluster-scoped data, caches metadata every 30 seconds, and relays it to node Agents over HTTPS on port 5005. A 32-character token authenticates the connection.

The Cluster Agent also dispatches cluster checks so each runs exactly once cluster-wide and serves HPA external metrics. Its admission controller mutates pod specs at scheduling time. The Datadog Helm chart has enabled the Cluster Agent by default since chart version 2.7.0 in January 2021, as described in Datadog’s Cluster Agent architecture.

Supported platforms and architectures

The Agent 7 support matrix covers Linux, Windows, macOS, Docker, and Kubernetes. Specifics as of Agent 7.82.2:

• Linux (64-bit x86): Amazon Linux 2/2022/2023, Debian 8+, Ubuntu 14.04+, RHEL/CentOS/Oracle Linux 7+, AlmaLinux/Rocky 8+ (Agent 7.33.0+), SUSE 12+, Fedora 26+.
• Linux (64-bit ARM v8): Amazon Linux 2, Debian 9+, Ubuntu 16.04+, RHEL/CentOS/Oracle 8+, Fedora 27+, all from Agent 7.16.0. ARMv7, s390x, and ppc64le do not appear in the support matrix.
• Windows: Server 2016, 2019, 2022, and 2025 with full support; Server 2012/R2 capped at Agent 7.46.0. Desktop 10 and 11 fully supported.
• macOS: 12.0+ on Intel from Agent 7.39.0; 12.0+ on Apple Silicon from Agent 7.70.0.
• Docker: The datadog/agent image publishes to Docker Hub, registry.datadoghq.com, Google Artifact Registry, Amazon ECR, and Azure ACR, in amd64 and arm64 variants plus Windows Server Core and LTSC builds.
• Kubernetes: GA support through the Datadog Operator or Helm, plus DaemonSet manifests.

The supported platform determines which installation path you use.

Installing and configuring the Datadog Agent

Install paths depend on where the Agent lands. The four main routes:

1. One-line host install (Linux): DD_API_KEY="<KEY>" bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)". macOS uses install_mac_os.sh with DD_AGENT_MAJOR_VERSION=7; Windows uses the datadog-agent-7-latest.amd64.msi installer run as Administrator.
2. Docker: Pull datadog/agent:7 (or registry.datadoghq.com/agent:7) and run it as a container with your API key in the environment.
3. Helm: The Datadog chart deploys the node Agent as a DaemonSet plus the Cluster Agent.
4. Datadog Operator: Manages Agent deployment declaratively through a Kubernetes custom resource, which fits GitOps workflows.

Configuration lives in datadog.yaml. Exactly one field is strictly required: api_key, which associates the Agent’s data with your organization. A second field, site, is effectively required for any deployment outside US1 because it defaults to datadoghq.com; EU deployments need datadoghq.eu, while other regions have their own values. The Agent configuration documentation lists these default file paths:

A single host is the easy case. Fleets are where Agent management becomes a project of its own.

Deploying at scale

Datadog supports the standard configuration-management tools for large fleets, each with a current recommended artifact:

• Ansible: The datadog.dd collection (role datadog.dd.agent) is current; the older standalone datadog.datadog role is explicitly labeled legacy. Requires Ansible 2.10+.
• Chef: The datadog cookbook on Chef Supermarket, requiring chef-client 12.7+.
• Puppet: The datadog_agent module on Puppet Forge; v4.x requires Puppet 7.34.x or later and drops Agent v5 support.
• Fleet Automation: Datadog’s built-in fleet manager for remote upgrades and policy-based configuration. Core capabilities went GA on December 18, 2025, but scope matters: remote upgrades cover Linux and Windows VMs only, need Agent 7.69+ (Linux) or 7.71+ (Windows) plus 4 GB of disk headroom, and remote container configuration remains in Preview. If your fleet is mostly Kubernetes, Fleet Automation does not yet manage it end to end.

Kubernetes uses a fixed deployment pattern: node Agents run as a DaemonSet, one pod per node, with the Cluster Agent Deployment brokering API-server access as described earlier. Helm and the Operator both encode this pattern, so scaling to more nodes means scaling the DaemonSet, not adding configuration.

Agent 7 vs Agent 6

Run Agent 7 unless a Python 2 custom check blocks you. The difference between the major versions is the embedded Python runtime: Agent 7 supports only Python 3 for integrations and custom checks, while Agent 6 (from v6.14.0) bundles both Python 2 and Python 3, selectable via python_version in datadog.yaml or the DD_PYTHON_VERSION environment variable. Datadog documents the runtime split in its Agent version guide.

Migration from 6 to 7 has one real gate: custom check compatibility. Datadog’s guidance is to verify checks with pylint --py3k, or flip Agent 6.14+ to the Python 3 runtime as a test before upgrading. The upgrade does not migrate custom checks automatically, by design, because Datadog cannot guarantee backwards compatibility.

Image tags follow the split. latest and 7 track the newest Agent 7 release; latest-jmx and 7-jmx add a JVM for JMXFetch. The latest-py2 and latest-py2-jmx tags are Agent 6 Python 2 builds, not Agent 7 variants, and py2 tags no longer appear in currently observed Docker Hub listings. Since versions move monthly, verify the current tag state before standardizing your manifests.

Key features and capabilities

Four operational characteristics decide whether the Agent fits your fleet: its resource footprint, its metric resolution, its behavior during outages, and its tagging model.

Resource overhead starts small and grows with configuration. Datadog’s published Agent baseline for Agent v7.34.0 on an AWS c5.xlarge is ~0.08% CPU and ~130 MB RSS with a baseline configuration that enables only the process check, and 830–880 MB on disk for Linux.

Enabling integrations and JMX checks raises consumption. Running the trace and process Agents adds more. The Trace Agent alone scales from 0.05 cores and 35 MB at 2,000 spans/second to 2.0 cores and 130 MB at 130,000 spans/second, according to Datadog’s APM resource figures.

On GKE Autopilot, Datadog’s documented resource requests total 500m CPU and 1,056Mi memory across the Agent, Trace Agent, Process Agent, and System Probe containers. Datadog warns that Autopilot’s 50m/100Mi defaults “may lead the Agent container to quickly OOMKill.”

Practitioners have hit worse in specific setups: an Agent 7.57.0 issue reported progressive memory growth on small ECS Fargate tasks that 7.56.2 did not show. Datadog has been cutting the on-disk side; its binary reduction report shows that between v7.60.0 and v7.68.0 the core Agent binary shrank from 236 MiB to 103 MiB.

Metric resolution is the Agent’s strongest argument over API polling. Checks default to 15-second collection, but min_collection_interval goes down to 1 second for integrations and custom checks. StatsD metrics also support 1-second resolution, and Datadog stores metric data at 1-second granularity, according to its collection resolution guide.

Buffering behavior differs sharply by signal type, and the asymmetry matters for incident planning:

• Metrics: In-memory retry queue capped at 15 MB by default; metrics drop beyond that. Datadog disables on-disk spillover (Agent 7.27.0+) by default; when you enable it, the Agent buffers up to a reference 2 GB and writes only while disk usage sits below 80%. In-memory payloads die with an Agent restart; on-disk payloads survive and resend. Datadog specifies these defaults in its Agent network configuration.
• File-tailed logs: No buffer needed. The Agent applies backpressure, stops reading, and resumes from the registry offset when intake returns. The main loss window is file rotation: tailing continues only 60 seconds after a rotation.
• TCP/UDP logs: A 100-line buffer with no persistence; new lines drop when it fills. Datadog’s log reliability documentation covers both log-input behaviors.

Tagging ties the signals together. On Kubernetes, the Cluster Agent’s admission controller injects Datadog’s standard env, service, and version tags from application labels at pod scheduling time, along with DD_AGENT_HOST and DD_TRACE_AGENT_URL, so telemetry from the same workload correlates without per-service configuration. Datadog documents the injection mechanism in its admission controller guide.

How the Datadog Agent fits in

Agent checks versus API polling

For Datadog Agent checks versus CloudWatch API polling, local collection delivers lower latency and local outage handling; the cost is running and maintaining the Agent itself. The gap is measurable: Datadog’s own crawler polls AWS CloudWatch every 10 minutes, producing 15–20 minutes of end-to-end latency for standard 5-minute AWS metrics, against roughly 15 seconds for Agent checks. Datadog explicitly recommends installing the Agent to get system metrics “with virtually zero delay,” according to its cloud metric delay guide. API-only collection also has no local buffering story; the retry queue and log backpressure described above exist only because a local process holds state.

The Agent model carries an operational cost. One Agent per host means core Agent, Trace Agent, Process Agent, and System Probe on every node, SDK instrumentation for every traced service, and a Cluster Agent to keep the fleet from overwhelming the API server. groundcover offers a contrasting eBPF-native architecture.

Flora collection

groundcover’s Flora eBPF sensor deploys as a single DaemonSet, one pod per node, and reads telemetry directly from the Linux kernel. Flora captures network calls, HTTP requests, logs, metrics, traces, and Kubernetes events. Flora requires no SDKs or language-specific agents. It also requires no application restarts. Every service on the cluster becomes visible within hours of deployment, with zero application instrumentation.

One honest caveat, acknowledged by groundcover itself: kernel-level capture does not produce application-internal business-context spans, so waterfall traces require instrumentation. groundcover accepts OpenTelemetry as a first-class source alongside Flora data for exactly that case.

Resource comparison

The resource comparison has independent support. The New Stack’s May 2023 benchmark measured Flora at +9% CPU and +0% memory overhead versus +249% CPU and +227% memory for the Datadog agent, with Flora consuming 73% less total CPU.

BYOC architecture and pricing

Architecture differs as much as collection. groundcover’s default deployment is BYOC (Bring Your Own Cloud, where your data plane runs in your own cloud account). Telemetry lands in ClickHouse for logs and traces. ClickHouse also stores events, while VictoriaMetrics stores metrics inside your VPC. groundcover manages only the control plane for UI and orchestration.

Pricing is flat per node, with no per-GB ingestion or custom-metric charges. groundcover also charges no per-user fees, so collecting everything carries no volume penalty. groundcover’s Datadog pricing guide documents a team running roughly 700 Kubernetes nodes with 5 TB/day of logs and 500K custom metrics paying $2.54M/year on Datadog, versus $297K/year on groundcover, an 87% reduction with full APM enabled and no trace sampling.

The figure comes from groundcover, and no independent third party has audited it. The mechanism behind it, flat per-node billing against Datadog’s per-host, per-GB, per-metric, and per-span meters, is what a skeptical reader can evaluate. If you want to validate the architecture on your own cluster, the free plan includes BYOC and requires no credit card. Deploy Flora on one cluster and evaluate full-cluster visibility within hours.

Production failures put either collection model under pressure. Diagnosing the Datadog Agent starts with its built-in flare command.

Troubleshooting the Datadog Agent

The primary support tool is the flare command, which gathers the Agent’s configuration files and logs into an archive for Datadog support. It collects integration configs from conf.d paths, Agent logs, APM debug logs when you enable tracing, environment variables, expvar runtime data, and system-probe diagnostics. The Agent scrubs passwords, API keys, proxy credentials, and SNMP community strings before upload. The flare prompts for confirmation before sending; decline and it prints the local archive path instead. Datadog documents the archive contents and review flow in its flare troubleshooting guide.

Invocation differs by platform:

Omit the case ID and the Agent creates a new support case from your login email. On Kubernetes, the --local flag targets a specific component such as the trace-agent or process-agent. Operators can also trigger flares remotely from the Fleet Automation UI, though not from the Kubernetes Fleet view and not on GovCloud sites.

For day-to-day diagnosis before opening a case, the Agent CLI exposes status for check and forwarder health, check <name> for running a single integration in the foreground, and restart for cycling the service. Configuration lives at the datadog.yaml paths listed in the install section, and the flare bundles the corresponding Agent logs if you need to inspect them without hunting through the filesystem.