Self-hosted LLM observability: security benefits and deployment architecture

In January 2025, Wiz Research found an exposed DeepSeek database exposing plaintext chat history, API keys, and backend operational metadata to anyone who connected. LLM trace data carries exactly that payload: full prompts, completions, tool arguments, retrieved documents. Route it to a SaaS observability vendor and every one of those records crosses your network boundary, becomes subject to a third party's access controls, and inherits their breach surface.

What is self-hosted LLM observability

LLM observability is the capture and analysis of traces, token usage, latency, and cost across model calls. Unlike a database query, an LLM call is non-deterministic: the same endpoint returns variable token counts, variable latency, and variable cost on every invocation, so you need per-call records, not aggregate dashboards. Traditional APM targets deterministic call patterns. Prometheus is a metrics system; traces, logs, and LLM span data require separate backends.

The records LLM observability produces are different from APM spans in one legally meaningful way: they can contain the content of the interaction. A prompt with a customer's name and a health question is personal data under GDPR Article 4(1), and Article 4(2) counts storage in a log as processing. The same prompt is PHI under 45 CFR 160.103 if it relates to health and includes an identifier.

Self-hosted LLM observability keeps the whole trace pipeline, from instrumentation through storage and query, inside your network boundary. No third party receives prompt or completion content, which removes an entire category of processor and business-associate obligations before you write a single control.

How self-hosted LLM observability works

A self-hosted deployment starts with the data plane boundary and telemetry capture path. Trace structure determines whether the system can explain RAG and agent failures. Storage sizing follows from those choices.

Deployment architecture options

Fully on-premise means you operate everything: sensors, storage, UI, and authentication, on infrastructure you control. It carries the highest operational burden and the tightest boundary.

BYOC (Bring Your Own Cloud) splits the system: the data plane, meaning telemetry storage and compute for ingestion, runs in your own VPC, while the vendor manages the control plane for UI and orchestration. The mature implementations use an outbound-only agent pattern: a component inside your environment pulls instructions from the vendor's control plane, so the vendor holds no standing inbound access to your account. The test for whether a "BYOC" pitch is real is blunt: can the vendor see the payload of a request to your application? The BYOC test gives the answer for real BYOC: no.

Air-gapped deployments put everything, including authentication, inside the customer environment with no external connectivity. groundcover supports all three models: BYOC is the default architecture at every tier including the free plan, with on-premises and air-gapped options for regulated environments. Verify the boundary for any tool you evaluate, because labels drift: LangSmith's self-hosted deployment, when not air-gapped, requires beacon egress for license verification and usage reporting.

Tracing structure for LLM and agent pipelines

The OTel conventions organize LLM traces as spans grouped into sessions, and they attach individual observations to each span. The OpenTelemetry GenAI semantic conventions define the span vocabulary: inference spans named {gen_ai.operation.name} {gen_ai.request.model}, RAG retrieval spans named retrieval {gen_ai.data_source.id}, tool executions as execute_tool {gen_ai.tool.name}, and agent invocations as invoke_agent {gen_ai.agent.name}. Multi-turn sessions correlate through the gen_ai.conversation.id attribute.

Span-level structure is what makes RAG and multi-agent pipelines debuggable. A retrieval span carries gen_ai.retrieval.query.text and the retrieved documents; a tool span carries gen_ai.tool.call.arguments and gen_ai.tool.call.result. Those fields are also exactly where sensitive data concentrates, which is the architectural argument for keeping the spans in your own storage.

Instrumentation: OpenTelemetry and eBPF

Two instrumentation paths feed a self-hosted backend. The first is OpenTelemetry: SDKs or framework hooks emit gen_ai.* spans, an OTel Collector inside your VPC receives them over OTLP, and it exports to your in-boundary storage. Note that every gen_ai.*attribute currently carries Development status. The OTel v1.37.0 release replaced per-message events with gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions attributes. The OTel project is explicit about content defaults: "By default, no prompt content or tool arguments are captured with GenAI telemetry, as these can contain sensitive data." Content capture is opt-in, a decision you can afford to make when the destination is your own cluster.

The second path is eBPF. The standard mechanism attaches uprobes to SSL_write and SSL_read, reading the plaintext buffer from user space before encryption, which yields per-call visibility into LLM API traffic with zero code changes. Practical requirements: Linux kernel 5.8+ with BTF enabled for capability-scoped deployment, and unstripped Go binaries, since Go's statically linked crypto/tls needs the symbol table for probe attachment. groundcover's Flora eBPF sensor deploys as a single DaemonSet and captures per-call LLM telemetry at the kernel level: models, cost, latency, prompts, agent behavior, and tool execution, with privacy controls to strip content at capture time.

Storage and infrastructure requirements

ClickHouse is a common backend for high-volume trace and log/event storage. Realistic baselines from official sizing guidance:

• ClickHouse compute: 32 GB RAM or more is the recommended baseline; observability workloads need roughly 1 vCPU per 10 MB/s of sustained ingest, with ~10x compression on stored telemetry.
• Minimum viable trace store: Langfuse documents 2 CPU and 8 GiB memory as minimum ClickHouse resources per service, with 3 replicas and a 100Gi persistent volume as the production starting point.
• PostgreSQL: most LLM observability stacks pair ClickHouse with PostgreSQL for transactional metadata; Traceloop's all-in-one Helm deployment ships PostgreSQL plus ClickHouse and Kafka together.
• VictoriaMetrics: for the metrics side, compressed samples average ~1 byte each, and the project recommends keeping 50% of RAM free and at least 20% of disk free as headroom.
• Object storage: MinIO provides S3-compatible object storage for tiered setups; ClickHouse documents MinIO compatibility across all S3 functions and tables.

Security and compliance benefits of self-hosting

SaaS observability platforms typically send telemetry to vendor-managed infrastructure unless you deploy a BYOC or self-hosted data plane. Documented incidents show the failure modes:

• LangSmith SDK redaction bypass: CVE-2026-41182 showed LangSmith's SDK redaction controls failing to apply to streaming token events, so raw token values reached the backend despite hide_outputs being set.
• Amazon Bedrock AgentCore SDK logging: CVE-2026-15737 showed the Amazon Bedrock AgentCore SDK logging raw user prompts and agent responses into OTel span attributes in CloudWatch Logs.
• New Relic staging breach: in 2023, an unauthorized actor executed search queries and exfiltrated results from New Relic's staging environment.

Defaults compound the exposure: Cloudflare AI Gateway stores prompts and model responses under its logging defaults, and Coder's AI Gateway retains captured prompts for 60-day retention by default.

Self-hosting changes your regulatory posture under three specific frameworks:

• GDPR: an observability vendor receiving trace data is a processor requiring a binding DPA under Article 28, an entry in your Article 30 records, and a mechanism to delete a specific user's data from observability logs under Article 17. Keeping traces in your own infrastructure removes the processor relationship entirely; you still owe data minimisation (Article 5(1)(c)) and storage limitation (Article 5(1)(e)), but you enforce them on systems you control.
• HIPAA: any platform that receives PHI-containing prompts meets the business associate definition and requires a BAA, with a 60-day breach notification duty flowing back to you. The Safe Harbor de-identification standard removes 18 identifier categories, and IP addresses are on the list, which puts ordinary trace metadata in scope.
• SOC 2 Type II: the audit covers operating effectiveness of controls over a period. The criteria most exposed by trace pipelines are CC6.1 (logical access over protected information assets), CC6.7 (restricting transmission and movement of information), and CC7.2 (monitoring for anomalies). An architecture where prompt data never transits to a third party gives your auditor a shorter path through CC6.7 than a DPA plus a vendor's subprocessor list.

One more boundary deserves attention: embeddings. The Vec2Text paper demonstrated 92% exact recovery of 32-token inputs from their embeddings, and recovered 89% of full names from MIMIC-III clinical note embeddings. Treat vector stores and embedding spans as regulated data.

These processor, BAA, and audit-boundary mechanics drive regulated teams toward BYOC. The boundary extends to AI-assisted investigation: groundcover's AI Agent Mode runs Amazon Bedrock inside your own AWS account, so telemetry never leaves your environment for AI analysis. If you want to validate the boundary on your own cluster, groundcover's free tier includes the full BYOC architecture; your trace data stays in your VPC from day one.

Security hardening controls

A self-hosted deployment shifts control implementation to you. The controls worth specifying up front:

• Network isolation: run the data plane in a dedicated VPC, and if using BYOC, confirm the vendor connection is outbound-only with no standing inbound access.
• SSO: verify tier gating before committing. Langfuse ships OIDC SSO free but does not support SAML 2.0; Helicone gates SAML behind Enterprise; groundcover provides SSO via OIDC/SAML across tiers.
• RBAC scoped by area and environment: groundcover's RBAC scopes policies by namespace and cluster, with data-type filters. Langfuse gates project-level RBAC roles behind its Enterprise license key, and Phoenix disables authentication by default in self-hosted deployments, offering three roles once enabled.
• PII controls at ingestion: OTel's opt-in content capture means prompts stay out of spans unless you decide otherwise. Pair that default with ingestion-time redaction in the trace pipeline you operate.
• Retention and backups: GDPR's storage limitation principle applies to your own logs too. Set explicit TTLs; ClickHouse system log tables such as trace_log have no default TTL and quietly consume disk.

Key capabilities to evaluate

After you verify the deployment boundary, test whether the platform can still do the observability job. Evaluate four capability areas:

• Tracing depth: full pipeline spans covering RAG retrieval, tool calls, and agent invocations, with session correlation across turns via conversation identifiers. Anything that stops at request/response pairs will not explain a multi-agent failure.
• Evaluation: LLM-as-judge scoring of outputs, and faithfulness scoring that checks whether a response is grounded in the retrieved context, which is the practical mechanism for hallucination detection in RAG pipelines. In a self-hosted setup, confirm where the judge model runs, since evaluation traffic carries the same sensitive content as production traffic.
• Cost tracking: per-call token accounting via gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, aggregated per model and user, then per session. Session-level cost attribution depends on the same conversation IDs that power trace correlation.
• Prompt management and versioning: versioned prompts with controlled promotion between environments. Check the license gate: Langfuse puts Protected Prompt Labels, audit logs, and data retention policies behind its Enterprise license key even in the self-hosted MIT-core product.

How self-hosted LLM observability fits in

Tool choice and deployment method interact: a platform is only as self-hostable as its production deployment path.

Deploying with Docker Compose and Kubernetes

Docker Compose is the standard entry point. Langfuse supports Docker Compose for low-scale deployments, and Helicone recommends it for setups and local development. Neither vendor positions Compose as production infrastructure.

Production means Helm on Kubernetes. Langfuse publishes a verified chart, v1.5.41 as of 2026-07-28. Helicone's current charts live in [Helicone/helicone-helm-v3](https://github.com/Helicone/helicone-helm-v3), split into core, ai-gateway, infrastructure, and monitoring charts. Phoenix ships an OCI chart via Docker Hub. For infrastructure as code, Langfuse released Terraform modules for AWS, Azure, and GCP in May 2025, Helicone documents AWS modules for EKS, S3, and Aurora, and LangSmith's Terraform path provisions VPC, EKS, RDS, ElastiCache, S3, and IAM before Helm installs the application.

Framework integrations

Application-side instrumentation flows through three common frameworks. LiteLLM uses callback lists: litellm.success_callback = ["langfuse"] or ["helicone"], with a CustomLogger class recommended for async workloads. LangChain passes handlers through run config, config={"callbacks": [langfuse_handler]}, built on BaseCallbackHandler in langchain_core. LlamaIndex has moved to its instrumentation module with dispatchers and span handlers, while set_global_handler("arize_phoenix") remains the one-line path for Phoenix.

OTLP ingestion is the escape hatch from all of this. Langfuse added a native OTel endpoint in v3.22.0, and groundcover ingests OTLP spans as a first-class source alongside Flora's eBPF signals, so services already instrumented with OTel SDKs coexist with kernel-captured data in the same backend.

Self-hostable tools compared

Scope the comparison to tools with a genuine self-hosting path, and read the license file rather than the marketing page:

The Phoenix license row shows why verification matters. The project markets itself as fully open source, but ELv2 is source-available with a hosting restriction, a material constraint if you plan to expose Phoenix as an internal managed service.

What's next for self-hosted LLM observability

Agent observability is where the standards work is concentrated, and none of it is stable yet. The OTel GenAI agent spans (invoke_agent, execute_tool, create_agent) all carry Development status, and cross-trace session correlation remains an open problem: Jaeger maintainers note that a 10-turn conversation with a tool-using agent generates 10 separate traces, and OTel spec issue #4993 proposes CONTINUATION links for long-running executions. Long-running agent sessions put pressure on trace storage and correlation: Anthropic reports agent conversations spanning hundreds of turns, and LangSmith caps a single trace at 25,000 runs.

MCP is becoming the interface between observability data and AI investigation. The MCP spec dated 2026-07-28 formally documents W3C Trace Context propagation in _meta and deprecates protocol-level logging in favor of stderr or OpenTelemetry. The groundcover MCP Server connects observability data to external AI agents, IDEs, and developer workflows, so AI-assisted investigation can use production context without moving telemetry out of your account.

For self-hosting teams, the question to press on each roadmap is whether evaluation workloads will run inside the same boundary as the traces they score.

‍