OpenTelemetry semantic conventions standardize how services name attributes, metrics, and spans, so a field like http.response.status_code has the same meaning across instrumentations. When an instrumentation upgrade adopts stable conventions, attribute names, metric units, and span names can change. If services upgrade at different times, dashboards and alerts may stop counting requests from part of your fleet, even while the telemetry continues reaching your backend.
A safe migration starts by identifying which conventions each service emits and establishing a baseline for your existing queries. You can use OTEL_SEMCONV_STABILITY_OPT_IN for dual emission in instrumentations that support it, or use the OpenTelemetry Collector to normalize supported changes. This guide explains how to migrate HTTP and database conventions, update dashboards and SLOs, compare request rates, error rates, and latency, and retire legacy names without losing visibility.
Why OpenTelemetry Semantic Convention Migrations Break Observability
An instrumentation upgrade can change the fields your monitoring queries depend on without interrupting telemetry collection. The HTTP migration replaces http.status_code with http.response.status_code and changes the server duration metric from http.server.duration in milliseconds to http.server.request.duration in seconds. The database migration also changes attribute names and values, such as db.system="mssql" becoming db.system.name="microsoft.sql_server".
Suppose checkout-api emits http.response.status_code after an upgrade, while payments-api still emits http.status_code. A dashboard filtering only on the stable attribute counts checkout’s 5xx responses but misses those from payments. Both services can be exporting telemetry correctly; the query simply doesn’t match one service’s attributes. A latency alert can also become inaccurate if you switch to the seconds-based histogram without converting its threshold from milliseconds.
This is different from telemetry actually being dropped. Check the raw spans and metric series before changing dashboards. If the legacy data is present, update your queries or normalize the conventions in the Collector. If it’s missing, inspect the instrumentation and Collector pipeline for export failures or transformations that removed it.
Which HTTP and Database Semantic Conventions Changed?
HTTP and database semantic conventions can differ across instrumentation versions, affecting attribute names, values, metric names, and units. Before making changes, identify the fields your services emit and compare them with those used in your dashboards, alerts, and SLO queries.
HTTP Semantic Convention Changes
HTTP instrumentation can emit different attribute and metric names for the same request data, affecting dashboards, alerts, and queries. The table below shows examples of changes you may encounter when migrating HTTP semantic conventions.
The duration changes affect latency alerts and SLO calculations. A threshold of 300 milliseconds must become 0.3 seconds when querying the stable histogram. Histogram boundaries also change, so percentile estimates may differ. On server duration metrics, server.address and server.port are opt-in rather than included by default.
HTTP span names and captured header attributes can change too. Server spans use the HTTP method and a low-cardinality route when available, such as GET /orders/{id}, while client spans typically use the method alone, such as GET. Unknown methods represented by _OTHER use HTTP as the span name. Header attribute names also preserve hyphens, so a legacy key such as http.request.header.content_type becomes http.request.header.content-type. Check queries that filter on span names or headers.
Database Semantic Convention Changes
Database instrumentation can also emit different attribute names, values, and metrics for the same operations. The table below shows examples of changes you may encounter when migrating database semantic conventions.
Changing db.system to db.system.name also requires updating filter values: mssql, oracle, and db2 become microsoft.sql_server, oracle.db, and ibm.db2. The db.namespace attribute requires database-specific handling rather than a direct rename from db.name. SQL databases can combine the database and schema using |, while Redis represents the database index as a string. Confirm the actual namespace format before updating your queries or Collector transformations.
When migrating db.statement to db.query.text, sanitize non-parameterized queries to prevent sensitive literal values from being collected. Parameterized query text can retain its placeholders, while parameter values remain opt-in; use db.query.summary for low-cardinality grouping instead of grouping metrics by full SQL statements. The database conventions define db.client.operation.duration as a recommended histogram measured in seconds, with no direct legacy equivalent. Verify that your instrumentation emits it before creating queries. Database connection-pool metrics remain in development, and their names and units can vary across instrumentation versions.
Database client span names also change during migration. Legacy SQL spans could use names such as SELECT shop.orders, while stable conventions prefer db.query.summary when available. Otherwise, the span name uses db.operation.name with a low-cardinality target such as db.collection.name or db.namespace. Check dashboards and trace searches that filter on database operation names.
Migrating Instrumentation With OTEL_SEMCONV_STABILITY_OPT_IN
OTEL_SEMCONV_STABILITY_OPT_IN lets supported OpenTelemetry instrumentation emit stable semantic conventions while preserving older attributes and metrics during migration. You can enable both sets temporarily, update the queries that depend on legacy names, and then switch to stable-only output. However, support depends on the instrumentation package and version, so check what your services currently emit before setting the variable.
The variable accepts comma-separated values. Use http/dup or database/dup to request both legacy and stable conventions, or http and database to request stable conventions only. When an instrumentation supports both a value and its /dup variant, /dup takes precedence.
For example, with compatible Python instrumentation packages that support both HTTP and database migration, enable dual emission using:
Restart the instrumented service and inspect its exported spans and metrics. Confirm that both attribute sets appear where expected and that the old and new duration metrics are available before updating dashboards or alerts. Do not assume the variable worked simply because the service started successfully.
Support varies across languages and instrumentation packages:
Dual emission can increase telemetry volume. Spans can carry both legacy and stable attributes, while HTTP metrics may be emitted under separate names such as http.server.duration and http.server.request.duration. Avoid counting both metric families as independent requests, and compare their values using the correct units.
Once your queries, alerts, and recording rules use the stable conventions, replace /dup with the stable-only values where the instrumentation supports them. Verify that request rates, error rates, and latency remain consistent before removing legacy references. For instrumentation that does not support the variable, use Collector-side normalization instead of relying on an unsupported setting.
Migrating Telemetry With the OpenTelemetry Collector
The OpenTelemetry Collector can normalize semantic convention differences before telemetry reaches your backend, allowing services with different instrumentation versions to coexist. Use the Schema processor when telemetry includes a valid schema_url, or the Transform processor when you need explicit attribute mappings. Neither method automatically handles every change in attribute values, metric units, or histogram boundaries.
Using schema_url and the Schema Processor
schema_url identifies the semantic convention version associated with a telemetry resource or instrumentation scope. The Schema processor uses this value to apply supported renames and normalize telemetry to a configured target version.
By default, translated attributes replace their legacy names. During migration, the migration option preserves both names for renames between the specified from and target versions, allowing existing queries to continue working while you update them. For example, when translating database spans from schema 1.24.0 to 1.26.0, it can preserve db.operation alongside db.operation.name.
The Schema processor is currently alpha, so validate its behavior before introducing it into a production pipeline. It also cannot infer a source version when telemetry has no usable schema_url.
Using the Transform Processor for Attribute Renames
The Transform processor uses OpenTelemetry Transformation Language (OTTL) to modify telemetry based on explicit conditions. It is useful when schema_url is missing or when you need to normalize a specific attribute without translating an entire schema version.
The following configuration combines both processors in a trace pipeline. The Schema processor runs first, while the Transform processor copies http.status_code to http.response.status_code only for payments-api spans that lack the stable attribute.
Save the configuration as collector.yaml, then validate and start it using the OpenTelemetry Collector Contrib distribution:
This is a local trace-validation configuration: the receiver listens on localhost, and the debug exporter prints spans instead of forwarding them to your backend. Keep detailed debug output out of production because span attributes may contain sensitive information. Once the transformations are verified, add the processors to your existing trace pipeline while retaining its production exporter.
The OTTL rule preserves the legacy attribute and avoids overwriting an existing stable value. Its error_mode: ignore setting logs statement errors without dropping the payload; propagate can cause the Collector to drop it. Keep the original attribute until the dashboards and alerts that depend on it have been updated.
Handling Missing schema_url and Verifying Transformations
When telemetry has no schema_url, the Schema processor skips translation rather than guessing which conventions the service uses. Inspect the debug output for service.name, the instrumentation scope name and version, and any resource or scope schema URL. Match these details to the instrumentation package before adding a scoped OTTL rule.
Avoid applying global renames when the old attribute’s meaning depends on the operation. For example, net.peer.name should not be blindly rewritten across client and server spans because the target address attributes describe different endpoints. Similarly, changing db.system to db.system.name requires mapping values such as mssql to microsoft.sql_server, while renaming db.statement to db.query.text does not sanitize sensitive SQL literals.
Metric migrations need separate verification. A schema-based rename from http.server.duration to http.server.request.duration does not convert histogram values from milliseconds to seconds or change their bucket boundaries. Do not apply that rename to a metrics pipeline and assume the resulting histogram follows the stable convention.
Before removing legacy attributes, send known requests through the pipeline and verify the original and translated values in the exported spans. Monitor processor_schema.translated for schema translations and the processor’s per-signal skipped counters for telemetry without schema URLs. Once your queries, dashboards, and alerts use the new names, remove the Schema processor’s temporary migration configuration and add guarded delete_key statements only for legacy attributes you have verified can be retired.
Updating Dashboards, Alerts, and SLO Queries
Before retiring legacy conventions, find every dashboard, alert, recording rule, and SLO definition that uses the old attribute or metric names. If you manage these configurations in Git, search for both dotted OpenTelemetry names and their Prometheus equivalents:
Check the metric names and labels in your backend before replacing them. The examples below assume classic Prometheus histograms using UnderscoreEscapingWithSuffixes, which converts dots to underscores and appends unit suffixes. Other Prometheus translation strategies can produce different names.
Update Latency Alerts and SLO Thresholds
HTTP duration migrations can change both the metric name and its unit. For example, an alert for p95 latency above 300 milliseconds might originally use:
With the stable seconds-based histogram, the equivalent threshold is 0.3:
Update the metric name and threshold together. Changing only the name leaves the alert comparing seconds against 300, while changing only the threshold leaves it comparing milliseconds against 0.3.
Bucket-based SLOs need the same adjustment. For example, the proportion of requests completing within 500 milliseconds can be calculated using the stable histogram:
Confirm that your histogram exposes the 0.5 bucket before using this query. If it does not, configure an appropriate boundary rather than assuming the SLO can be measured at that exact threshold.
Handle Mixed-Version Services Without Double Counting
During a phased migration, some services may emit http.server.duration, others http.server.request.duration, and dual-emitting services may produce both. Adding the two metric families together double-counts requests from dual-emitting services.
Keep separate legacy and stable queries during the overlap, restricted to the service instances emitting each convention. Avoid combining the results with a broad or on (job, instance) expression unless those labels uniquely identify the corresponding streams and their dimensions are compatible. Otherwise, legacy-only series can disappear from the result.
Do not combine p95 values calculated separately from the two histograms. The legacy and stable histograms use different units and bucket boundaries, so their percentiles cannot be merged directly. Keep latency alerts for the two populations separate until the instrumentation produces a common histogram format.
Validate Queries Before Removing Legacy Names
Compare request rates, 5xx error rates, latency, and SLO burn rates before switching off dual emission. For the same dual-emitting service instances and time window, request counts and error counts should agree across the two conventions. Latency percentiles should be comparable after converting milliseconds to seconds, although different histogram boundaries can produce different estimates.
Also verify that every service instance contributes to the new queries and that alerts fire at the intended thresholds. Once your dashboards, alerts, and recording rules use the stable conventions and no required traffic depends on legacy metrics, remove the old queries and disable dual emission.
OpenTelemetry Semantic Convention Migration Walkthrough
To migrate OpenTelemetry semantic conventions, pin compatible SDK and instrumentation versions (Python SDK 1.45.2 and Flask instrumentation 0.66b2 here), verify legacy telemetry, and enable http/dup. Update dashboards, alerts, and SLOs while both conventions are emitted. Once the new queries match your baseline, switch to stable-only emission with http. Remove the flag only when your installed instrumentation emits stable conventions by default.
You’ll need Python 3.10+, the OpenTelemetry Collector Contrib distribution, Prometheus, and a Bash-compatible terminal. Run the application, Collector, and Prometheus on the same machine for this test.
Phase 1: Set Up the Service and Verify Legacy Telemetry
Create a virtual environment and install the pinned packages:
Save the following application as app.py. It provides successful requests, simulated HTTP 500 responses, and a slow request for testing latency alerts.
Configure the Collector to receive OTLP telemetry, print spans and metrics for inspection, and expose metrics to Prometheus. Save this as collector.yaml:
Save this Prometheus configuration as prometheus.yml:
Validate and start the Collector, then start Prometheus in another terminal:
In the application’s terminal, configure OTLP export and ensure the migration variable is unset:
Send successful and failing requests from another terminal:
Inspect the Collector’s debug output for legacy span attributes such as http.method and http.status_code, and the http.server.duration histogram measured in milliseconds. Check http://localhost:8889/metrics for the Prometheus series before creating queries, since exported names depend on the translation strategy.
Phase 2: Enable Dual Emission
Stop the Flask process, set http/dup in the same terminal, and restart it:
Send requests again:
Inspect the new server spans. The HTTP status code should be available under both http.status_code and http.response.status_code, and the method under both http.method and http.request.method.
Also confirm that the Collector receives both http.server.duration in milliseconds and http.server.request.duration in seconds. The Prometheus exporter should expose corresponding histogram families, such as http_server_duration_milliseconds_* and http_server_request_duration_seconds_*. These metrics represent the same requests, so do not add their counts together.
Phase 3: Update and Validate Monitoring Queries
Keep http/dup enabled while updating your dashboards, alerts, recording rules, and SLO queries to use stable attribute names, metric names, and units.
For example, the legacy HTTP 5xx error-rate ratio uses http_status_code and the milliseconds-based histogram:
The stable query uses http_response_status_code and the seconds-based histogram:
Send a mixture of successful, failing, and slow requests:
Ensure Prometheus has scraped both histogram families before and after the test requests, then run both error-rate queries over the same time window. If the queries return a zero rate or NaN, send another batch of requests and repeat after the next scrape. Compare the dual-emitting service’s series rather than including legacy-only series from Phase 1. The error-rate ratios should agree. Compare p95 latency after converting milliseconds to seconds; different histogram boundaries can produce different estimates.
Do not expect increase() to equal the exact number of requests in the shell loop. Prometheus extrapolates counter increases across the query range, so the returned value can differ from the requests sent.
Phase 4: Switch to Stable-Only Emission
After the updated dashboards, alerts, and SLO queries are working, stop Flask and replace http/dup with http:
Send another request and inspect the Collector output:
New server spans should contain http.request.method and http.response.status_code without their legacy equivalents. New metric exports should include http.server.request.duration without http.server.duration.
Prometheus may still display legacy series briefly because the exporter retains inactive metrics until they expire. Check the Collector output to confirm that no new legacy observations are being emitted rather than relying only on whether an old PromQL query returns data.
Once the updated queries continue to return the expected results, remove legacy dashboard panels, alerts, and compatibility rules. If the migration fails, restore OTEL_SEMCONV_STABILITY_OPT_IN=http/dup and restart the service while you correct the affected queries.
Using Migrated OpenTelemetry Data With groundcover
groundcover integrates with your existing OpenTelemetry Collector pipelines, allowing you to send migrated traces and metrics directly to its observability backend. You can retain your instrumentation, schema transformations, and existing export destinations while combining OpenTelemetry telemetry with groundcover’s eBPF-powered Kubernetes observability.
Send Migrated OpenTelemetry Data to groundcover
Open Data Sources → OpenTelemetry → OpenTelemetry Collectors in groundcover to get your BYOC endpoint and create an ingestion key. Add the following OTLP/HTTP exporter to your Collector configuration:
Add otlp_http/groundcover to the exporters list in your existing traces and metrics pipelines. Keep the schema and transform processors configured earlier so migrated attributes reach groundcover. You can also retain your existing exporters to send telemetry to multiple destinations.
Follow the groundcover OpenTelemetry Collector integration guide for the complete configuration.
Monitor and Validate Migrated Telemetry
Once telemetry reaches groundcover, inspect your service traces and metrics to verify the migration. Check that HTTP spans contain http.request.method and http.response.status_code, and confirm that http.server.request.duration is available for your updated monitoring queries. Compare request throughput, error rates, and latency against your pre-migration baseline.
groundcover derives APM measurements from OpenTelemetry spans, allowing you to monitor request rates, errors, and latency across instrumented services. Its eBPF sensor also captures Kubernetes service activity and dependencies without application code changes. By combining eBPF and OpenTelemetry, you can investigate application performance alongside automatically collected infrastructure telemetry.
With groundcover’s Bring Your Own Cloud architecture, your observability data stays in your cloud environment. You can get started for free without a credit card.
FAQs
Not always. Support depends on the instrumentation package and version. Some newer instrumentations already emit stable conventions without supporting /dup, so check your installed package before enabling the flag.
Yes. groundcover accepts migrated traces and metrics from your existing OpenTelemetry Collector pipelines. You can retain your instrumentation and transformation rules, validate the updated telemetry, and monitor application performance alongside eBPF-collected Kubernetes data.
Yes, for supported changes. Use OTEL_SEMCONV_STABILITY_OPT_IN where available, or normalize telemetry with the Collector’s Schema or Transform processor. Metric unit conversions and database attribute value changes require explicit handling.
Inspect the instrumentation scope and version, then use scoped OTTL transformations for telemetry without schema_url. If the correct mapping is unclear, retain the legacy attribute until you can verify its meaning.
Stop when dashboards, alerts, recording rules, and SLOs use stable conventions and your request-rate, error-rate, and latency checks pass. Switch to http or database, verify legacy emission stops, and keep /dup available for rollback.
Enable http/dup or database/dup on supported instrumentations to emit both conventions during migration. This adds legacy and stable attributes to existing spans, although separate metric streams can increase data volume. groundcover’s per-host pricing avoids per-GB ingestion charges, so dual emission does not double its licensing cost, although BYOC infrastructure usage may increase.





