OpenTelemetry Setup Guide: Instrument Services and Build a Useful Observability Stack

Overview

OpenTelemetry provides a common approach for generating and transporting telemetry from applications and infrastructure. Its main signals are traces, metrics, and logs. Used together, they help a team answer different questions: traces show where time is spent across a request, metrics reveal patterns and resource behavior, and logs provide detailed event context.

A practical OpenTelemetry setup usually has four layers:

1. Instrumentation: Libraries, agents, or manual code create telemetry inside services.
2. Context propagation: Trace and span context travels between services so a distributed request can be connected end to end.
3. Collection and processing: OpenTelemetry Collectors receive, filter, enrich, batch, sample, and route data.
4. Backends and analysis: Metrics, traces, and logs are stored and explored in systems appropriate to each signal, often with Prometheus and Grafana included in the metrics workflow.

Start with one service or one user journey rather than instrumenting an entire platform at once. Choose a request that crosses a meaningful boundary, such as an API gateway, application service, queue, and database. That path gives the team a concrete test case for propagation, latency analysis, and error correlation.

OpenTelemetry is a transport and instrumentation standard, not a complete incident-management process. The value comes from connecting telemetry to service ownership, runbooks, alert policies, and review habits. For broader operational context, pair this work with an on-call handoff checklist and a documented incident response process.

What to track

Instrumentation coverage

Maintain a simple inventory of production services and record whether each one has automatic instrumentation, manual instrumentation, or no instrumentation. Track the language runtime, instrumentation package or agent, deployment environment, service owner, and last verification date. Coverage is not binary: a service may emit server spans but lack database, messaging, or outbound HTTP visibility.

For each important request path, verify that you can identify:

• The incoming operation and service name.
• The downstream calls and their duration.
• Database, cache, or message-queue activity where relevant.
• Exceptions and status codes.
• Deployment or version metadata needed to compare behavior.

Context propagation

Distributed tracing becomes difficult to trust when a trace stops at a service boundary. Check propagation across HTTP, asynchronous messaging, scheduled jobs, and background workers. Test both successful and failed requests. A trace identifier should remain consistent across the intended path, while each operation receives its own span.

Be deliberate with baggage and attributes. Do not place secrets, authorization tokens, raw session data, or unnecessary personal information into telemetry. Define approved attributes such as service version, deployment environment, region, route, and sanitized tenant or request classifications where appropriate.

Signal health and cardinality

Track whether exporters are succeeding, whether the Collector is rejecting data, and whether queues or memory limits are being reached. For metrics, review label cardinality. User IDs, unbounded URLs, request IDs, and error messages can create excessive series or make dashboards expensive and slow. Prefer stable dimensions such as route templates, status class, service, and dependency.

For traces, monitor ingestion volume, sampling outcomes, missing parent spans, and unusually large spans. For logs, check structured fields, severity consistency, correlation identifiers, and duplicate events. A useful observability stack measures its own delivery path rather than assuming telemetry is always available.

Core service indicators

Build dashboards around service-level questions instead of signal type alone. A useful starting set includes request rate, error rate, latency distributions, saturation, dependency health, and queue behavior. Use histograms or percentile views for latency rather than relying only on averages. Include deployment markers so a change in behavior can be compared with a release.

Use a Kubernetes troubleshooting guide alongside application telemetry when services run in Kubernetes. Application spans can show that a request is slow, while container and cluster signals may explain whether the cause is resource pressure, scheduling, networking, or a failing dependency.

Cadence and checkpoints

Observability configuration should be reviewed on a recurring schedule because services, dependencies, traffic patterns, and incident lessons change. A monthly review works for active platforms; a quarterly review may be sufficient for stable services. Teams handling frequent deployments can add lightweight checks to every release.

Per release

• Confirm the service name, environment, and version attributes are correct.
• Run a representative request and verify trace continuity.
• Check that new routes and dependencies do not create unbounded metric labels.
• Review whether the release changed error, latency, or traffic dashboards.

Monthly

• Review Collector error logs, rejected spans, export failures, and queue or memory pressure.
• Inspect dashboards for unused panels, broken queries, stale links, and unclear ownership.
• Compare alert notifications with incidents and known customer-impacting failures.
• Review sampling rules and confirm that high-value errors and slow requests remain discoverable.
• Check storage and retention behavior against the team’s operational needs.

Quarterly

• Map important customer journeys and confirm that each has an observable entry point and dependency path.
• Remove attributes that are no longer useful and document any new sensitive-data controls.
• Test a failure scenario, such as a dependency timeout or deployment regression, from detection through diagnosis.
• Update ownership, escalation paths, and links to runbooks.

Collector configuration should be treated like production code. Keep it in version control, review changes, separate environment-specific values, and test routing or filtering changes before broad rollout. If the platform uses an internal developer platform, make safe defaults available in service templates so teams do not have to recreate instrumentation conventions manually. The internal developer platform blueprint provides useful context for that kind of standardization.

How to interpret changes

A change in telemetry is not automatically a change in system health. First determine whether the application changed, the traffic changed, or the collection path changed. A sudden drop in spans may indicate lower traffic, a sampling adjustment, exporter failure, or an instrumentation regression. Compare application request metrics with Collector and backend ingestion metrics before concluding that user activity declined.

When latency increases, separate the request into spans and compare the slow group with a normal group. Look for a common dependency, route, deployment version, region, or resource condition. An increase in average latency with stable upper percentiles may be harmless distribution noise; a widening tail usually deserves more attention, particularly if errors or timeouts rise at the same time.

When errors increase, distinguish between application failures, dependency failures, and telemetry-export failures. A backend outage can create missing traces without causing application errors, while an application failure may be fully visible if error sampling is configured correctly. Alerts should point to an operational decision, such as rollback, capacity adjustment, dependency escalation, or investigation through a runbook.

Review alert quality after incidents. If responders repeatedly open an alert without finding customer impact, refine its threshold, window, grouping, or severity. If an incident was discovered through a user report rather than telemetry, record the missing signal and add it to the next observability checkpoint.

When to revisit

Revisit this OpenTelemetry setup after major architecture changes, new asynchronous workflows, a migration to Kubernetes, a change in telemetry backend, or an incident that exposed a diagnostic gap. Also review it when service ownership changes, retention requirements change, or Collector resource usage grows unexpectedly.

Use a small recurring checklist:

1. Select one critical user journey and follow it from entry point to dependencies.
2. Confirm trace propagation, useful attributes, and version markers.
3. Compare application signal volume with Collector export and backend ingestion.
4. Review the highest-volume metrics and remove unsafe or unnecessary dimensions.
5. Test one alert and one runbook as a responder would during an incident.
6. Record one improvement, assign an owner, and set a review date.

Keep the process practical. The goal is not to maximize telemetry; it is to make important system behavior easier to understand under normal operations and during failure. Combine OpenTelemetry with focused dashboards, clear alert ownership, and tested troubleshooting procedures. When Kubernetes symptoms are involved, consult the CrashLoopBackOff troubleshooting guide rather than relying on traces alone. A disciplined monthly or quarterly review keeps the observability stack aligned with the services it is meant to explain.