#Troubleshooting the instrumentation

When using OpenTelemetry instrumentation, you may encounter two main types of issues: problems with instrumentation injection into workloads, and problems with instrumentation libraries generating telemetry data. This document provides troubleshooting methods for both types of issues.

#Overview

Instrumentation troubleshooting primarily focuses on the following aspects:

• Injection issues: Whether instrumentation is successfully injected into application pods
• Data generation issues: Whether instrumentation libraries correctly generate and send telemetry data

#Troubleshooting instrumentation injection into your workload

When instrumentation fails to inject correctly into workloads, you can perform the following checks to diagnose the problem.

#Checking the Instrumentation object

First verify that the Instrumentation object was successfully created:

kubectl get instrumentation -n <workload_namespace>

If you do not see the expected Instrumentation object, check:

1. Whether the Instrumentation resource was correctly created
2. Whether the namespace is correct
3. Whether there are error messages in the Operator logs

#Checking the init-container

Verify that the opentelemetry-auto-instrumentation init-container started successfully:

kubectl get events -n <workload_namespace>

View detailed pod information:

kubectl describe pod <pod_name> -n <workload_namespace>

In the output, look for the init-container section and confirm:

• Whether the init-container exists
• Whether the init-container completed successfully
• Whether there are any errors or warnings

#Checking resource deployment order

Ensure resources are deployed in the correct order:

1. First deploy the Instrumentation object
2. Then deploy or restart application pods

If application pods were created before the Instrumentation object, instrumentation will not be injected. You need to restart the pods:

kubectl rollout restart deployment <deployment_name> -n <workload_namespace>

#Searching for errors in Operator logs

Check the Operator logs for instrumentation-related errors:

kubectl logs -l app.kubernetes.io/name=opentelemetry-operator -n opentelemetry-operator2 --tail=100

Common errors include:

• Configuration validation failures
• Insufficient permissions
• Resource conflicts

#Verifying pod annotations

Confirm that application pods have the correct instrumentation annotations:

kubectl get pod <pod_name> -n <workload_namespace> -o jsonpath='{.metadata.annotations}' | jq

You should see annotations similar to the following:

{
  ...
  "instrumentation.opentelemetry.io/inject-java": "true"
  ...
}

If annotations are missing or incorrect:

1. Check the annotations configuration in the Deployment or Pod template
2. Confirm the annotation namespace and language type are correct
3. Verify the Instrumentation object configuration

#Checking injected environment variables

Verify that instrumentation correctly injected environment variables:

kubectl exec <pod_name> -n <workload_namespace> -- env | grep OTEL

You should see OpenTelemetry-related environment variables, such as:

OTEL_SERVICE_NAME=my-service
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_RESOURCE_ATTRIBUTES=service.name=my-service

#Verifying instrumentation libraries

Check whether instrumentation libraries were correctly installed in the container:

kubectl exec <pod_name> -n <workload_namespace> -- sh -c 'ls -lh /otel-auto-instrumentation-*'

This directory should contain instrumentation libraries and related files.

#Troubleshooting telemetry data generation by the instrumentation libraries

If instrumentation was successfully injected but is not generating telemetry data, you can troubleshoot using the following methods.

#Verifying the endpoint configuration

Confirm that instrumentation is sending data to the correct endpoint:

kubectl get instrumentation <instrumentation_name> -n <workload_namespace> -o jsonpath='{.spec.exporter.endpoint}'

Example output

http://otel-collector:4317

kubectl get instrumentation <instrumentation_name> -n <workload_namespace> -o yaml | grep -A 1 'OTEL_'

The default endpoint is http://localhost:4317. If you are using a custom Collector endpoint, ensure:

1. The endpoint address is correct
2. The Collector Service is accessible
3. Port configuration is correct (gRPC uses 4317, HTTP uses 4318)

#Checking application logs for errors

Check application logs for instrumentation-related error messages:

kubectl logs <application_pod> -n <workload_namespace>

Look for OpenTelemetry-related errors, such as:

• Connection failures
• Authentication errors
• Configuration errors
• Library loading failures

#Verifying Collector is receiving data

Confirm whether the Collector is receiving telemetry data:

Check Collector logs

kubectl logs <collector_pod> -n opentelemetry-collector

If the Debug Exporter is enabled, you should be able to see received data.

Check Collector metrics

kubectl port-forward -n opentelemetry-collector <collector_pod> 8888:8888

Then access http://localhost:8888/metrics and look for:

• otelcol_receiver_accepted_spans: Should be greater than 0
• otelcol_receiver_refused_spans: Should be 0 or very small