netdata-instrumentation

Netdata instrumentation

This skill adds OpenTelemetry instrumentation to application code that will export metrics (and, where the SDK is mature enough, logs) to Netdata over OTLP/gRPC.

When to use this skill

• The user is adding observability to a service for the first time.
• The user wants to replace a vendor SDK (Datadog, New Relic, Dynatrace) with OpenTelemetry.
• The user is wiring auto-instrumentation into an existing service.
• The user wants to know what resource attributes Netdata's dashboard and MCP tools rely on.
• The user is choosing between SDK-level and Collector-level exporting.

Key facts

• Export protocol: OTLP/gRPC on the Netdata OTLP port (default 4317).
• Signals: metrics and logs are accepted. Traces are not yet accepted by Netdata (planned Q2 2026). If the service only needs traces, send them to Jaeger, Tempo, or an external vendor; do not configure a trace exporter pointed at Netdata.
• Resource attributes that matter:
  • service.name (required by OTel). Netdata groups charts by this.
  • service.version. Used in dashboards and alert rules.
  • deployment.environment (or deployment.environment.name in the newer semconv). Lets the MCP tools filter by env.
  • host.name. Filled by most SDKs automatically.
• Prefer auto-instrumentation where it exists (Node.js, Python, Java, .NET, Ruby). Hand-code only for Go and PHP, where auto-instrumentation is immature or not yet stable.
• Metrics default to delta temporality in many SDKs; Netdata accepts both delta and cumulative. Pick cumulative for gauges and sums when the backend will be swapped later.
• Environment variables control the exporter without code changes:
  • OTEL_SERVICE_NAME
  • OTEL_EXPORTER_OTLP_ENDPOINT (scheme + host + port, e.g. http://netdata.example.internal:4317)
  • OTEL_RESOURCE_ATTRIBUTES (comma-separated key=value list)
  • OTEL_EXPORTER_OTLP_PROTOCOL=grpc (be explicit; some SDKs default to http/protobuf)

Step-by-step

1. Pick the language rule that matches the service. See References.
2. Install the OTel packages with the versions listed in that rule.
3. Copy the minimal SDK init into the service's startup path. In Node.js/Python this is typically a -r/--import preload; in Java it is the Java agent jar; in Go/Ruby/.NET/PHP it is an in-process call before the first work happens.
4. Set the environment variables above. The endpoint must use http:// (or https:// with TLS) and port 4317. Do not put /v1/metrics on the URL; that is the OTLP/HTTP path and the gRPC exporter does not want it.
5. Deploy. Generate some traffic. Verify metrics arrived (see Verification).
6. If metrics render in Netdata with unhelpful dimension names, add a mapping file. See skills/netdata-otel-setup/rules/metric-mapping.md.

Common mistakes

• Using the HTTP exporter (exporter-otlp-http, port 4318) against Netdata. Netdata accepts gRPC only.
• Forgetting to set service.name. Without it, charts group under an "unknown_service" bucket and look broken.
• Setting both the SDK endpoint and the Collector endpoint to the same Netdata URL. Pick one export path: SDK to Netdata, or SDK to Collector to Netdata. Double export doubles the sample count.
• Calling shutdown() in the wrong place. Async SDKs need a chance to flush on SIGTERM; a blind process.exit(0) loses the last batch.
• Enabling trace exporting against Netdata. The gRPC connection will succeed but the traces are silently dropped.
• Hardcoding the endpoint in source. Use the env var. Let ops move the endpoint without a code change.

Choosing SDK export vs Collector export

Two deployment shapes:

• SDK -> Netdata directly: simplest. The SDK's OTLP gRPC exporter opens a connection to Netdata. Fine for single-service setups or dev environments. The downside is that every service carries its own export config and retry logic.
• SDK -> Collector -> Netdata: a local (DaemonSet or sidecar) Collector intercepts the SDK's OTLP output, adds enrichment (Kubernetes metadata, cluster name, etc.), and forwards to Netdata. The SDK points at http://localhost:4317 or http://$HOST_IP:4317; the Collector handles the real destination, TLS, and retries.

Production-shape defaults:

• Single-service or dev: SDK -> Netdata directly.
• Kubernetes, multi-team: SDK -> DaemonSet Collector -> Netdata Parent.
• Long-lived serverless / Lambda: SDK -> Netdata directly; the Collector lifecycle does not fit a request-scoped runtime.

Verification

Run an MCP query against the Netdata instance that is supposed to be receiving the traffic. See the MCP integration skill for the full transport setup; here is the minimum check:

# From a shell where the agent with MCP access is configured:
# list_metrics filtered by service name

Or via the HTTP API for a quick check without an MCP client:

curl -s 'http://NETDATA_HOST:19999/api/v2/contexts' \
  | jq --arg svc "$OTEL_SERVICE_NAME" \
       '.contexts | to_entries[] | select(.key | contains($svc))'

A non-empty result means at least one metric from the service arrived within the last few minutes. For the canonical working instrumentation, see tests/e2e/sample-apps/ in this repo.

References

• rules/nodejs.md
• rules/python.md
• rules/java.md
• rules/go.md
• rules/dotnet.md
• rules/ruby.md
• rules/php.md
• OpenTelemetry semconv: https://opentelemetry.io/docs/specs/semconv/
• Netdata OTel integration: https://learn.netdata.cloud/docs/collecting-metrics/opentelemetry/opentelemetry-metrics