Telemetry Package

The telemetry package provides OpenTelemetry integration for distributed tracing and metrics collection, with specialized instrumentation for gRPC and HTTP clients.

Core Functions

Init

func Init(ctx context.Context, cfg Config, logger log.Logger) (cleanUp func(), err error)

Initializes OpenTelemetry instrumentation with tracing and metrics exporters. Returns a cleanup function that must be called before application shutdown to flush remaining telemetry data.

ctx

context.Context

required

Context for initialization. Used for setting up exporters.

cfg

Config

required

Telemetry configuration including OpenTelemetry settings.

logger

log.Logger

required

Logger instance for telemetry subsystem messages.

cleanUp

func()

Cleanup function that should be deferred to ensure proper shutdown of telemetry providers.

error

Returns an error if telemetry initialization fails.

Example

package main

import (
    "context"
    "log"
    
    "github.com/raystack/salt/log"
    "github.com/raystack/salt/telemetry"
)

func main() {
    ctx := context.Background()
    logger := log.NewLogrus()
    
    cfg := telemetry.Config{
        AppName:    "my-service",
        AppVersion: "1.0.0",
        OpenTelemetry: telemetry.OpenTelemetryConfig{
            Enabled:                true,
            CollectorAddr:          "localhost:4317",
            TraceSampleProbability: 1.0,
        },
    }
    
    cleanup, err := telemetry.Init(ctx, cfg, logger)
    if err != nil {
        log.Fatalf("Failed to initialize telemetry: %v", err)
    }
    defer cleanup()
    
    // Your application code here...
}

Configuration

Config

type Config struct {
    AppVersion    string
    AppName       string
    OpenTelemetry OpenTelemetryConfig
}

Top-level telemetry configuration.

AppVersion

string

required

Application version for resource attributes.

AppName

string

default:"service"

Application name for service identification.

OpenTelemetry

OpenTelemetryConfig

required

OpenTelemetry-specific configuration.

OpenTelemetryConfig

type OpenTelemetryConfig struct {
    Enabled                      bool
    CollectorAddr                string
    PeriodicReadInterval         time.Duration
    TraceSampleProbability       float64
    VerboseResourceLabelsEnabled bool
}

OpenTelemetry configuration options.

Enabled

bool

default:"false"

Enable or disable OpenTelemetry monitoring.

CollectorAddr

string

default:"localhost:4317"

Address of the OpenTelemetry collector (OTLP gRPC endpoint).

PeriodicReadInterval

time.Duration

default:"1s"

Interval for reading and exporting metrics.

TraceSampleProbability

float64

default:"1"

Probability of sampling a trace (0.0 to 1.0). 1.0 means sample all traces.

VerboseResourceLabelsEnabled

bool

default:"false"

Include verbose resource labels (OS, host, process information) in telemetry data.

Example Configuration (YAML)

app_name: my-service
app_version: 1.0.0
open_telemetry:
  enabled: true
  collector_addr: otel-collector:4317
  periodic_read_interval: 1s
  trace_sample_probability: 0.1
  verbose_resource_labels_enabled: false

Package: otelgrpc

Provides OpenTelemetry instrumentation for gRPC clients with automatic metrics collection.

Meter

type Meter struct {
    // Contains unexported fields
}

Instrumentation meter for gRPC clients that tracks request duration, request size, and response size.

NewMeter

func NewMeter(hostName string, opts ...Option) Meter

Creates a new gRPC client meter for collecting telemetry.

hostName

string

required

The gRPC server hostname in “host:port” format.

opts

...Option

Optional configuration options.

Meter

Configured meter instance for gRPC instrumentation.

Example

import (
    "github.com/raystack/salt/telemetry/otelgrpc"
    "google.golang.org/grpc"
)

meter := otelgrpc.NewMeter("api.example.com:9090")

conn, err := grpc.Dial(
    "api.example.com:9090",
    grpc.WithUnaryInterceptor(meter.UnaryClientInterceptor()),
    grpc.WithStreamInterceptor(meter.StreamClientInterceptor()),
)

Meter Methods

UnaryClientInterceptor

func (m *Meter) UnaryClientInterceptor() grpc.UnaryClientInterceptor

Returns a gRPC unary client interceptor that automatically records metrics for each RPC call.

grpc.UnaryClientInterceptor

Interceptor function that can be passed to grpc.Dial via grpc.WithUnaryInterceptor.

StreamClientInterceptor

func (m *Meter) StreamClientInterceptor() grpc.StreamClientInterceptor

Returns a gRPC stream client interceptor that automatically records metrics for streaming calls.

grpc.StreamClientInterceptor

Interceptor function that can be passed to grpc.Dial via grpc.WithStreamInterceptor.

RecordUnary

func (m *Meter) RecordUnary(ctx context.Context, p UnaryParams)

Manually records metrics for a unary gRPC call.

ctx

context.Context

required

Request context.

UnaryParams

required

Parameters containing call information (method, request, response, error, timing).

RecordStream

func (m *Meter) RecordStream(ctx context.Context, start time.Time, method string, err error)

Manually records metrics for a streaming gRPC call.

ctx

context.Context

required

Request context.

start

time.Time

required

Start time of the stream.

method

string

required

gRPC method name.

err

error

Error if the stream failed.

Utility Functions

ParseFullMethod

func ParseFullMethod(fullMethod string) []attribute.KeyValue

Parses a gRPC full method name into service and method attributes.

fullMethod

string

required

Full gRPC method name in format “/package.Service/Method”.

[]attribute.KeyValue

OpenTelemetry attributes for the service and method.

ExtractAddress

func ExtractAddress(addr string) (host, port string)

Splits a host:port address into separate host and port components.

addr

string

required

Address string in “host:port” format.

host

string

The host component.

port

string

The port component. Defaults to “80” if not specified.

Package: otelhttpclient

Provides OpenTelemetry instrumentation for HTTP clients following OpenTelemetry Semantic Conventions.

NewHTTPTransport

func NewHTTPTransport(baseTransport http.RoundTripper) http.RoundTripper

Wraps an HTTP RoundTripper with OpenTelemetry instrumentation that automatically collects metrics for HTTP client requests.

baseTransport

http.RoundTripper

required

Base transport to wrap. If nil, http.DefaultTransport is used.

http.RoundTripper

Instrumented transport that collects telemetry data.

Example

import (
    "net/http"
    
    "github.com/raystack/salt/telemetry/otelhhtpclient"
)

client := &http.Client{
    Transport: otelhttpclient.NewHTTPTransport(nil),
}

resp, err := client.Get("https://api.example.com/users")

Metrics Collected

The HTTP transport collects the following metrics according to OpenTelemetry semantic conventions:

http.client.duration - Request duration in milliseconds
http.client.request.size - Request body size in bytes
http.client.response.size - Response body size in bytes

AnnotateRequest

func AnnotateRequest(req *http.Request, route string) *http.Request

Adds telemetry-related annotations to the request context. Use this to add high-level route information to prevent metric cardinality explosion.

req

*http.Request

required

HTTP request to annotate.

route

string

required

Route template (e.g., “/users/:id”). Must be a template, not the actual URL path.

*http.Request

Request with updated context containing the route annotation.

Always use route templates (e.g., /users/:id) instead of actual paths (e.g., /users/123) to prevent high cardinality metrics that could overwhelm your telemetry system.

Example

req, _ := http.NewRequest("GET", "https://api.example.com/users/123", nil)
req = otelhttpclient.AnnotateRequest(req, "/users/:id")

resp, err := client.Do(req)

LabelerFromContext

func LabelerFromContext(ctx context.Context) (*otelhttp.Labeler, bool)

Retrieves the labeler annotation from the context if it exists.

ctx

context.Context

required

Request context.

*otelhttp.Labeler

The labeler instance.

bool

True if the labeler was found in the context.

Best Practices

Sampling Strategy

Adjust trace sampling based on traffic volume:

// Production: sample 10% of traces
TraceSampleProbability: 0.1

// Development: sample all traces
TraceSampleProbability: 1.0

Resource Labels

Enable verbose labels only when needed:

// Production: minimal labels
VerboseResourceLabelsEnabled: false

// Debugging: full labels
VerboseResourceLabelsEnabled: true

HTTP Route Cardinality

Always use route templates:

// Good: low cardinality
req = AnnotateRequest(req, "/users/:id")
req = AnnotateRequest(req, "/orders/:orderId/items/:itemId")

// Bad: high cardinality
req = AnnotateRequest(req, "/users/12345")
req = AnnotateRequest(req, fmt.Sprintf("/orders/%s", orderID))

Graceful Shutdown

Always defer the cleanup function:

cleanup, err := telemetry.Init(ctx, cfg, logger)
if err != nil {
    log.Fatal(err)
}
defer cleanup()  // Ensures telemetry data is flushed

// Application code...

gRPC Client Instrumentation

Add interceptors at connection time:

meter := otelgrpc.NewMeter("api.example.com:9090")

conn, err := grpc.Dial(
    target,
    grpc.WithUnaryInterceptor(meter.UnaryClientInterceptor()),
    grpc.WithStreamInterceptor(meter.StreamClientInterceptor()),
)

Integration with Collectors

The package exports telemetry data via OTLP (OpenTelemetry Protocol) over gRPC. It’s compatible with:

OpenTelemetry Collector
Jaeger (with OTLP receiver)
Grafana Tempo
Any OTLP-compatible backend

Example Collector Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  jaeger:
    endpoint: jaeger:14250
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

Core

CLI

Authentication

Infrastructure

Utilities

Telemetry Package

Core Functions

Init

Example

Configuration

Config

OpenTelemetryConfig

Example Configuration (YAML)

Package: otelgrpc

Meter

NewMeter

Example

Meter Methods

UnaryClientInterceptor

StreamClientInterceptor

RecordUnary

RecordStream

Utility Functions

ParseFullMethod

ExtractAddress

Package: otelhttpclient

NewHTTPTransport

Example

Metrics Collected

AnnotateRequest

Example

LabelerFromContext

Best Practices

Sampling Strategy

Resource Labels

HTTP Route Cardinality

Graceful Shutdown

gRPC Client Instrumentation

Integration with Collectors

Example Collector Configuration

Core

CLI

Authentication

Infrastructure

Utilities

​Core Functions

​Init

​Example

​Configuration

​Config

​OpenTelemetryConfig

​Example Configuration (YAML)

​Package: otelgrpc

​Meter

​NewMeter

​Example

​Meter Methods

​UnaryClientInterceptor

​StreamClientInterceptor

​RecordUnary

​RecordStream

​Utility Functions

​ParseFullMethod

​ExtractAddress

​Package: otelhttpclient

​NewHTTPTransport

​Example

​Metrics Collected

​AnnotateRequest

​Example

​LabelerFromContext

​Best Practices

​Sampling Strategy

​Resource Labels

​HTTP Route Cardinality

​Graceful Shutdown

​gRPC Client Instrumentation

​Integration with Collectors

​Example Collector Configuration

Core Functions

Init

Example

Configuration

Config

OpenTelemetryConfig

Example Configuration (YAML)

Package: otelgrpc

Meter

NewMeter

Example

Meter Methods

UnaryClientInterceptor

StreamClientInterceptor

RecordUnary

RecordStream

Utility Functions

ParseFullMethod

ExtractAddress

Package: otelhttpclient

NewHTTPTransport

Example

Metrics Collected

AnnotateRequest

Example

LabelerFromContext

Best Practices

Sampling Strategy

Resource Labels

HTTP Route Cardinality

Graceful Shutdown

gRPC Client Instrumentation

Integration with Collectors

Example Collector Configuration