Skip to main content

Telemetry and metrics

Virtual MCP Server (vMCP) provides comprehensive observability through OpenTelemetry instrumentation. You can export traces and metrics to monitor backend operations and workflow executions.

Telemetry types​

vMCP supports two types of telemetry:

  • Traces: Track requests across vMCP and its backends, showing the full path of tool calls, resource reads, and workflow executions
  • Metrics: Counters and histograms for backend request rates, error rates, and latency distributions

For general ToolHive observability concepts including trace structure and metrics, see the observability overview.

Enable telemetry​

Configure telemetry by creating a shared MCPTelemetryConfig resource, then referencing it from your VirtualMCPServer. This enables reuse across multiple resources, Kubernetes-native secret references for OTLP auth headers, CA bundle ConfigMap references, and per-server serviceName overrides.

telemetry-config.yaml
apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPTelemetryConfig
metadata:
  name: shared-otel
  namespace: toolhive-system
spec:
  openTelemetry:
    enabled: true
    endpoint: otel-collector.monitoring:4318
    insecure: true
    tracing:
      enabled: true
      samplingRate: '0.05'
    metrics:
      enabled: true
  prometheus:
    enabled: true
virtualmcpserver.yaml
apiVersion: toolhive.stacklok.dev/v1alpha1
kind: VirtualMCPServer
metadata:
  name: my-vmcp
  namespace: toolhive-system
spec:
  groupRef:
    name: my-group
  telemetryConfigRef:
    name: shared-otel
    serviceName: my-vmcp
  incomingAuth:
    type: anonymous

The serviceName field on telemetryConfigRef overrides the telemetry service name for this specific VirtualMCPServer, giving it a distinct identity in your observability backend. If omitted, it defaults to the server name with a thv- prefix.

When you update the MCPTelemetryConfig resource, the operator detects the change and triggers a rolling update of affected deployments.

MCPTelemetryConfig fields​

FieldDescriptionDefault
spec.openTelemetry.enabledEnable OpenTelemetry exportfalse
spec.openTelemetry.endpointOTLP collector endpoint (hostname:port)-
spec.openTelemetry.insecureUse HTTP instead of HTTPSfalse
spec.openTelemetry.tracing.enabledEnable tracingfalse
spec.openTelemetry.tracing.samplingRateTrace sampling rate (0.0-1.0)"0.05"
spec.openTelemetry.metrics.enabledEnable OTLP metrics exportfalse
spec.prometheus.enabledExpose /metrics endpointfalse

Export to observability backends​

Export to Jaeger via OpenTelemetry Collector​

Deploy an OpenTelemetry Collector configured to export to Jaeger:

otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]

Then create an MCPTelemetryConfig resource that points to the collector:

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPTelemetryConfig
metadata:
  name: jaeger-otel
  namespace: toolhive-system
spec:
  openTelemetry:
    enabled: true
    endpoint: otel-collector:4318
    insecure: true
    tracing:
      enabled: true
    metrics:
      enabled: true

Metrics collection​

vMCP supports two methods for collecting metrics:

  • Push via OpenTelemetry: Set spec.openTelemetry.metrics.enabled: true to push metrics to your OTel Collector via OTLP
  • Pull via Prometheus: Set spec.prometheus.enabled: true to expose a /metrics endpoint on the vMCP service port (4483) for Prometheus to scrape

Backend metrics​

These metrics track requests to individual MCP server backends:

MetricTypeDescription
toolhive_vmcp_backends_discoveredGaugeNumber of backends discovered
toolhive_vmcp_backend_requestsCounterTotal requests per backend
toolhive_vmcp_backend_errorsCounterTotal errors per backend
toolhive_vmcp_backend_requests_durationHistogramDuration of backend requests
mcp.client.operation.durationHistogramMCP client operation duration (mcp_client_operation_duration on /metrics)

Workflow metrics​

These metrics track workflow execution across backends:

MetricTypeDescription
toolhive_vmcp_workflow_executionsCounterTotal workflow executions
toolhive_vmcp_workflow_errorsCounterTotal workflow execution errors
toolhive_vmcp_workflow_durationHistogramDuration of workflow executions

Optimizer metrics​

When the vMCP optimizer is enabled, these metrics track tool-finding and tool-calling performance:

MetricTypeDescription
toolhive_vmcp_optimizer_find_tool_requestsCounterTotal FindTool calls
toolhive_vmcp_optimizer_find_tool_errorsCounterTotal FindTool errors
toolhive_vmcp_optimizer_find_tool_durationHistogramDuration of FindTool calls
toolhive_vmcp_optimizer_find_tool_resultsHistogramNumber of tools returned per call
toolhive_vmcp_optimizer_token_savings_percentHistogramToken savings percentage per call
toolhive_vmcp_optimizer_call_tool_requestsCounterTotal CallTool calls
toolhive_vmcp_optimizer_call_tool_errorsCounterTotal CallTool errors
toolhive_vmcp_optimizer_call_tool_not_foundCounterCallTool calls where tool was not found
toolhive_vmcp_optimizer_call_tool_durationHistogramDuration of CallTool calls

Distributed tracing​

vMCP creates client-side spans for backend operations with the following span names:

  • tools/call <tool_name> - Tool calls to backends
  • resources/read - Resource reads from backends
  • prompts/get <prompt_name> - Prompt retrieval from backends
  • list_capabilities - Backend capability discovery

Each span includes attributes for the target backend (target.workload_id, target.workload_name, target.base_url) and the relevant MCP attributes (mcp.method.name, gen_ai.tool.name, mcp.resource.uri).

Next steps​