OpenLLMetry
Open-source observability platform for tracking, monitoring, and debugging LLM applications using OpenTelemetry standards. Integrates with popular observability tools and LLM providers.
OpenLLMetry
OpenLLMetry is an open-source observability platform designed specifically for Large Language Model (LLM) applications. It provides comprehensive tracking, monitoring, and debugging capabilities using the OpenTelemetry standard, enabling developers to gain deep insights into their LLM workflows.
Key Features
- LLM-Specific Instrumentation: Automatically captures and traces LLM interactions, including prompts, completions, and tool usage
- OpenTelemetry Integration: Seamlessly integrates with existing OpenTelemetry-based observability stacks
- Multi-Provider Support: Works with major LLM providers including OpenAI, Anthropic, Mistral AI, Cohere, and more
- Multi-Platform Compatibility: Supports Python and TypeScript implementations
- Trace Visualization: Provides detailed trace views for debugging complex LLM workflows
- Performance Monitoring: Tracks latency, token usage, and error rates across LLM calls
- Cost Optimization: Helps identify expensive LLM operations and optimize usage
- Extensible Architecture: Modular design allows for custom instrumentation and extensions
Tech / Self-Hosting
OpenLLMetry is open-source software that can be self-hosted. The project provides comprehensive documentation for deployment and configuration. It requires an OpenTelemetry-compatible backend for storing and visualizing traces, with support for popular platforms like Instana, Dynatrace, Grafana Tempo, Honeycomb, New Relic, Datadog, and Splunk.