Uncorking Queueing Bottlenecks with OpenTelemetry

This presentation discusses how Gearset utilizes OpenTelemetry to enhance visibility in asynchronous workflows, addressing the complexities and bottlenecks of distributed, event-driven systems.

Key Points:

• Background: OpenTelemetry was adopted to address queue bottlenecks which were not easily detectable with traditional metrics and logs.
• Importance of Distributed Tracing: It is crucial for event-driven systems to connect the whole workflow from producers to consumers.
• Implementation Challenges: At Gearset, implementing OpenTelemetry involved customizing middleware to propagate trace contexts across systems.
• Benefits Gained: Greater insight into system behaviors, unknown dependencies, and the ability to visualize the operation lifetime of requests contributed significantly to operational improvements.

Lessons Learned:

• Cultural Shift: Moving from dashboard metrics to a trace-driven debugging approach proved crucial for precise problem-solving.
• Operational Transparency: Visualization of service interconnectivity and latency through tracing exposed inefficiencies and allowed for optimization.
• Proactive Alerting: Incorporating service level objectives (SLOs) based on rich tracing data allows for more effective and meaningful alerts that are linked to customer experience rather than simply infrastructure metrics.

Conclusion: The integration of OpenTelemetry significantly improved Gearset’s ability to diagnose and preemptively address queueing bottlenecks, providing richer data insights and transforming their incident response strategy.

This is the end of the AI-generated content.

Queues are the backbone of scalable, asynchronous systems, but they can easily create a tangled web of complexity. When things slow down, the bottleneck could be anywhere, from producer lag to consumer exhaustion, and standard metrics often fail to show the full picture.

In this session, we’ll explore how Gearset uses OpenTelemetry to bring visibility to our asynchronous workflows. We’ll share the lessons learned at Gearset while scaling our message-driven systems, and dive into:

• Connecting the dots: Why distributed tracing is non-negotiable for event-driven systems.
• Implementation: Using OTel standards to bridge the gap between producers and consumers.
• The hard parts: Tackling long-running traces, measuring true end-to-end duration, and creating lasting cultural change
• Outcomes: How we shifted from "drowning in dashboards" to precise, trace-driven debugging, with real world examples