Let's be honest - observability can suck. Ever feel like you're swimming in dashboard soup? You know the feeling: tons of single-use dashboards, building new ones during every incident only to lose them in the chaos, and spending ages creating visualizations that no one ever looks at again. Even with all the right tools, something still feels off.

This talk shares the journey of how our team tackled this exact problem while building an on-call tool with high expectations for reliability. You'll hear how we went from feeling lost in our own tooling to becoming confident enough to sleep soundly while on-call. Through building clear system boundaries, a thoughtful approach to dashboards, and running hands-on drills with our team, our observability approach became the key thing that makes our system feel reliable.

Key takeaways include:

• Principles for building dashboards that are useful, long lived, and widely adopted
• How to layer your stack, by connecting your observability tooling to build a debugging flow that puts UX first
• Creating a culture change towards observability, getting your team invested through hands-on drills