Profiling JVM applications it not very easy in the first place. Many profilers, such as JVisualVM and jstack, will simply lie to your face about which call stacks are hottest and where your bottlenecks lie. Profiling in production environments has even more challenges, because you have to carefully manage overhead, account for other processes running on the system, and choose non-invasive tools that don't require an application restart. This is the workshop for you -- we will learn, understand, and experiment first-hand with JVM profiling tools designed for production use.

We will build a simple checklist for verifying JVM application performance, and finding the area to focus on in a closer investigation. Then, we will experiment with two approaches for CPU profiling on Linux: the perf multi-tool, combined with perf-map-agent, and the async-profiler project, an innovative tool that brings perf together with traditional JVM profiling techniques. We will visualize stack traces using flame graphs, and understand where the CPU bottlenecks lie, through a series of hands-on labs. In the second half of this workshop, we will talk about more complicated scenarios: diagnosing errors when opening files, tracing database queries, monitoring system I/O load, understanding the reasons for excessive garbage collection, figuring out why threads are blocked off-CPU, and more.

Outline:

• Linux and JVM performance information sources: tracepoints, {k,u}probes, USDT, JMX, JVMTI, Java SA, JFR, GC/JIT logs
• Linux JVM performance checklist: top, sar, free, iostat, pidstat, vmstat, jstack, jstat (based on hsperfdata), jcmd, jattach
• Linux perf, CPU sampling, getting stack reports, problem with symbols
• Generating perf maps with perf-map-agent, inline frames, source info
• Visualizing JVM profiles with flame graphs
• JVMTI profiling with an agent & jattach: GetAllStackTraces, exceptions, class loads, monitor contention [link] etc.
• Full-stack JVM profiling with async-profiler, how it works, pros and cons
• Introduction to BPF
• BPF scenarios and the BCC toolkit, Java-specific tools, JVM USDT probes, -XX:+ExtendedDTraceProbes
• Dedicated BCC tools: fileslower, opensnoop, gethostlatency, runqlat, cpudist
• General-purpose BCC tools: trace, argdist, funccount, stackcount, funclatency
• Heap allocation profiling (based on TLAB sampling) with async-profiler vs. funccount/stackcount vs. grav

Node.js, the popular cross-platform JavaScript server framework, has grown in popularity and scale to the point where very challenging, non-trivial, production-only issues pop up in complex deployment environments. Although Node.js is equipped with the built-in V8 profiler, it is limited in scope to CPU time, which doesn’t account for the majority of Node.js performance issues. Also, profiling in production environments requires non-invasive tools with a low overhead, which do not require an application restart for the sake of profiling. In this workshop, we will learn, understand, and experiment with Node.js profiling tools designed for production use.

We will experiment with the built-in V8 profiler, and learn how to analyze its log output to zoom-in on CPU hot-spots in the V8 runtime, your JavaScript code, or external modules. We will also explore the Linux perf multi-tool, which can perform general-purpose CPU profiling, and how to get the Node.js runtime to generate symbolic information required to resolve stack traces emitted by perf. We will visualize stack traces using flame graphs, and understand where the CPU bottlenecks lie. We will also look at core dump analysis for crashes and memory leaks, and identify heap objects using the llnode plugin for lldb. In the second half of this workshop, we will talk about more complicated scenarios: diagnosing intermittent errors when connecting to a database or running queries, monitoring system I/O load, understanding the reasons for excessive V8 garbage collection, figuring out why your event loop thread is blocked, and more.

Outline:

• Node.js observability tools
• CPU profiling with V8
• Linux perf and CPU sampling
• Visualizing CPU profiles with flame graphs
• Introduction to BPF
• BPF scenarios and the BCC toolkit
• USDT probes
• Blocked-time analysis

PerfView is a free, standalone, small-impact tool produced by the CLR team that helps in performance investigations. It offers several unique features for understanding large performance reports, such as sampling, pattern folding, differencing, and grouping. You can use it for ETW analysis on large-scale Windows servers as well as low-end client devices and phones -- and you can even use it as a general-purpose analysis tool for Linux performance traces. In this session we will use PerfView to systematically measure and improve CPU time, wall-clock time, and memory usage (yes, PerfView can solve memory leaks as well!).