cmd/trace: analyze Go 1.22+ traces scalably #65315
Description
Currently cmd/trace loads the entire expanded trace into memory, always. However, the Go 1.22+ trace API is a streaming API, so it's totally possible for the tool to split up traces and only touch the necessary parts.
This will require some refactoring, and will definitely be easier once we remove the old tracer and the old tracer parser (putting old trace parsing behind the new API in internal/trace/v2).
Activity
cstockton commentedon Dec 7, 2025
Just wanted to say awesome work and this is a very exciting feature to me, I am going to do some reading on the new trace API. I think we could make some incredible visual tooling with it. I've done some work in this area #18744 along with some (not so polished) UI's, I'll evaluate how many changes there are to see if I can update my old library.
dwisiswant0 commentedon Dec 16, 2025
+1, this has significantly impacted me as well. (#76849)
I also tried to work around this by extracting individual profiles only, but that doesn’t help. Running
go tool trace -pprof={net,sync,syscall,sched}consistently results in the process being OOM-killed, which suggests the full trace is still being parsed/expanded even when only a single profile is requested.mknyszek commentedon Dec 19, 2025
@dwisiswant0 If you're up to try out something that should work better, give https://go.dev/cl/731509 a try.
I could definitely use your feedback, but be wary of rough edges. 🙈
EDIT: I've made a good bit of progress and sanded down quite a few rough edges. Still, be wary of bugs.
gopherbot commentedon Dec 19, 2025
Change https://go.dev/cl/706375 mentions this issue:
internal/trace: add Index for scalably exploring a tracegopherbot commentedon Dec 19, 2025
Change https://go.dev/cl/731504 mentions this issue:
cmd/trace: parse trace events on demandgopherbot commentedon Dec 19, 2025
Change https://go.dev/cl/731509 mentions this issue:
cmd/trace: reorient web UI in terms of trace slice selectionsromaindoumenc commentedon Mar 23, 2026
I tried with patchset 20 (GOVERSION='go1.27-devel_578e81d808'), and I think the full trace is still processed in memory – getting OOM killed after eating up all 32GB of RAM on a 2GB trace.
Happy to share the trace itself if it helps @mknyszek .
mknyszek commentedon Mar 23, 2026
Please do share the trace. It definitely doesn't load the whole trace into memory anymore, but it does keep summary information around that should be relatively compact.
romaindoumenc commentedon Mar 23, 2026
Please find the trace attach (it is a bit of a gzip bomb, uncompressing to 1.2GB), shasum
ba09eeb5f630d849ee6059b3fc7aeff242e9e0ba.The code generating said trace seems to be stuck in an expensive loop (I am investigating the actual root cause, I can revert back if this is any use).
trace_example_loop.gz
mknyszek commentedon Mar 23, 2026
Oh no, almost the entirety of this trace is region start/end events.
Region events are one of the few events retained by the summary structure, hence the enormous memory usage. Worse still, these region events are generated in a trace that only has 2 generations (see the
EndOfGeneration) which are the split points that allow the trace parser to observe only a subset of the trace.I honestly am not sure what to do about this. Without summary information, choosing how to slice and dice the trace is quite hard because you don't have a sense of what's going on and what might be interesting to look at. We can probably make things slightly better by not keeping the actual events around, but that wouldn't help all that much in this case since the summary information needs most of the region information anyway to produce useful summaries (like histograms of regions and such). We could also possibly produce the region summary information on-demand? That would require more re-architecting and I probably won't have time to get to that.
We could maybe do something more user-friendly and tell the user that there's a problem and just not show summary information at all in such cases, but I'm also not sure what heuristic we should use for that.
mknyszek commentedon Mar 23, 2026
@prattmic also notes that we could do better here by having more generations. In this particular case we could add splitting by trace size, which is something I considered long ago but never added.
Though, that wouldn't help the OOM issue, since the summary structures would still retain ~99.9% of events anyway.