runtime: fatal error: found bad pointer in Go heap when compiling on amd64 host for arm64 (QEMU)

@wadey

Update(2026-02-18): This is a QEMU bug, which is fixed in the new versions of QEMU. QEMU 10.0.6+ is known to work. 7.1.0+ may also work.
For users with an older version of QEMU, there is a workaround on the Go side: set GODEBUG=madvdontneed=0.

This splits off the QEMU emulation issue from #74552 (comment).

When running go install on amd64 host with qemu arm64 emulation it seems to crash.

#27 [linux/arm64 tools 4/4] RUN --mount=type=cache,target=/root/.cache/go-build,id=gobuild     --mount=type=cache,target=/go/pkg/mod,id=gopkg     go install ./...
#27 0.888 runtime: pointer 0x39d8218b4000 to unallocated span span.base()=0x39d8218c0000 span.limit=0x39d8218c1ec0 span.state=0
#27 0.896 fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)
#27 0.901 
#27 0.901 runtime stack:
#27 0.903 runtime.throw({0x71ec51?, 0x49e000?})
#27 0.904 	runtime/panic.go:1229 +0x38 fp=0x39d821579ea0 sp=0x39d821579e70 pc=0x93918
#27 0.905 runtime.badPointer(0x554740a080, 0x39d8218b4000, 0x0, 0x0)
#27 0.905 	runtime/mbitmap.go:1334 +0x128 fp=0x39d821579ef0 sp=0x39d821579ea0 pc=0x330f8
#27 0.906 runtime.findObject(0x39d8218a4000?, 0x55475485a8?, 0x39d8217bbc20?)
#27 0.906 	runtime/mbitmap.go:1386 +0xac fp=0x39d821579f30 sp=0x39d821579ef0 pc=0x9213c
#27 0.906 runtime.wbBufFlush1(0x39d821311808)
#27 0.906 	runtime/mwbbuf.go:243 +0x120 fp=0x39d821579fa0 sp=0x39d821579f30 pc=0x56cb0
#27 0.906 runtime.wbBufFlush.func1()
#27 0.906 	runtime/mwbbuf.go:181 +0x24 fp=0x39d821579fc0 sp=0x39d821579fa0 pc=0x8e8f4
#27 0.906 runtime.systemstack(0x39d821576000)
#27 0.906 	runtime/asm_arm64.s:399 +0x68 fp=0x39d821579fd0 sp=0x39d821579fc0 pc=0x992a8
#27 0.907

and

#24 1.163 runtime: marked free object in span 0x55474772a8, elemsize=128 freeindex=1 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
#24 1.172 0x7b979736000 alloc unmarked
#24 1.177 0x7b979736080 free  unmarked
#24 1.178 0x7b979736100 free  unmarked
#24 1.178 0x7b979736180 free  unmarked
#24 1.178 0x7b979736200 free  unmarked
...
#24 1.181 fatal error: found pointer to free object
#24 1.197 
#24 1.197 runtime stack:
#24 1.198 runtime.throw({0x703f56?, 0x0?})
#24 1.199 	runtime/panic.go:1229 +0x38 fp=0x7b97932ba00 sp=0x7b97932b9d0 pc=0x93918
#24 1.200 runtime.(*mspan).reportZombies(0x55474772a8)
#24 1.200 	runtime/mgcsweep.go:893 +0x314 fp=0x7b97932ba80 sp=0x7b97932ba00 pc=0x4a834

Full log can be found at https://gist.github.com/egonelbre/15e79abdd3d298028791d33f2392d865

The relevant information about qemu version:

$ docker buildx create --name multiplatform --driver docker-container --bootstrap
$ docker exec buildx_buildkit_multiplatform0 buildkit-qemu-aarch64 --version
qemu-aarch64 version 10.0.4 (v10.0.4)
Copyright (c) 2003-2025 Fabrice Bellard and the QEMU Project developers

$ docker exec buildx_buildkit_multiplatform0 uname -a
Linux 6f5951803e5a 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 Linux

One common denominator seems to be Ubuntu 22.04 host that also @wadey mentioned.

A related issue on arm64 #76985.

Looks like a latest Ubuntu 22.04 does not trigger the issue, so there must be some additional important detail:

First attempt to setup on a Ubuntu 22.04 VM

# Add Docker's official GPG key:
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

# Install Docker
VERSION_STRING=5:29.1.5-1~ubuntu.22.04~jammy
sudo apt install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-

# Start Docker
sudo systemctl start docker

# Install qemu emulation that buildkit uses
sudo docker run --privileged --rm tonistiigi/binfmt:qemu-v10.0.4 --install arm64

# Run tests
sudo docker run -it --rm --platform linux/arm64 golang:1.26 go test -v regexp

Some additional details, in case they are relevant:

Working system kernel:

user@host:~$ uname -a
Linux instance-20260212-065741 6.8.0-1046-gcp #49~22.04.1-Ubuntu SMP Mon Jan 19 12:29:23 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

Non-working system kernel:

user@host:~# uname -a
Linux build-1 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

lscpu working system

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          48 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   2
  On-line CPU(s) list:    0,1
Vendor ID:                AuthenticAMD
  Model name:             AMD EPYC 7B12
    CPU family:           23
    Model:                49
    Thread(s) per core:   2
    Core(s) per socket:   1
    Socket(s):            1
    Stepping:             0
    BogoMIPS:             4499.99
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb
                           rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic mo
                          vbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext ssbd ibr
                          s ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsav
                          eerptr arat npt nrip_save umip rdpid
Virtualization features:  
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):      
  L1d:                    32 KiB (1 instance)
  L1i:                    32 KiB (1 instance)
  L2:                     512 KiB (1 instance)
  L3:                     16 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0,1
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Mitigation; untrained return thunk; SMT enabled with STIBP protection
  Spec rstack overflow:   Mitigation; Safe RET
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected
  Vmscape:                Not affected

lscpu non-working system

Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             46 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      12
  On-line CPU(s) list:       0-11
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Xeon(R) CPU @ 2.30GHz
    CPU family:              6
    Model:                   63
    Thread(s) per core:      2
    Core(s) per socket:      6
    Socket(s):               1
    Stepping:                0
    BogoMIPS:                4599.99
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma
                              cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat md_clear arch_capabilitie
                             s
Virtualization features:
  Hypervisor vendor:         KVM
  Virtualization type:       full
Caches (sum of all):
  L1d:                       192 KiB (6 instances)
  L1i:                       192 KiB (6 instances)
  L2:                        1.5 MiB (6 instances)
  L3:                        45 MiB (1 instance)
NUMA:
  NUMA node(s):              1
  NUMA node0 CPU(s):         0-11
Vulnerabilities:
  Gather data sampling:      Not affected
  Indirect target selection: Mitigation; Aligned branch/return thunks
  Itlb multihit:             Not affected
  L1tf:                      Mitigation; PTE Inversion
  Mds:                       Mitigation; Clear CPU buffers; SMT Host state unknown
  Meltdown:                  Mitigation; PTI
  Mmio stale data:           Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Reg file data sampling:    Not affected
  Retbleed:                  Mitigation; IBRS
  Spec rstack overflow:      Not affected
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI SW loop, KVM SW loop
  Srbds:                     Not affected
  Tsa:                       Not affected
  Tsx async abort:           Not affected
  Vmscape:                   Not affected

Which version of QEMU is used on the working vs non-working systems? (Edit: Ah, you included the version on the non-working system already, and the setup script specifies the same version, so I suppose it's the same in both cases.)

You don't mention flakes, so I assume this is quite reproducible? Most/all builds fail?

cc @golang/runtime @golang/compiler

If this works OK with 1.25 and fails reliably with 1.26, I think the most helpful thing you can do at the moment would be to bisect the Go between 1.25 and 1.26 to try to pinpoint which change caused the failure.

My immediate suspicion is that this is a limitation/bug in QEMU's arm64 translation that some change in Go has started triggering.

Bad host 1 (this is a c6a.large AWS instance in us-east-1):

Ubuntu 22.04.5 LTS

binfmt-support 2.2.1-2
qemu-user-static 1:6.2+dfsg-2ubuntu6.27
docker.io      28.2.2-0ubuntu1~22.04.1

6.8.0-1044-aws #46~22.04.1-Ubuntu SMP Tue Dec  2 12:52:18 UTC 2025 x86_64

lscpu

$ lscpu 

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          48 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   2
  On-line CPU(s) list:    0,1
Vendor ID:                AuthenticAMD
  Model name:             AMD EPYC 7R13 Processor
    CPU family:           25
    Model:                1
    Thread(s) per core:   2
    Core(s) per socket:   1
    Socket(s):            1
    Stepping:             1
    BogoMIPS:             5299.99
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_
                          freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bm
                          i1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid
Virtualization features:  
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):      
  L1d:                    32 KiB (1 instance)
  L1i:                    32 KiB (1 instance)
  L2:                     512 KiB (1 instance)
  L3:                     4 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0,1
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Mitigation; Safe RET
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected
  Vmscape:                Not affected

Running this test causes a different crash each time within moments.

$ docker run --rm --platform linux/arm64 golang:1.26 env GOMAXPROCS=1 go test crypto/...

panic: runtime error: growslice: len out of range

goroutine 12 [running]:
cmd/go/internal/modindex.importRaw({0x1d35fbcfc2b8, 0x11}, {0x1d35fbe40492, 0x7})
	cmd/go/internal/modindex/scan.go:263 +0x954
cmd/go/internal/modindex.indexPackage({0x1d35fbcfc2b8, 0x11}, {0x1d35fbe40480, 0x19})
	cmd/go/internal/modindex/scan.go:86 +0x60
...

$ docker run --rm --platform linux/arm64 golang:1.26 env GOMAXPROCS=1 go test crypto/...

unexpected fault address 0xffffffffff68e000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xffffffffff68e000 pc=0x14bc78]

goroutine 12 gp=0x1ff1e54ce000 m=2 mp=0x1ff1e53f2808 [running]:
runtime.throw({0x6f0394?, 0x5547458520?})
	runtime/panic.go:1229 +0x38 fp=0x1ff1e548cd90 sp=0x1ff1e548cd60 pc=0x93918
runtime.sigpanic()
	runtime/signal_unix.go:945 +0x21c fp=0x1ff1e548cdf0 sp=0x1ff1e548cd90 pc=0x95e3c
go/token.(*node).safeHeight(...)
	go/token/tree.go:208
...

$ docker run --rm --platform linux/arm64 golang:1.26 env GOMAXPROCS=1 go test crypto/...

runtime: marked free object in span 0x554742a640, elemsize=24 freeindex=9 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
0x1e1452dd8000 alloc unmarked
0x1e1452dd8018 alloc unmarked
0x1e1452dd8030 alloc unmarked
...

Bad host 2 (this is a c5.4xlarge AWS instance in us-east-1):

Ubuntu 22.04.5 LTS

binfmt-support 2.2.1-2
qemu-user-static 1:6.2+dfsg-2ubuntu6.27
docker.io      28.2.2-0ubuntu1~22.04.1

6.8.0-1045-aws #47~22.04.1-Ubuntu SMP Thu Jan 29 21:28:23 UTC 2026 x86_64

lscpu

$ lscpu 

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   16
  On-line CPU(s) list:    0-15
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
    CPU family:           6
    Model:                85
    Thread(s) per core:   2
    Core(s) per socket:   8
    Socket(s):            1
    Stepping:             4
    BogoMIPS:             5999.99
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmul
                          qdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx
                          512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
Virtualization features:  
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):      
  L1d:                    256 KiB (8 instances)
  L1i:                    256 KiB (8 instances)
  L2:                     8 MiB (8 instances)
  L3:                     24.8 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-15
Vulnerabilities:          
  Gather data sampling:   Unknown: Dependent on hypervisor status
  Itlb multihit:          KVM: Mitigation: VMX unsupported
  L1tf:                   Mitigation; PTE Inversion
  Mds:                    Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Reg file data sampling: Not affected
  Retbleed:               Vulnerable
  Spec rstack overflow:   Not affected
  Spec store bypass:      Vulnerable
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline
  Srbds:                  Not affected
  Tsx async abort:        Not affected
  Vmscape:                Not affected

Same crashes as bad host 1

Good host 1 (this is a m6i.large AWS instance in us-east-1):

Ubuntu 24.04 LTS

binfmt-support 2.2.2-7
qemu-user-static 1:8.2.2+ds-0ubuntu1.12
docker.io      28.2.2-0ubuntu1~24.04.1

6.8.0-1008-aws #8-Ubuntu SMP Sat Apr 20 00:46:25 UTC 2024 x86_64

lscpu

$ lscpu 

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   2
  On-line CPU(s) list:    0,1
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
    CPU family:           6
    Model:                106
    Thread(s) per core:   2
    Core(s) per socket:   1
    Socket(s):            1
    Stepping:             6
    BogoMIPS:             5799.91
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmul
                          qdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invp
                          cid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd ida arat avx512vbmi pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bit
                          alg tme avx512_vpopcntdq rdpid md_clear flush_l1d arch_capabilities
Virtualization features:  
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):      
  L1d:                    48 KiB (1 instance)
  L1i:                    32 KiB (1 instance)
  L2:                     1.3 MiB (1 instance)
  L3:                     54 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0,1
Vulnerabilities:          
  Gather data sampling:   Unknown: Dependent on hypervisor status
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Mitigation; Clear CPU buffers; SMT Host state unknown
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI Syscall hardening, KVM SW loop
  Srbds:                  Not affected
  Tsx async abort:        Not affected

No crashes with the test.

Bisecting is going to be difficult because I think I would have to cross compile Go for each bisect version for arm64? Unless perhaps we can bisect using the tip docker images, like golang:tip-20260124 etc (and I'm not sure docker hub keeps that many of these historically)

@wadey

@wadey I think you can use the same host to cross-compile you a binary and then hoist the binary into a docker image with mounting a directory.

QEMU user space emulation have been known to be buggy from time to time. If I recall correctly, one problem it had was it can give addresses that are actually amd64-like, which would never occur with real arm64 machine/kernel.

A quick bisect with available tip versions.

On "Bad host 1" from above, golang:tip-20251026 passes and golang:tip-20251102 fails.

$ docker run --rm --platform linux/arm64 golang:tip-20251026 go version
go version go1.26-devel_54e3adc Sun Oct 26 14:11:34 2025 -0700 linux/arm64

$ docker run --rm --platform linux/arm64 golang:tip-20251102 go version
go version go1.26-devel_388c41c Sun Nov 2 07:51:05 2025 -0800 linux/arm64

54e3adc...388c41c

I have a guess that it might be this bug, that was fixed later in 2022 (lines up with qemu in 22.04 being the issue): https://gitlab.com/qemu-project/qemu/-/issues/326

And this commit is in the bisect range: 2793728

It looks like this fix is in qemu v7.1.0 and later, and my bad hosts have v6.2

@nikpivkin which QEMU version are you using?

qemu-x86_64 --version
qemu-x86_64 version 7.0.0 (v7.0.0)

I know that to solve the problem, you need to upgrade the QEMU version, but I tried to use the suggested workaround.

@egonelbre

Thanks for the investigation. As @egonelbre mentioned, it is a QEMU bug that has been fixed in the new versions of QEMU, and there is a simple workaround on the Go side (GODEBUG=madvdontneed=0), we can close this. Thanks.

I edited the top post to mention the version and the workarund.

I have a guess that it might be this bug, that was fixed later in 2022 (lines up with qemu in 22.04 being the issue): https://gitlab.com/qemu-project/qemu/-/issues/326

And this commit is in the bisect range: 2793728

It looks like this fix is in qemu v7.1.0 and later, and my bad hosts have v6.2

For posterity since it isn't directly linked from the issue, the merged fix commit is https://gitlab.com/qemu-project/qemu/-/commit/892a4f6a750abceeda4c1ee8324f58f82fd6bd89, which is indeed in v7.1.0.

Note that this fix does not fix cases where the host page size is larger than the guest page size. In that case, MADV_DONTNEED is still a no-op. I could imagine this being a problem for arm64 hosts with amd64 guests, as arm64 can have a larger page size.

Change https://go.dev/cl/746600 mentions this issue: MinimumRequirements: add QEMU user space emulator

I have a few meta questions for folks affected by this issue:

This issue was reported shortly after the Go 1.26.0 release. Though this ended up being a QEMU bug, we'd of course prefer to find subtle issues like this before the final release. We release Go release candidates for around two months prior to final releases, which we encourage users to test and find issues not covered by our own testing.

For folks that encountered this issue, I'm curious:

Did you test with a Go 1.26 release candidate? If not, why not? Lack of awareness, some source of friction making it difficult, etc?
Do you have any ideas of things we might do to make release candidate testing easier / make you more likely to test?

To be clear, this isn't about QEMU specifically, I've just been thinking about ways we improve our prerelease process to find more bugs before the final release and I figure that a group of folks hit by an issue at release is a good audience for questions. :)

Perhaps if you don't have the bandwidth to update your base os packages like docker / qemu, you also don't have the bandwidth to test pre releases...

Yes, I usually start using rc versions locally whenever they come out. I don't test them in CI with full testsuite every release, it depends on how much time I have available. Currently the QEMU problem was in a different part of the system that was used to build some images i.e. a part of the system I didn't consider.

I feel like rc testing is easy enough.

As for updating base OS packages. QEMU 6.20 is the current version in Ubuntu 22.04 LTS, which is supported until 2027. So just an apt update is not sufficient.

MinimumRequirements: add QEMU user space emulator

ci: work around QEMU Go compiler panic (golang/go#77572)

Re: the "QEMU 10.0.6+ is known to work" note — qemu-aarch64 10.2.3 (shipped today by tonistiigi/binfmt:latest, the standard GitHub Actions/Buildx arm64 registration) still corrupts for us - at Datadog - under GOMEMLIMIT-driven scavenger pressure, with the exact signatures from this issue ("marked free object in span", "found bad pointer in Go heap", plus acquireSudog / concurrent-map-writes variants). Minimal 24-line reproducer, 20-run rates (default 15/20 crash; GODEBUG=madvdontneed=0 20/20 clean), and a QEMU-side report: https://gitlab.com/qemu-project/qemu/-/work_items/4654. The workaround from this issue remains effective; we now scope it automatically to emulated arm64 runs in our CI.

Navigation Menu

Repository navigation

runtime: fatal error: found bad pointer in Go heap when compiling on amd64 host for arm64 (QEMU) #77572

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

runtime: fatal error: found bad pointer in Go heap when compiling on amd64 host for arm64 (QEMU) #77572

Description

Activity

egonelbre commented on Feb 12, 2026

prattmic commented on Feb 12, 2026

prattmic commented on Feb 12, 2026

prattmic commented on Feb 12, 2026

wadey commented on Feb 12, 2026

wadey commented on Feb 12, 2026

wadey commented on Feb 12, 2026

wadey commented on Feb 12, 2026

egonelbre commented on Feb 12, 2026

cherrymui commented on Feb 12, 2026

wadey commented on Feb 12, 2026

wadey commented on Feb 12, 2026

13 remaining items

nikpivkin commented on Feb 18, 2026

cherrymui commented on Feb 18, 2026

prattmic commented on Feb 18, 2026

gopherbot commented on Feb 18, 2026

prattmic commented on Feb 18, 2026

seankhliao commented on Feb 18, 2026

egonelbre commented on Feb 18, 2026

darccio commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions