Repository navigation
Intermittent "Internal CLR error. (0x80131506)" on Windows 11 ARM #131636
Copy link
Copy link
Open
PowerShell/PowerShell
#28186Labels
arch-arm64area-VM-coreclruntriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner
Description
Description
Originally reported here: #86775 (comment)
I have recently seen an error happening seemingly randomly on Windows 11 ARM GitHub Actions runners as part of the actions/setup-dotnet action (example):
Current runner version: '2.336.0'
Runner Image Provisioner
Hosted Compute Agent
Version: 20260707.563
Commit: 02667638d2b423fbc733a8e32a88b44996a3ba6e
Build Date: 2026-07-07T19:33:50Z
Worker ID: {b6f9993e-1226-410e-abd5-7b6c5c143708}
Azure Region: westus2
Operating System
Microsoft Windows 11
10.0.26200
Enterprise
Runner Image
Image: windows-11-arm64
Version: 20260719.114.1
Included Software: https://github.com/actions/runner-images/blob/win11-arm64/20260719.114/images/windows/Windows11-Arm64-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/win11-arm64%2F20260719.114
Run actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command & 'C:\a\_actions\actions\setup-dotnet\a98b56852c35b8e3190ac28c8c2271da59106c68\externals\install-dotnet.ps1' -SkipNonVersionedFiles -Runtime dotnet -Channel LTS
dotnet-install: .NET Core Runtime with version '10.0.10' is already installed.
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command & 'C:\a\_actions\actions\setup-dotnet\a98b56852c35b8e3190ac28c8c2271da59106c68\externals\install-dotnet.ps1' -SkipNonVersionedFiles -Channel 8.0
Fatal error.
Internal CLR error. (0x80131506)
Error: Failed to install dotnet, exit code: 3221225477. Fatal error.
Internal CLR error. (0x80131506)
A separate more detailed occurrence (example):
Current runner version: '2.336.0'
Runner Image Provisioner
Hosted Compute Agent
Version: 20260707.563
Commit: 02667638d2b423fbc733a8e32a88b44996a3ba6e
Build Date: 2026-07-07T19:33:50Z
Worker ID: {86c09818-4216-413f-beed-e895ad04d8f9}
Azure Region: westus2
Operating System
Microsoft Windows 11
10.0.26200
Enterprise
Runner Image
Image: windows-11-arm64
Version: 20260727.122.1
Included Software: https://github.com/actions/runner-images/blob/win11-arm64/20260727.122/images/windows/Windows11-Arm64-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/win11-arm64%2F20260727.122
Fatal error.
Internal CLR error. (0x80131506)
at System.Runtime.CompilerServices.CastHelpers.ChkCastAny_NoCacheLookup(Void*, System.Object)
at System.Runtime.CompilerServices.CastHelpers.ChkCast_Helper(Void*, System.Object)
at Microsoft.PowerShell.TaskbarJumpList.CreateElevatedEntry(System.String)
at Microsoft.PowerShell.TaskbarJumpList+<>c.<CreateRunAsAdministratorJumpList>b__0_0()
at System.Threading.Thread+StartHelper.Callback(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
Error: Failed to install dotnet, exit code: 3221225477. Fatal error.
Internal CLR error. (0x80131506)
at System.Runtime.CompilerServices.CastHelpers.ChkCastAny_NoCacheLookup(Void*, System.Object)
at System.Runtime.CompilerServices.CastHelpers.ChkCast_Helper(Void*, System.Object)
at Microsoft.PowerShell.TaskbarJumpList.CreateElevatedEntry(System.String)
at Microsoft.PowerShell.TaskbarJumpList+<>c.<CreateRunAsAdministratorJumpList>b__0_0()
at System.Threading.Thread+StartHelper.Callback(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
Reproduction Steps
Run actions/setup-dotnet on a windows-11-arm GitHub Actions runner.
Expected behavior
The action does not error.
Actual behavior
The CLR fails with an internal error Internal CLR error. (0x80131506).
Regression?
No response
Known Workarounds
Retry the job.
Configuration
.NET 10.0.10 running on Microsoft Windows 11 10.0.26200 Enterprise on ARM.
Other information
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
arch-arm64area-VM-coreclruntriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner
Type
Projects
- StatusShow more project fieldsNo status
Activity
dotnet-policy-service commentedon Jul 31, 2026
Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.
martincostello commentedon Sep 29, 2026
Still ongoing with the 10.0.401 SDK.
This is quite disruptive to the OpenTelemetry .NET SDK CI when it causes merge queue builds to fail.
ci: launch Windows terminal tests under `conhost`
martincostello commentedon Oct 12, 2026
Note
The dump analysis in this comment was done with the assistance of Claude Code.
We captured a full crash dump of this on a
windows-11-armrunner and did some analysis. This is only one dump, but it points at the runtime rather thansetup-dotnetor PowerShell script logic.Environment
windows-11-vs2026-arm6420261004.176.1, Windows 11 10.0.26200C:\Program Files\PowerShell\7\pwsh.exe, which bundles its own CoreCLR10.0.1226.41902(.NET 10.0.12), ARM64pwshwas launched, and ~180 ms after the previouspwshlaunched bysetup-dotnethad exited.What crashed
The faulting thread is the unnamed thread that PowerShell starts for
TaskbarJumpList.CreateRunAsAdministratorJumpList(a new STAThreadrunningCreateElevatedEntry, which makes several COM calls).The fault is at
coreclr+0x604ac, a null dereference (ldr x8,[x8,#0x38]withx8 == 0). The .NET Runtime event (ID 1023) reports the same IP,0x00007FF9EE4304ACin a module based at0x00007FF9EE3D0000.Matching the disassembly against the
v10.0.12sources, this isStubHelpers::GetCOMIPFromRCW(src/coreclr/vm/stubhelpers.cpp), the FCALL used for every CLR-to-COM call:x0 = this,x1 = MethodDesc(a CLR-to-COM method desc with slot 2) andx2 = &ppTarget.PassiveGetSyncBlock()reads the 32-bit object header atobj-4, checks bit 27 (BIT_SBLK_IS_HASH_OR_SYNCBLKINDEX), and uses the low 26 bits as an index intog_pSyncTable. If the bit isn't set, the index is 0, andg_pSyncTable[0].m_SyncBlockis NULL.->GetInteropInfoNoCreate()that follows is the faulting dereference (SyncBlock + 0x38). There is no null check.The receiver's header was lost
The receiver is a 24-byte
System.__ComObjectat0x1a457ca34b0. Judging by the call order inCreateElevatedEntry, it looks like theout object instancefrom the secondCoCreateInstance, and the call is probablypShortCutCollection.AddObject(...). I inferred that from the method slot and call order, not from symbols.Its header word is
0x00000000, but everything the runtime keeps for it is intact:g_pSyncTable[9]{ m_SyncBlock = 0x1a453a234c8, m_Object = 0x1a457ca34b0 }, so the sync table points back at this objectm_dwSyncIndex = 0x80000009,m_pInteropInfo(+0x38) non-nullobj-4)0x00000000, but it should be0x08000009The adjacent COM objects created earlier by the same method have correct headers:
0x08000004,0x08000005and0x08000006(two__ComObjects and theCShellLink). SoCreateObjectRefdidGetSyncBlock(), created the RCW and registered it, and the header index was later zero.SyncBlock::SetIndexis a compare-exchange loop, so I don't think the initial write was torn. Something cleared the header afterwards.ObjHeader::ResetIndexandGCResetIndexexist, so a cleanup path or a race with another runtime thread would be consistent with this. I can't tell which from one dump without symbols. Two other runtime threads were mid-execution incoreclrwhen the dump was taken.Caveats
ChkCastAny_NoCacheLookupcalled fromCreateElevatedEntry) differs from where this dump faults (GetCOMIPFromRCW). I suspect the printed managed stack isn't reliable here because the FCALL has no frame, but I haven't proven it.coreclr.pdbfor this build returns 404 frommsdl.microsoft.com, so I worked from exports and the public source.cdb.exe, so I couldn't use SOS or the DAC on an ARM64 dump. I wasn't able to name the type of the object or inspect GC state.Capturing a dump
WER LocalDumps and the Application log (IDs 1000/1026) showed nothing, because the runtime's fatal-error path terminates the process itself. These environment variables, set before
setup-dotnet, produced the dump (and the createdump output in the step log):Setting these from a
cmdstep avoids depending onpwsh, which is the process that crashes.What would help
coreclr.pdbfor10.0.1226.41902(ARM64), or access to an ARM64 machine to run!syncblkand!dumpheapagainst the dump.martincostello commentedon Oct 12, 2026
Further Claude Code analysis from a second dump.
Update: second dump, same signature, now confirmed by SOS
A second failure (CoreCLR 10.0.1226.41902, ARM64, same image) gave a second full dump with the same fault offset (
coreclr+0x604ac,GetCOMIPFromRCWnull deref). Running SOS on an ARM64 runner against it,verifyheapreports:__ComObjectcreated byTaskbarJumpList.CreateElevatedEntry, sync block index 9 both times. Its sync table entry, sync block, interop info and RCW are intact, but its object header is zero.CShellLinkfrom the same method have correct headers (0x08000003..05in this dump).GCResetIndex(syncblk.h) is a plain, non-atomic read-modify-write that clears the index bits and is called fromSyncBlockCache::GCWeakPtrScanElement. I couldn't see what cleared this header, but a GC that leaves a live object's header zeroed would explain the null dereference.It reproduced about 1 in 560 job runs (38 successes before the failure). I haven't tested whether GC frequency matters; I'm planning to try
DOTNET_GCgen0MaxBudgetto see if it raises the rate.martincostello commentedon Oct 12, 2026
Additional detail from a stress run with Claude Code.
Update: this reproduces on x64 with a small gen0 budget, and it doesn't depend on the GC implementation
I can reproduce this on a plain x64 Windows 11 machine (PowerShell 7.6.6 / CoreCLR 10.0.12) with no CI involved. The crash needs PowerShell's jump-list thread, and a smaller gen0 budget makes it much more likely.
Minimal repro (Node because it spawns without hiding the window, so PowerShell runs
TaskbarJumpList; launching from cmd/pwsh doesn't set the show-window startup flags and the jump list is skipped):Run it with
DOTNET_GCgen0MaxBudget=40000(256 KiB, hex). It crashes within a few seconds to a minute withFatal error. Internal CLR error (0x80131506)and theChkCastAny_NoCacheLookup/CreateElevatedEntrystack from the report.Results (all against the same pwsh build):
DOTNET_GCgen0MaxBudget=100000(1 MiB), ARM64: 4 crashes in 493 launchesDOTNET_GCgen0MaxBudget=40000(256 KiB), ARM64: 6 crashes in 143 launchesDOTNET_GCgen0size=4000000(64 MiB), ARM64: 0 crashes in 9,624 launchesDOTNET_GCgen0MaxBudget=40000also crashes withDOTNET_gcConcurrent=0and withDOTNET_GCName=clrgc.dll(the segments GC), so it is not specific to the regions GC or to background GCIn every dump I analysed,
verifyheapreportsSyncBlockMismatch ... System.__ComObject should have a SyncBlock index of N but instead had no SyncBlock. The object is always a 24-byte__ComObjectin gen1. Sync table entry, sync block and RCW are intact; only the object header's sync-block index is gone (header word0x00000000). The two crash sites I see are the same corruption observed by different readers:StubHelpers::GetCOMIPFromRCWreadsPassiveGetSyncBlock()->GetInteropInfoNoCreate()with no null check, so it dereferences null.RCWHolder::Init(reached fromChkCastAny_NoCacheLookupviaComObject::SupportsInterface) calls the creatingGetSyncBlock(), which sees no index and allocates a second, empty sync block. In one dump the same__ComObjectowns sync blocks 6 (with the RCW) and 8 (empty), andGetInteropInfoNoCreate()returns null.I haven't found what clears the header.
ObjHeader::GCResetIndex()(a non-atomic read-modify-write) is the only header-clearing code I found besides the shutdown-onlyResetIndex, but I can't show it runs on these objects.martincostello commentedon Oct 12, 2026
Note
The analysis in this comment was done with the assistance of Claude Code.
Root cause found: PowerShell's
PropVariantoverflows into the header of the next object (standalone repro included)This is not a runtime bug, and I think this is actually an issue for PowerShell/PowerShell.
What happens
Microsoft.PowerShell.PropVariant(source) is a managed class with explicit-layout fields at offsets 0 and 8, so a managed instance holds 16 bytes of data. It is passed to ole32PropVariantClearas a nativePROPVARIANT, which is 24 bytes on 64-bit.PropVariantClearzeroes the whole structure, so it overwrites the 8 bytes after the managed object. Those 8 bytes are the header word of the next object allocated, which holds its sync block index.In
TaskbarJumpList.CreateElevatedEntrythe next object is the__ComObjectreturned by the secondCoCreateInstance. Once its header is zeroed the runtime can no longer find its sync block, so the next cast or COM call on it crashes (ChkCastAny_NoCacheLookup, or a null interop info inGetCOMIPFromRCW).PropVarianthas a finalizer that callsDispose, andCreateElevatedEntrynever disposes it, so the clear usually happens on the finalizer thread after a GC finds the object dead. That is why the crash rate follows GC frequency (on the ARM64 runners I saw it go from about 1 in 2,000pwshlaunches to 1 in 24 withDOTNET_GCgen0MaxBudget=40000, and to none in 9,624 launches withDOTNET_GCgen0size=4000000).This matches everything in the crash dumps I captured on
windows-11-armrunners:verifyheapreportsSyncBlockMismatch ... System.__ComObject should have a SyncBlock index of N but instead had no SyncBlockon a 24-byte__ComObjectin gen1, the sync table entry, sync block and RCW are intact, and the three words before the object are all zero (a cleared 16 byte object followed by the object's own header).Repro without PowerShell
Requirements: Windows and the .NET 10 SDK (single-file apps need
dotnet run file.cs). I ran it on .NET 10.0.12 x64; the original crashes were on ARM64 runners, and I have not run the repro there.PropVariantOverflow.cs.dotnet run PropVariantOverflow.cs0x08000001to0x00000000, printsREPRODUCED, exit code 1.dotnet run PropVariantOverflow.cs -- --crashFatal error. Internal CLR error. (0x80131506)andat System.Runtime.CompilerServices.CastHelpers.ChkCastAny_NoCacheLookup, as in the original report (exit code 139).dotnet run PropVariantOverflow.cs -- --crash --finalizerDispose, which is how it happens inpwsh.--fixedPropVariantwith a 24 byte managed layout. The victim header is unchanged and the program exits normally.Notes:
unsafehelper to show the corruption before the crash. The victim is the object allocated directly after thePropVariant, which is why the output prints the distance between them (32 means adjacent).--finalizer: any GC legitimately discards the empty sync block of a plain object, whereas a COM object keeps its sync block because it holds the RCW.#:property PublishAot=falseis needed because file-based apps otherwise disable built-in COM interop (Built-in COM has been disabled via a feature switch).PropVariantis allocated the program makes one throwawayCoCreateInstancecall, becauseCreateElevatedEntryhas already made one (for the destination list) and the first call allocates helper objects that would sit between the two objects.PropVariantOverflow.csSuggested fix for PowerShell
Give the managed class the full size of the native structure, for example by adding a field at offset 16:
[StructLayout(LayoutKind.Explicit, Size = 24)]is not enough: the CLR ignoresSizefor the managed size of a class, so the instance stays at 16 bytes (I checked: the distance to the next object stays at 32). Disposing the instance explicitly inCreateElevatedEntrywould not help, becauseDisposecallsPropVariantClearas well, so the size fix is the one that matters.