Skip to content

Update to .NET 6 Preview 6 and use crossgen2 #15763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adityapatwardhan merged 21 commits into from
Jul 19, 2021
Merged

Update to .NET 6 Preview 6 and use crossgen2 #15763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
99c0b27
Update build to use crossgen2
adityapatwardhan Jul 13, 2021
a5e9a0c
Remove fallback to crossgen
adityapatwardhan Jul 13, 2021
9de4a23
Fix CA1846 issues
adityapatwardhan Jul 13, 2021
9e01831
Remove --no-restore switch from dotnet publish
adityapatwardhan Jul 14, 2021
d877f06
Update to .NET 6 Preview 6
adityapatwardhan Jul 14, 2021
7d941f6
Remove pdb generation for crossgen assembly as it fails on linux
adityapatwardhan Jul 14, 2021
e320bc2
Disable tests
adityapatwardhan Jul 14, 2021
ff274f6
Update the startup assemblies list
adityapatwardhan Jul 14, 2021
bc7a84c
Fix another parser test
adityapatwardhan Jul 14, 2021
dea13a0
Install crossgen package
adityapatwardhan Jul 15, 2021
b4e967b
Add logging
adityapatwardhan Jul 15, 2021
3dbec6f
Use x64 crossgen for all windows
adityapatwardhan Jul 15, 2021
cce7ee4
Remove crossgen install
adityapatwardhan Jul 15, 2021
75354ee
Add logging
adityapatwardhan Jul 16, 2021
42cb0fa
Verify PATH in test
adityapatwardhan Jul 16, 2021
88a662d
Add PATH
adityapatwardhan Jul 16, 2021
70bbe96
Add bootstrap
adityapatwardhan Jul 16, 2021
b7d7a55
Rollback since dependency is missing
adityapatwardhan Jul 16, 2021
f0e1f19
Cleanup debug messages
adityapatwardhan Jul 19, 2021
81fdb24
Disable tests for Test-Connection on macOS
adityapatwardhan Jul 19, 2021
8b22cc8
Address comments from Travis
adityapatwardhan Jul 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .vsts-ci/templates/windows-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ jobs:
displayName: Bootstrap

- pwsh: |
Import-Module .\build.psm1 -force
Start-PSBootstrap
Import-Module .\tools\ci.psm1
Restore-PSOptions -PSOptionsPath '$(System.ArtifactsDirectory)\build\psoptions.json'
$options = (Get-PSOptions)
Expand Down
1 change: 0 additions & 1 deletion PowerShell.Common.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,6 @@
<TargetFramework>net6.0</TargetFramework>
<LangVersion>9.0</LangVersion>
<PublishReadyToRun Condition=" '$(Configuration)' != 'Debug' ">true</PublishReadyToRun>
<PublishReadyToRunUseCrossgen2>false</PublishReadyToRunUseCrossgen2>

<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
Expand Down
34 changes: 31 additions & 3 deletions assets/wix/files.wxs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1620,6 +1620,27 @@
<Component Id="cmp68158D010ECB439582D041355BFACCBB">
<File Id="filB8694A0190CA422E9462870B514B813C" KeyPath="yes" Source="$(var.ProductSourcePath)\ref\System.Text.Encoding.CodePages.dll" />
</Component>
<Component Id="cmp690F24F04C044127A61754363A4D63E3">
<File Id="fil5BF6895A87684AC088DAB3759AA465BA" KeyPath="yes" Source="$(var.ProductSourcePath)\ref\Microsoft.Win32.Registry.dll" />
</Component>
<Component Id="cmpFAD6557C4AEC48B89DDC4E6485C020C4">
<File Id="fil47E7A45E0FDB48248C3741FCD3DA2368" KeyPath="yes" Source="$(var.ProductSourcePath)\ref\System.Security.AccessControl.dll" />
</Component>
<Component Id="cmpB5458B36303A472B80582AF1C2E7FB94">
<File Id="fil676439EF2BA242189585EA6953BF5A8C" KeyPath="yes" Source="$(var.ProductSourcePath)\ref\System.IO.FileSystem.AccessControl.dll" />
</Component>
<Component Id="cmp0EBD333D501A45089F58504074E229E7">
<File Id="fil474FACBE1BAA4CE6BBD7ED5CB4471A82" KeyPath="yes" Source="$(var.ProductSourcePath)\ref\System.IO.Pipes.AccessControl.dll" />
</Component>
<Component Id="cmpD11E9CA5069045578BA5D89C86F7FDEF">
<File Id="filB16D0834E1574612BF1B8D301434692C" KeyPath="yes" Source="$(var.ProductSourcePath)\ref\System.Security.Cryptography.Cng.dll" />
</Component>
<Component Id="cmp708913639F7A46E784CC0FA862A1A2CE">
<File Id="filC05CED23C9584EFE897E9919B0014851" KeyPath="yes" Source="$(var.ProductSourcePath)\ref\System.Security.Principal.Windows.dll" />
</Component>
<Component Id="cmpA75DEF5617C54DA1937CB37D5824BF79">
<File Id="fil3EE6096BD3C8407685E881F3CE6BF5C1" KeyPath="yes" Source="$(var.ProductSourcePath)\ref\System.Security.Cryptography.OpenSsl.dll" />
</Component>
</Directory>
<Directory Id="dirC72EEAF9434D4AD066E0F4D20D8E816C" Name="en-US">
<Component Id="cmpDC2EF4541FA5A2E63F36A8CD6C5FA51B">
Expand Down Expand Up @@ -3033,8 +3054,8 @@
<Component Id="cmp3547A55EA9D04784B4CF2FC717447E16">
<File Id="fil463DDA419769417A8D5A34C433E194F2" KeyPath="yes" Source="$(var.ProductSourcePath)\System.Speech.dll" />
</Component>
<Component Id="cmp378D79CF3DEC49979D7FA8A029141F0A">
<File Id="filC405FF3776264022973EF4C1E94C8C3B" KeyPath="yes" Source="$(var.ProductSourcePath)\mscordaccore_$(var.FileArchitecture)_$(var.FileArchitecture)_6.0.21.25307.dll" />
<Component Id="cmp8A922220F11C42439D6447B6654A8FC2">
<File Id="fil1EE3A80C6C944700A5C2B75C4AB30B21" KeyPath="yes" Source="$(var.ProductSourcePath)\mscordaccore_$(var.FileArchitecture)_$(var.FileArchitecture)_6.0.21.35212.dll" />
</Component>
</DirectoryRef>
</Fragment>
Expand Down Expand Up @@ -4013,7 +4034,14 @@
<ComponentRef Id="cmp2BB002B115C247FFB3353C80CAAC10B3" />
<ComponentRef Id="cmp68158D010ECB439582D041355BFACCBB" />
<ComponentRef Id="cmp3547A55EA9D04784B4CF2FC717447E16" />
<ComponentRef Id="cmp378D79CF3DEC49979D7FA8A029141F0A" />
<ComponentRef Id="cmp690F24F04C044127A61754363A4D63E3" />
<ComponentRef Id="cmpFAD6557C4AEC48B89DDC4E6485C020C4" />
<ComponentRef Id="cmpB5458B36303A472B80582AF1C2E7FB94" />
<ComponentRef Id="cmp0EBD333D501A45089F58504074E229E7" />
<ComponentRef Id="cmp8A922220F11C42439D6447B6654A8FC2" />
<ComponentRef Id="cmpD11E9CA5069045578BA5D89C86F7FDEF" />
<ComponentRef Id="cmp708913639F7A46E784CC0FA862A1A2CE" />
<ComponentRef Id="cmpA75DEF5617C54DA1937CB37D5824BF79" />
</ComponentGroup>
</Fragment>
</Wix>
184 changes: 78 additions & 106 deletions build.psm1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ function Get-EnvironmentInformation
if ($environment.IsWindows)
{
$environment += @{'IsAdmin' = (New-Object Security.Principal.WindowsPrincipal ([Security.Principal.WindowsIdentity]::GetCurrent())).IsInRole([Security.Principal.WindowsBuiltinRole]::Administrator)}
$environment += @{'nugetPackagesRoot' = "${env:USERPROFILE}\.nuget\packages"}
$environment += @{'nugetPackagesRoot' = "${env:USERPROFILE}\.nuget\packages", "${env:NUGET_PACKAGES}"}
}
else
{
Expand Down Expand Up @@ -419,7 +419,9 @@ Fix steps:

# setup arguments
# adding ErrorOnDuplicatePublishOutputFiles=false due to .NET SDk issue: https://github.com/dotnet/sdk/issues/15748
$Arguments = @("publish","--no-restore","/property:GenerateFullPaths=true", "/property:ErrorOnDuplicatePublishOutputFiles=false")
# removing --no-restore due to .NET SDK issue: https://github.com/dotnet/sdk/issues/18999
# $Arguments = @("publish","--no-restore","/property:GenerateFullPaths=true", "/property:ErrorOnDuplicatePublishOutputFiles=false")
$Arguments = @("publish","/property:GenerateFullPaths=true", "/property:ErrorOnDuplicatePublishOutputFiles=false")
if ($Output -or $SMAOnly) {
$Arguments += "--output", (Split-Path $Options.Output)
}
Expand Down Expand Up @@ -2323,78 +2325,85 @@ function Start-CrossGen {

function New-CrossGenAssembly {
param (
[Parameter(Mandatory= $true)]
[Parameter(Mandatory = $true)]
[ValidateNotNullOrEmpty()]
[String]
[String[]]
$AssemblyPath,
[Parameter(Mandatory= $true)]

[Parameter(Mandatory = $true)]
[ValidateNotNullOrEmpty()]
[String]
$CrossgenPath
$CrossgenPath,

[Parameter(Mandatory = $true)]
[ValidateSet("alpine-x64",
"linux-arm",
"linux-arm64",
"linux-x64",
"osx-arm64",
"osx-x64",
"win-arm",
"win-arm64",
"win7-x64",
"win7-x86")]
[string]
$Runtime
)

$outputAssembly = $AssemblyPath.Replace(".dll", ".ni.dll")
$platformAssembliesPath = Split-Path $AssemblyPath -Parent
$crossgenFolder = Split-Path $CrossgenPath
$niAssemblyName = Split-Path $outputAssembly -Leaf
$platformAssembliesPath = Split-Path $AssemblyPath[0] -Parent

try {
Push-Location $crossgenFolder
$targetOS, $targetArch = $Runtime -split '-'

# Generate the ngen assembly
Write-Verbose "Generating assembly $niAssemblyName"
Start-NativeExecution {
& $CrossgenPath /ReadyToRun /MissingDependenciesOK /in $AssemblyPath /out $outputAssembly /Platform_Assemblies_Paths $platformAssembliesPath
} | Write-Verbose
} finally {
Pop-Location
# Special cases where OS / Arch does not conform with runtime names
switch ($Runtime) {
'alpine-x64' {
$targetOS = 'linux'
$targetArch = 'x64'
}
'win-arm' {
$targetOS = 'windows'
$targetArch = 'arm'
}
'win-arm64' {
$targetOS = 'windows'
$targetArch = 'arm64'
}
'win7-x64' {
$targetOS = 'windows'
$targetArch = 'x64'
}
'win7-x86' {
$targetOS = 'windows'
$targetArch = 'x86'
}
}
}

function New-CrossGenSymbol {
param (
[Parameter(Mandatory= $true)]
[ValidateNotNullOrEmpty()]
[String]
$AssemblyPath,
[Parameter(Mandatory= $true)]
[ValidateNotNullOrEmpty()]
[String]
$CrossgenPath
)
$generatePdb = $targetos -eq 'windows'

# The path to folder must end with directory separator
$dirSep = [System.IO.Path]::DirectorySeparatorChar
$platformAssembliesPath = if (-not $platformAssembliesPath.EndsWith($dirSep)) { $platformAssembliesPath + $dirSep }

$platformAssembliesPath = Split-Path $AssemblyPath -Parent
$crossgenFolder = Split-Path $CrossgenPath

try {
Push-Location $crossgenFolder

$symbolsPath = [System.IO.Path]::ChangeExtension($assemblyPath, ".pdb")

$createSymbolOptionName = $null
if($Environment.IsWindows)
{
$createSymbolOptionName = '-CreatePDB'
Start-NativeExecution {
$crossgen2Params = @(
"-r"
$platformAssembliesPath
"--out-near-input"
"--single-file-compilation"
Comment thread
adityapatwardhan marked this conversation as resolved.
Outdated
"-O"
"--targetos"
$targetOS
"--targetarch"
$targetArch
)

}
elseif ($Environment.IsLinux)
{
$createSymbolOptionName = '-CreatePerfMap'
if ($generatePdb) {
$crossgen2Params += "--pdb"
}

if($createSymbolOptionName)
{
Start-NativeExecution {
& $CrossgenPath -readytorun -platform_assemblies_paths $platformAssembliesPath $createSymbolOptionName $platformAssembliesPath $AssemblyPath
} | Write-Verbose
}
$crossgen2Params += $AssemblyPath

# Rename the corresponding ni.dll assembly to be the same as the IL assembly
$niSymbolsPath = [System.IO.Path]::ChangeExtension($symbolsPath, "ni.pdb")
Rename-Item $niSymbolsPath $symbolsPath -Force -ErrorAction Stop
} finally {
Pop-Location
& $CrossgenPath $crossgen2Params
}
}

Expand All @@ -2403,24 +2412,13 @@ function Start-CrossGen {
}

# Get the path to crossgen
$crossGenExe = if ($environment.IsWindows) { "crossgen.exe" } else { "crossgen" }
$generateSymbols = $false
$crossGenExe = if ($environment.IsWindows) { "crossgen2.exe" } else { "crossgen2" }

# The crossgen tool is only published for these particular runtimes
$crossGenRuntime = if ($environment.IsWindows) {
if ($Runtime -match "-x86") {
"win-x86"
} elseif ($Runtime -match "-x64") {
"win-x64"
$generateSymbols = $true
} elseif (!($env:PROCESSOR_ARCHITECTURE -match "arm")) {
throw "crossgen for 'win-arm' and 'win-arm64' must be run on that platform"
}
} elseif ($Runtime -eq "linux-arm") {
throw "crossgen is not available for 'linux-arm'"
} elseif ($Runtime -eq "linux-x64") {
$Runtime
# We should set $generateSymbols = $true, but the code needs to be adjusted for different extension on Linux
# for windows the tool architecture is the host machine architecture, so it is always x64.
# we can cross compile for x86, arm and arm64
"win-x64"
Copy link
Copy Markdown
Member

@TravisEz13 TravisEz13 Jul 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you comment why?

Copy link
Copy Markdown
Member Author

@adityapatwardhan adityapatwardhan Jul 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comment

} else {
$Runtime
}
Expand All @@ -2440,31 +2438,9 @@ function Start-CrossGen {
Select-Object -First 1 | `
ForEach-Object { $_.FullName }
if (-not $crossGenPath) {
throw "Unable to find latest version of crossgen.exe. 'Please run Start-PSBuild -Clean' first, and then try again."
}
Write-Verbose "Matched CrossGen.exe: $crossGenPath" -Verbose

# Crossgen.exe requires the following assemblies:
# mscorlib.dll
# System.Private.CoreLib.dll
# clrjit.dll on Windows or libclrjit.so/dylib on Linux/OS X
$crossGenRequiredAssemblies = @("mscorlib.dll", "System.Private.CoreLib.dll")

$crossGenRequiredAssemblies += if ($environment.IsWindows) {
"clrjit.dll"
} elseif ($environment.IsLinux) {
"libclrjit.so"
} elseif ($environment.IsMacOS) {
"libclrjit.dylib"
}

# Make sure that all dependencies required by crossgen are at the directory.
$crossGenFolder = Split-Path $crossGenPath
foreach ($assemblyName in $crossGenRequiredAssemblies) {
if (-not (Test-Path "$crossGenFolder\$assemblyName")) {
Copy-Item -Path "$PublishPath\$assemblyName" -Destination $crossGenFolder -Force -ErrorAction Stop
}
throw "Unable to find latest version of crossgen2.exe. 'Please run Start-PSBuild -Clean' first, and then try again."
}
Write-Verbose "Matched CrossGen2.exe: $crossGenPath" -Verbose

# Common assemblies used by Add-Type or assemblies with high JIT and no pdbs to crossgen
$commonAssembliesForAddType = @(
Expand Down Expand Up @@ -2495,11 +2471,13 @@ function Start-CrossGen {

$fullAssemblyList = $commonAssembliesForAddType

foreach ($assemblyName in $fullAssemblyList) {
$assemblyPath = Join-Path $PublishPath $assemblyName
New-CrossGenAssembly -CrossgenPath $crossGenPath -AssemblyPath $assemblyPath
$assemblyFullPaths = @()
$assemblyFullPaths += foreach ($assemblyName in $fullAssemblyList) {
Join-Path $PublishPath $assemblyName
}

New-CrossGenAssembly -CrossgenPath $crossGenPath -AssemblyPath $assemblyFullPaths -Runtime $Runtime

#
# With the latest dotnet.exe, the default load context is only able to load TPAs, and TPA
# only contains IL assembly names. In order to make the default load context able to load
Expand All @@ -2523,12 +2501,6 @@ function Start-CrossGen {
# Microsoft.CodeAnalysis.VisualBasic.dll, and Microsoft.CSharp.dll.
if ($commonAssembliesForAddType -notcontains $assemblyName) {
Remove-Item $symbolsPath -Force -ErrorAction Stop

if($generateSymbols)
{
Write-Verbose "Generating Symbols for $assemblyPath"
New-CrossGenSymbol -CrossgenPath $crossGenPath -AssemblyPath $assemblyPath
}
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion global.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"sdk": {
"version": "6.0.100-preview.4.21255.9"
"version": "6.0.100-preview.6.21355.2"
}
}
2 changes: 1 addition & 1 deletion src/Microsoft.PowerShell.Commands.Management/Microsoft.PowerShell.Commands.Management.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@

<ItemGroup>
<!-- the following package(s) are from https://github.com/dotnet/corefx -->
<PackageReference Include="System.ServiceProcess.ServiceController" Version="6.0.0-preview.4.21253.7" />
<PackageReference Include="System.ServiceProcess.ServiceController" Version="6.0.0-preview.6.21352.12" />
</ItemGroup>

</Project>
6 changes: 3 additions & 3 deletions src/Microsoft.PowerShell.Commands.Utility/Microsoft.PowerShell.Commands.Utility.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@
</ItemGroup>

<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.10.0" />
<PackageReference Include="System.Threading.AccessControl" Version="6.0.0-preview.4.21253.7" />
<PackageReference Include="System.Drawing.Common" Version="6.0.0-preview.4.21253.7" />
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.0.0-1.final" />
<PackageReference Include="System.Threading.AccessControl" Version="6.0.0-preview.6.21352.12" />
<PackageReference Include="System.Drawing.Common" Version="6.0.0-preview.6.21352.12" />
<PackageReference Include="NJsonSchema" Version="10.4.5" />
</ItemGroup>

Expand Down
2 changes: 1 addition & 1 deletion src/Microsoft.PowerShell.CoreCLR.Eventing/Microsoft.PowerShell.CoreCLR.Eventing.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

<ItemGroup>
<!-- the following package(s) are from https://github.com/dotnet/corefx -->
<PackageReference Include="System.Diagnostics.EventLog" Version="6.0.0-preview.4.21253.7" />
<PackageReference Include="System.Diagnostics.EventLog" Version="6.0.0-preview.6.21352.12" />
</ItemGroup>

</Project>
18 changes: 9 additions & 9 deletions src/Microsoft.PowerShell.SDK/Microsoft.PowerShell.SDK.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,16 @@
<!-- the following package(s) are from https://github.com/dotnet/fxdac -->
<PackageReference Include="System.Data.SqlClient" Version="4.8.2" />
<!-- the following package(s) are from https://github.com/dotnet/corefx -->
<PackageReference Include="System.IO.Packaging" Version="6.0.0-preview.4.21253.7" />
<PackageReference Include="System.Net.Http.WinHttpHandler" Version="6.0.0-preview.4.21253.7" />
<PackageReference Include="System.Text.Encodings.Web" Version="6.0.0-preview.4.21253.7" />
<PackageReference Include="System.IO.Packaging" Version="6.0.0-preview.6.21352.12" />
<PackageReference Include="System.Net.Http.WinHttpHandler" Version="6.0.0-preview.6.21352.12" />
<PackageReference Include="System.Text.Encodings.Web" Version="6.0.0-preview.6.21352.12" />
<!-- the following package(s) are from https://github.com/dotnet/wcf -->
<PackageReference Include="System.ServiceModel.Duplex" Version="4.8.1" />
<PackageReference Include="System.ServiceModel.Http" Version="4.8.1" />
<PackageReference Include="System.ServiceModel.NetTcp" Version="4.8.1" />
<PackageReference Include="System.ServiceModel.Primitives" Version="4.8.1" />
<PackageReference Include="System.ServiceModel.Security" Version="4.8.1" />
<PackageReference Include="System.Private.ServiceModel" Version="4.8.1" />
<PackageReference Include="System.ServiceModel.Duplex" Version="4.9.0-preview1.21220.1" />
<PackageReference Include="System.ServiceModel.Http" Version="4.9.0-preview1.21220.1" />
<PackageReference Include="System.ServiceModel.NetTcp" Version="4.9.0-preview1.21220.1" />
<PackageReference Include="System.ServiceModel.Primitives" Version="4.9.0-preview1.21220.1" />
<PackageReference Include="System.ServiceModel.Security" Version="4.9.0-preview1.21220.1" />
<PackageReference Include="System.Private.ServiceModel" Version="4.9.0-preview1.21220.1" />
<!-- the source could not be found for the following package(s) -->
<PackageReference Include="Microsoft.NETCore.Windows.ApiSets" Version="1.0.1" />
<PackageReference Include="Microsoft.Windows.Compatibility" Version="6.0.0-preview.4.21253.7" />
Expand Down
2 changes: 1 addition & 1 deletion src/Microsoft.WSMan.Management/Microsoft.WSMan.Management.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
<ProjectReference Include="..\System.Management.Automation\System.Management.Automation.csproj" />
<ProjectReference Include="..\Microsoft.WSMan.Runtime\Microsoft.WSMan.Runtime.csproj" />
<!-- the following package(s) are from https://github.com/dotnet/corefx -->
<PackageReference Include="System.ServiceProcess.ServiceController" Version="6.0.0-preview.4.21253.7" />
<PackageReference Include="System.ServiceProcess.ServiceController" Version="6.0.0-preview.6.21352.12" />
</ItemGroup>

<PropertyGroup>
Expand Down
Loading