Skip to content

Update PowerShell to build against dotnet 5.0-RC.1#13643

Merged
rjmholt merged 19 commits into
PowerShell:masterfrom
daxian-dbw:build
Sep 23, 2020
Merged

Update PowerShell to build against dotnet 5.0-RC.1#13643
rjmholt merged 19 commits into
PowerShell:masterfrom
daxian-dbw:build

Conversation

@daxian-dbw
Copy link
Copy Markdown
Member

@daxian-dbw daxian-dbw commented Sep 16, 2020

PR Summary

Update code and project files to build PowerShell against dotnet 5.0-rc.1.

Major changes to code:

Known issues in .NET 5 RC.1

The .NET 5 RC.1 regressions and by-design breaking changes we ran into so far are:

  1. Performance regression on Linux and macOS: [Unix] Potential performance regression in Directory.EnumerateFiles dotnet/runtime#41739. The fix [release/5.0-rc2] Revert #40641 dotnet/runtime#41820 is made in .NET 5 RC.2
  2. System.Console.TreatControlCAsInput is backwards in .NET 5 RC.1 on Linux and macOS. Issue: System.Console.TreatControlCAsInput is backwards in .NET 5 RC.1 dotnet/runtime#42423. The fix Console.Unix: fix inverted TreatControlCAsInput dotnet/runtime#42432 was made in .NET 5 RC.2
  3. System.Console.ReadKey() returning Ctrl+J for ENTER on macOS. Issue: System.Console.ReadKey() returning Ctrl+J for ENTER on macOS dotnet/runtime#42418. The fix [release/5.0-rc2] Revert "Make Console.ReadKey() distinguish between CR and LF inputs" dotnet/runtime#42477 was made in .NET 5 RC.2
  4. A by-design breaking change made in .NET 5 rc.1: Tls1.0 and Tls1.1 were retired from the default on Linux machines where OpenSSL 1.1 and above is used. The .NET PR is retire TLS1.0/1.1 ciphers from default list dotnet/runtime#40746

PR Checklist

@iSazonov
Copy link
Copy Markdown
Collaborator

Should we track CA1416 in new issue and fix this later?

@daxian-dbw
Copy link
Copy Markdown
Member Author

@iSazonov I think there is nothing that needs to be fixed. For all the types and APIs that are supported only on Windows, we compile them only for Windows build. Do you spot anything that is not the case?

@iSazonov
Copy link
Copy Markdown
Collaborator

iSazonov commented Sep 16, 2020

Do you spot anything that is not the case?

I remember we had issues with referencing methods from wrong platform. So we could review the analyzer warnings and maybe use recommended workarounds.
Also not all platform-depended API we have under conditional compiling.

@daxian-dbw
Copy link
Copy Markdown
Member Author

daxian-dbw commented Sep 16, 2020

The new attribute doesn't offer a suggestion, it's like error CA1416: 'RegistryKey.GetValue(string?)' is supported on 'windows'.
I randomly reviewed some files failed with the warning and it looks to me they are either guarded in a if/def or protected by a runtime check on the OS platform.

But there are lots of failures due to the warning and I surely didn't review them all. So please feel free to open an issue to track , and maybe also review them if you got time. Thanks!

@iSazonov
Copy link
Copy Markdown
Collaborator

@daxian-dbw Tracking issue #13647.

@iSazonov
Copy link
Copy Markdown
Collaborator

CI Linux fail with "OpenSslCryptographicException: error:141A90B5:SSL routines:ssl_cipher_list_to_bytes:no ciphers available".

Do they disable Tls 1.0 and 1.1 in CI Linux image in OpenSSL?

@daxian-dbw
Copy link
Copy Markdown
Member Author

daxian-dbw commented Sep 17, 2020

@iSazonov We are investigating but got nowhere so far :( How to check if the Tls 1.0 and 1.1 is disabled on Ubuntu?

FYI, the version of openssl on the CI agent is: OpenSSL 1.1.0h 27 Mar 2018 (Library: OpenSSL 1.1.1g 21 Apr 2020)

@TylerLeonhardt
Copy link
Copy Markdown
Member

TylerLeonhardt commented Sep 17, 2020

Dropping this here so it's not missed. Ctrl+C doesn't work on macOS

this ^ is because of dotnet/runtime#42423

@benaadams
Copy link
Copy Markdown

benaadams commented Sep 18, 2020

Use single file publish? (also outputs a copy native libs on Windows, should be a single file on Linux though)

<PublishSingleFile>true</PublishSingleFile>

Perhaps assembly trimming? Though that might not work for powershell (as you could invoke apis from script?)

<PublishTrimmed>true</PublishTrimmed>
<TrimMode>Link</TrimMode>

On Windows can include the R2R with the single file to make startup faster (in exchange for size) https://devblogs.microsoft.com/dotnet/customizing-trimming-in-net-core-5/

<PublishReadyToRun>True</PublishReadyToRun>
.NET Blog
This second post on app trimming goes into more detail about to annotate code to control the trimming process.

@iSazonov
Copy link
Copy Markdown
Collaborator

iSazonov commented Sep 18, 2020

How to check if the Tls 1.0 and 1.1 is disabled on Ubuntu?

I see 3 cases:

  • OpenSSL could be compiled without the support. I don't think it is our case.
  • The support disabled in OpenSSL config file (min_protocol and -no_tls1, -no_tls1_1 options in the config). We can run the tests locally (WSL2?) and see results. If no problems exist locally it is question for the CI image support team.
  • It is a bug in .Net 5.0 for Linix - on client side (PowerShell) or/and on server side (our test WebListener). We can check this locally and separately.

@iSazonov
Copy link
Copy Markdown
Collaborator

@benaadams

Use single file publish?

We don't use the option today. (I tried locally and catch some issues). And we don't have PowerShell Committee conclusion whether we want to support the single file publish.

Perhaps assembly trimming?

We don't use this. And you're right this is not acceptable in general. But it could be an option for specific minimal PowerShell distributions.

On Windows can include the R2R with the single file to make startup faster

Yes, we use the feature. Welcome to investigate and contribute more in the scenario if you want.

@daxian-dbw
Copy link
Copy Markdown
Member Author

@iSazonov We learnt from .NET team that this is a by-design breaking change made in .NET 5 rc.1, which retired Tls1.0 and Tls1.1 from the default on Linux machines where OpenSSL 1.1 and above is used. The .NET PR is dotnet/runtime#40746

@iSazonov
Copy link
Copy Markdown
Collaborator

@daxian-dbw Thanks! Do you team to collect such by-design changes? It would be nice to see its in Release notes for 7.1 GA (I hope this reduce "false" feedbacks from PowerShell users and helps adoption to the new PowerShell version.).

@daxian-dbw
Copy link
Copy Markdown
Member Author

daxian-dbw commented Sep 19, 2020

@iSazonov The breaking changes will all be documented in https://docs.microsoft.com/en-us/dotnet/core/compatibility/3.1-5.0, though the TSL one is not yet documented.

Lists the breaking changes from version 3.1 to version 5.0 of .NET, ASP.NET Core, and EF Core.

@rjmholt
Copy link
Copy Markdown
Collaborator

rjmholt commented Sep 21, 2020

After grappling with this issue for some time, I'm going to mark the tests that are currently failing as pending. We now expect the particular command to fail on Linux due to the configuration of both .NET 5 and the test environment, and I think it should be fixed as a patch to RC2. As such I'll open an issue to track fixing the tests and possibly PowerShell's own error messages (although after spending some time looking into it, I'm not sure how we do the latter reliably).

@SteveL-MSFT
Copy link
Copy Markdown
Member

@iSazonov, @benaadams regarding single file, we don't want to take such a big change for an RC (which means not in 7.1). Certainly something we would look at for 7.2. File trimming would not work for PowerShell as it exists today as PowerShell is a platform for scripts that expect as many APIs as possible. As Ilya noted, it may be useful when we make more progress on minimal PowerShell where scripts that need .NET APIs would install those assemblies with PowerShellGet 3.x.

@daxian-dbw daxian-dbw marked this pull request as ready for review September 23, 2020 17:40
@rjmholt rjmholt merged commit 10237bd into PowerShell:master Sep 23, 2020
@rjmholt rjmholt added the CL-BuildPackaging Indicates that a PR should be marked as a build or packaging change in the Change Log label Sep 23, 2020
@daxian-dbw daxian-dbw deleted the build branch September 23, 2020 21:07
@iSazonov
Copy link
Copy Markdown
Collaborator

What is a milestone should be for the PR and follow PRs?

@fMichaleczek
Copy link
Copy Markdown
Contributor

fMichaleczek commented Sep 24, 2020

File trimming would not work for PowerShell as it exists today as PowerShell is a platform for scripts that expect as many APIs as possible

@SteveL-MSFT I am not agree when it comes to old API. Why blacklist things when it could be remove at source ?

https://github.com/PowerShell/underhanded-powershell

I don't want this

# Windows/v7.1.0-preview.7
[Microsoft.VisualBasic.Interaction]::InputBox("Old stuff ?", "VBonCore", "Yes")

Trimming should be study from a security POV and the toolchain update to release at least a minimal edition. A PowerShell.SecureSdk project without distribution binaries is enough.

A security contest is welcome ! . A rank by binaries file size with minimal binaries command compatibilty.

GitHub
Underhanded PowerShell Contest Repository. Contribute to PowerShell/underhanded-powershell development by creating an account on GitHub.

@rjmholt
Copy link
Copy Markdown
Collaborator

rjmholt commented Sep 24, 2020

@SteveL-MSFT I am not agree when it comes to old API. Why blacklist things when it could be remove at source ?

Yeah we've discussed that and it's something we're gradually investigating as a sort of "minimal PowerShell".

But what's old to you is required by others -- take anything out and we'll break someone, or at the very least make their migration path significantly more difficult.

But also, my understanding is that trimming is an automatic process based on static code discovery, and PowerShell uses reflection pretty much everywhere. To make trimming work for us, we'd need to build a very large list of APIs we need to preserve from the trimming process.

As I say, it's something we're investigating gradually, since there are a number of large hurdles that we'd need to overcome to make it work.

@danmoseley
Copy link
Copy Markdown

@daxian-dbw thank you for helping find those last issues in RC1 so we could fix in RC2 and avoid shipping them. The more often Powershell can pick up .NET Core, the better for us!

@ghost
Copy link
Copy Markdown

ghost commented Sep 29, 2020

🎉v7.1.0-rc.1 has been released which incorporates this pull request.:tada:

Handy links:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CL-BuildPackaging Indicates that a PR should be marked as a build or packaging change in the Change Log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants