Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .vscode/extensions.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
"ms-vscode.PowerShell",
"twxs.cmake",
"DavidAnson.vscode-markdownlint",
"ms-azure-devops.azure-pipelines"
"ms-azure-devops.azure-pipelines",
"vitaliymaz.vscode-svg-previewer"
]
}
2 changes: 1 addition & 1 deletion .vsts-ci/linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ jobs:
displayName: Set TEMP to AGENT_TEMPDIRECTORY
condition: succeededOrFailed()

- template: ../tools/releaseBuild/azureDevOps/templates/insert-nuget-config.yml
- template: ../tools/releaseBuild/azureDevOps/templates/insert-nuget-config-azfeed.yml

- powershell: |
tools/travis.ps1 -Stage Bootstrap
Expand Down
2 changes: 1 addition & 1 deletion .vsts-ci/mac.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ jobs:
displayName: Set Build Name for Non-PR
condition: ne(variables['Build.Reason'], 'PullRequest')

- template: ../tools/releaseBuild/azureDevOps/templates/insert-nuget-config.yml
- template: ../tools/releaseBuild/azureDevOps/templates/insert-nuget-config-azfeed.yml

- powershell: |
tools/travis.ps1 -Stage Bootstrap
Expand Down
2 changes: 1 addition & 1 deletion .vsts-ci/windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ steps:
displayName: Set Build Name for Non-PR
condition: ne(variables['Build.Reason'], 'PullRequest')

- template: ../tools/releaseBuild/azureDevOps/templates/insert-nuget-config.yml
- template: ../tools/releaseBuild/azureDevOps/templates/insert-nuget-config-azfeed.yml

- powershell: |
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
Expand Down
58 changes: 40 additions & 18 deletions tools/releaseBuild/README.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,55 @@
# VSTS Release Builds
# Azure Dev Ops Release Builds

## Requirements

Docker must be installed to run any of the release builds.

## Running Windows Release Builds
## Running Windows Release Builds Locally

From PowerShell on Windows, run `.\vstsbuild.ps1 -ReleaseTag <tag> -Name <buildName>`.

Windows Build Names:
For the package builds, run `.\vstsbuild.ps1 -ReleaseTag <tag> -Name <buildName> -BuildPath <path to extracted zip from build step> -SignedFilesPath <path to extracted 'symbol' zip from build step>`

* `win7-x64`
* Builds the Universal Windows x64 Package
* `win7-x86`
* Builds the Universal Windows x86 Package
* `win7-x64-symbols`
* Builds the Windows x64 Zip with symbols
* `win7-x86-symbols`
* Builds the Windows x86 Zip with symbols
Windows Build Names:

## Running Linux Release Builds
* `win7-x64-symbols`
* Builds the Windows x64 Zip with symbols
* `win7-x86-symbols`
* Builds the Windows x86 Zip with symbols
* `win7-arm-symbols`
* Builds the Windows ARM Zip with symbols
* `win7-arm64-symbols`
* Builds the Windows ARM64 Zip with symbols
* `win7-fxdependent-symbols`
* Builds the Windows FxDependent Zip with symbols
* `win7-x64-package`
* Builds the Windows x64 packages
* `win7-x86-package`
* Builds the Windows x86 packages
* `win7-arm-package`
* Builds the Windows ARM packages
* `win7-arm64-package`
* Builds the Windows ARM64 packages
* `win7-fxdependent-package`
* Builds the Windows FxDependent packages

## Running Linux Release Builds Locally

From PowerShell on Linux or macOS, run `.\vstsbuild.ps1 -ReleaseTag <tag> -Name <buildName>`.

Linux Build Names:

* `ubuntu.14.04`
* Builds the Ubuntu 14.04 Package and AppImage Package
* `ubuntu.16.04`
* Builds the Ubuntu 16.04 Package
* `centos.7`
* Builds the CentOS 7 Package
* `deb`
* Builds the Debian Packages, ARM32 and ARM64.
* `alpine`
* Builds the Alpine Package
* `rpm`
* Builds the RedHat variant Package

## Azure Dev Ops Build

The release build is fairly complicated. The definition is at `./azureDevOps/releaseBuild.yml`.

Here is a diagram of the build:

[![Release Build diagram](https://raw.githubusercontent.com/PowerShell/PowerShell/master/tools/releaseBuild/azureDevOps/diagram.svg?sanitize=true)](https://raw.githubusercontent.com/PowerShell/PowerShell/master/tools/releaseBuild/azureDevOps/diagram.svg?sanitize=true)
Binary file removed tools/releaseBuild/azureDevOps/diagram.png
Binary file not shown.
110 changes: 73 additions & 37 deletions tools/releaseBuild/azureDevOps/diagram.puml
Original file line number Diff line number Diff line change
@@ -1,31 +1,50 @@
@startuml

' Define the build tasks as business processes
archimate #Business "Build DEB" as BuildDEB <<business-process>>
archimate #Business "Build RPM" as BuildRPM <<business-process>>
archimate #Business "Build Alpine" as BuildAlpine <<business-process>>
archimate #Business "Build Linux-FxDependent" as BuildLinuxFx <<business-process>>
archimate #Business "Upload DEB" as UploadDEB <<business-process>>
archimate #Business "Upload RPM" as UploadRPM <<business-process>>
archimate #Business "Upload Alpine" as UploadAlpine <<business-process>>
archimate #Business "Upload Linux-FxDependent" as UploadLinuxFx <<business-process>>

archimate #Business "Build macOS" as BuildMac <<business-process>>
archimate #Business "macOS Package Signing and Upload" as SignMac <<business-process>>

archimate #Business "Build Windows - x64" as BuildWinX64 <<business-process>>
archimate #Business "Build Windows - x86" as BuildWinX86 <<business-process>>
archimate #Business "Build Windows - arm32" as BuildWinArm32 <<business-process>>
archimate #Business "Build Windows - arm64" as BuildWinArm64 <<business-process>>
archimate #Business "Build Windows - FxDependent" as BuildWinFx <<business-process>>
archimate #Business "Windows Package Signing and Upload" as SignWin <<business-process>>

archimate #Business "ComponentRegistration" as BuildCG <<business-process>>

archimate #Business "Compliance" as Compliance <<business-process>>
folder "Linux Builds" {
' Define the build tasks as business processes
agent "DEB" as BuildDEB
agent "RPM" as BuildRPM
agent "Alpine" as BuildAlpine
agent "Linux-FxDependent" as BuildLinuxFx

}

agent "macOS Build" as BuildMac

folder "Windows Builds" as WinBuilds {
agent "x64" as BuildWinX64
agent "x86" as BuildWinX86
agent "arm32" as BuildWinArm32
agent "arm64" as BuildWinArm64
agent "FxDependent" as BuildWinFx
}

agent "ComponentRegistration" as BuildCG

folder "Linux Package Scanning and Upload" {
agent "DEB" as UploadDEB
agent "RPM" as UploadRPM
agent "Alpine" as UploadAlpine
agent "Linux-FxDependent" as UploadLinuxFx
}

folder "Package Signing and Upload" {
agent "macOS" as SignMac

agent "Windows" as SignWin
}


agent "Compliance" as Compliance

folder "Create & Upload FxDependent DotNet SDK" {
agent "Linux" as BuildLinuxFxSdk
agent "Windows" as BuildWinFxSdk
}

' Define finishing the build as a goal filled
archimate #Goal "Finish" as Finish <<goal-filled>>
control "Finish" as Finish
control "Start" as Start

' map the various Upload task dependencies
BuildDEB -down-> UploadDEB
Expand All @@ -37,25 +56,42 @@ BuildAlpine -down-> UploadAlpine
BuildMac -down-> SignMac

' map all of the SignWin task dependencies
BuildWinX64 -down-> SignWin
BuildWinX86 -down-> SignWin
BuildWinArm32 -down-> SignWin
BuildWinArm64 -down-> SignWin
BuildWinFx -down-> SignWin
WinBuilds -down-> SignWin
'BuildWinX64 -down-> SignWin
'BuildWinX86 -down-> SignWin
'BuildWinArm32 -down-> SignWin
'BuildWinArm64 -down-> SignWin
'BuildWinFx -down-> SignWin

' map all of the Compliance task dependencies
BuildWinX86 -down-> Compliance
BuildWinX64 -down-> Compliance
BuildWinFx -down-> Compliance

BuildLinuxFx -down-> BuildLinuxFxSdk
SignWin -down-> BuildWinFxSdk

' map all leafs to finish
Compliance -down-> Finish
UploadAlpine -down-> Finish
UploadDEB -down-> Finish
UploadRPM -down-> Finish
UploadLinuxFx -down-> Finish
SignWin -down-> Finish
SignMac -down-> Finish
BuildCG -down-> Finish
Compliance ~~ Finish
UploadAlpine ~~ Finish
UploadDEB ~~ Finish
UploadRPM ~~ Finish
UploadLinuxFx ~~ Finish
SignMac ~~ Finish
BuildCG ~~ Finish
BuildLinuxFxSdk ~~ Finish
BuildWinFxSdk ~~ Finish

Start ~~ BuildDEB
Start ~~ BuildRPM
Start ~~ BuildAlpine
Start ~~ BuildLinuxFx
Start ~~ BuildMac
Start ~~ BuildWinX64
Start ~~ BuildWinX86
Start ~~ BuildWinFx
Start ~~ BuildWinArm32
Start ~~ BuildWinArm64
Start ~~ BuildCG

@enduml
98 changes: 98 additions & 0 deletions tools/releaseBuild/azureDevOps/diagram.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions tools/releaseBuild/azureDevOps/templates/compliance.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ jobs:
${{ parameters.parentJobs }}
pool:
name: Package ES CodeHub Lab E
# APIScan can take a long time
timeoutInMinutes: 90

steps:
- template: SetVersionVariables.yml
Expand Down
14 changes: 11 additions & 3 deletions tools/releaseBuild/azureDevOps/templates/linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
}
displayName: 'Skip Alpine or fxdependent for PS v6.0.*'

- template: insert-nuget-config.yml
- template: insert-nuget-config-azfeed.yml


- powershell: |
Expand Down Expand Up @@ -81,20 +81,28 @@ jobs:
displayName: 'Run Defender Scan'

- task: AzureFileCopy@1
displayName: 'Upload to azure deb and tar.gz'
displayName: 'Upload to Azure - DEB and tar.gz'
inputs:
SourcePath: '$(System.ArtifactsDirectory)\finished\release'
azureSubscription: '$(AzureFileCopySubscription)'
Destination: AzureBlob
storage: '$(StorageAccount)'
ContainerName: '$(AzureVersion)'

- template: upload-final-results.yml
parameters:
artifactPath: $(System.ArtifactsDirectory)\finished\release

- task: AzureFileCopy@1
displayName: 'Upload to azure rpm'
displayName: 'Upload to Azure - RPM'
inputs:
SourcePath: '$(System.ArtifactsDirectory)\rpm\release'
azureSubscription: '$(AzureFileCopySubscription)'
Destination: AzureBlob
storage: '$(StorageAccount)'
ContainerName: '$(AzureVersion)-unsigned'
condition: and(eq(variables['buildName'], 'RPM'),succeeded())

- template: upload-final-results.yml
parameters:
artifactPath: $(System.ArtifactsDirectory)\rpm\release
24 changes: 10 additions & 14 deletions tools/releaseBuild/azureDevOps/templates/mac-package-signing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,19 +52,10 @@ jobs:
signConfigXml: '$(System.ArtifactsDirectory)\package.xml'
outPathRoot: '$(Build.StagingDirectory)\signedMacOSPackages'

- powershell: |
dir "$(System.ArtifactsDirectory)\*.tar.gz" -Recurse | foreach-object {
$fullname = $_.fullname
Write-Host "##vso[artifact.upload containerfolder=signed;artifactname=signed]$fullname"
}
displayName: 'Upload tar.gz'

- powershell: |
dir "$(Build.StagingDirectory)\signedMacOSPackages\powershell-*.zip" -Recurse | foreach-object {
$fullname = $_.fullname
Write-Host "##vso[artifact.upload containerfolder=signed;artifactname=signed]$fullname"
}
displayName: 'Upload macOS signed package'
- template: upload-final-results.yml
parameters:
artifactPath: $(System.ArtifactsDirectory)
artifactFilter: "*.tar.gz"

- powershell: |
$destination = "$(System.ArtifactsDirectory)\azureMacOs"
Expand All @@ -73,7 +64,12 @@ jobs:
Expand-Archive -Path $zipPath -DestinationPath $destination
$targzPath = dir "$(System.ArtifactsDirectory)\*.tar.gz" -Recurse | select-object -expandproperty fullname
Copy-Item -Path $targzPath -Destination $destination
displayName: 'Extract and copy macOS artifacts for upload to azure'
displayName: 'Extract and copy macOS artifacts for upload'

- template: upload-final-results.yml
parameters:
artifactPath: $(System.ArtifactsDirectory)\azureMacOs
artifactFilter: "*.pkg"

- task: AzureFileCopy@1
displayName: 'AzureBlob File Copy'
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
parameters:
artifactPath:
artifactFilter: '*'
condition: succeeded()
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want succeeded() or failOrSucceeded()? Maybe we should be able to upload whatever artifacts were built.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should think about retry scenarios. We don't want to leave this folder potentially polluted.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a parameter for container name to make this task more general and default it to finalResults

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change can be made later.


steps:
- powershell: |
Get-ChildItem -Path '${{ parameters.artifactPath }}' -File -filter '${{ parameters.artifactFilter }}' -ErrorAction SilentlyContinue |
Select-Object -ExpandProperty FullName |
ForEach-Object {
Write-Host "##vso[artifact.upload containerfolder=finalResults;artifactname=finalResults]$_"
}
displayName: Upload Final Artifacts ${{ parameters.artifactFilter }} from ${{ parameters.artifactPath }}
condition: ${{ parameters.condition }}
24 changes: 9 additions & 15 deletions tools/releaseBuild/azureDevOps/templates/upload.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,11 @@ parameters:
msi: yes

steps:
- powershell: |
$packagePath = "$(Build.StagingDirectory)\signedPackages\PowerShell-${{ parameters.version }}-win-${{ parameters.architecture }}.msi"
if(Test-Path -Path $packagePath)
{
Write-Host "##vso[artifact.upload containerfolder=signedResults;artifactname=signedResults]$packagePath"
}
displayName: '[create script] upload signed msi - ${{ parameters.architecture }}'
condition: and(succeeded(), eq('${{ parameters.msi }}', 'yes'))
- template: upload-final-results.yml
parameters:
artifactPath: $(Build.StagingDirectory)\signedPackages
artifactFilter: PowerShell-${{ parameters.version }}-win-${{ parameters.architecture }}.msi
condition: and(succeeded(), eq('${{ parameters.msi }}', 'yes'))

- task: AzureFileCopy@1
displayName: 'upload signed msi to Azure - ${{ parameters.architecture }}'
Expand All @@ -23,13 +20,10 @@ steps:
ContainerName: '$(AzureVersion)'
condition: and(succeeded(), eq('${{ parameters.msi }}', 'yes'))

- powershell: |
$packagePath = "$(System.ArtifactsDirectory)\signed\PowerShell-${{ parameters.version }}-win-${{ parameters.architecture }}.zip"
if(Test-Path -Path $packagePath)
{
Write-Host "##vso[artifact.upload containerfolder=signedResults;artifactname=signedResults]$packagePath"
}
displayName: '[create script] upload signed zip - ${{ parameters.architecture }}'
- template: upload-final-results.yml
parameters:
artifactPath: $(System.ArtifactsDirectory)\signed
artifactFilter: PowerShell-${{ parameters.version }}-win-${{ parameters.architecture }}.zip

- task: AzureFileCopy@1
displayName: 'upload signed zip to Azure - ${{ parameters.architecture }}'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
Write-Host -Object "##$vstsCommandString"
displayName: 'Skip FxDependent for PS v6.0.*'

- template: insert-nuget-config.yml
- template: insert-nuget-config-azfeed.yml

- powershell: |
docker container prune --force
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
# Diagnostics is not critical it passes every time it runs
continueOnError: true

- template: insert-nuget-config.yml
- template: insert-nuget-config-azfeed.yml

- powershell: |
./tools/releaseBuild/vstsbuild.ps1 -ReleaseTag $(ReleaseTagVar) -Name win-x64-component-registration
Expand Down