Add smoke tests to GH Action (microsoft#9964)

kimadeline · brettcannon · web-flow · commit 0f65671035ff · 2020-02-07T15:31:40.000-08:00
* Update branch name (for testing) * TypeScript tests * python unit tests * ipython unit tests * single, multi, debugger tests * functional tests * Comment out VSIX build+upload when testing the action * Cache out/ directory * rename step id to more specific pythonFiles * single workspace tests * multi workspace tests * debugger tests * Add todo for when the PR is ready for review * Fix out/ cache key * Fix pythonFiles cache key * Retry single workspace tests * Use same cache key everywhere * Try out multi workspace tests * Try out debugger tests * Uncomment compile + upload step * run venv tests * Update branch name to trigger workflow * Adding comments + enable full test suite * Finalize versions of python we want to use * Try 3.7 instead of 3.8 * Run the workflow only on upstream master * Clean up comments * Revert to 3.8 * Update .github/workflows/ci.yml Co-Authored-By: Brett Cannon <54418+brettcannon@users.noreply.github.com> * Apply suggestions from code review Co-Authored-By: Brett Cannon <54418+brettcannon@users.noreply.github.com> * Rename test suite matrix key * Try bang operator * Make python cache keys more specific * Move repo condition up * Run a subset of tests to see if stuff still works * Update the branch to test the workflow * Try leaving no space between ! and the expression * Undo ! testing * Update pythonFiles cache key * Smoke tests * Don't install vsce globally * Where my env var value at * well this one didn't work * Use correct env var syntax * Remove curly brackets * Use xvfb for smoke tests * Take prep work outside of xvfb step * Uncomment things * Comment things out for testing * Merge master, fix things * forgot to comment condition * Uncomment testing checks * Newlines * Whitespaces microsoft#2 * Run tests on windows * Remember to package the Python files * Update build number earlier * Use existing VSIX artifact instead of repackaging * Skip repo condition for a test run * List directory content (debug) * Update ci.yml * Update ci.yml * Undo download artifact changes to compare * Re-add brackets * Force update build nb on bash shell * Do not call gulp:cleanExceptTests 😒 * Download artifact in place * Only run npx tsc -p ./ * Comment windows for now, re-run all prep (comparison) * Correct path for artifact download * Extract vsix from download location * Remove path when downloading * Cleanup now that the smoke tests run * Remove repo conditions (ready to be merged) * Newline * Don't need the python files in smoke tests * Ready for re-review * npm ci --prefer-offline Co-authored-by: Brett Cannon <54418+brettcannon@users.noreply.github.com>
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -161,6 +161,12 @@ jobs:
       - name: Clean directory
         run: npm run clean
 
+      # Use the GITHUB_RUN_ID environment variable to update the build number.
+      # GITHUB_RUN_ID is a unique number for each run within a repository.
+      # This number does not change if you re-run the workflow run.
+      - name: Update build number
+        run: npm run updateBuildNumber -- --buildNumber $GITHUB_RUN_ID
+
       - name: Package the VSIX
         run: npm run package
 
@@ -180,7 +186,7 @@ jobs:
       matrix:
         # We're not running CI on macOS for now because it's one less matrix entry to lower the number of runners used,
         # macOS runners are expensive, and we assume that Ubuntu is enough to cover the UNIX case.
-        os: [ubuntu-latest]
+        os: [ubuntu-latest, windows-latest]
         # Run the tests on the oldest and most recent versions of Python.
         python: [2.7, 3.8]
         test-suite: [ts-unit, python-unit, venv, single-workspace, multi-workspace, debugger, functional]
@@ -346,3 +352,49 @@ jobs:
       - name: Run functional tests
         run: npm run test:functional
         if: matrix.test-suite == 'functional'
+
+  smoke-tests:
+    name: Smoke tests
+    # The value of runs-on is the OS of the current job (specified in the strategy matrix below) instead of being hardcoded.
+    runs-on: ${{ matrix.os }}
+    needs: [build-vsix]
+    if: github.repository == 'microsoft/vscode-python'
+    strategy:
+      matrix:
+        # We're not running CI on macOS for now because it's one less matrix entry to lower the number of runners used,
+        # macOS runners are expensive, and we assume that Ubuntu is enough to cover the UNIX case.
+        os: [ubuntu-latest, windows-latest]
+        python: [3.8]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Retrieve cached npm files
+        uses: actions/cache@v1
+        with:
+          path: ~/.npm
+          key: ${{runner.os}}-${{env.CACHE_NPM_DEPS}}-${{hashFiles('package-lock.json')}}
+
+      - name: Install dependencies (npm ci)
+        run: npm ci --prefer-offline
+
+      - name: Download VSIX
+        uses: actions/download-artifact@v1
+        with:
+          name: ${{env.ARTIFACT_NAME_VSIX}}
+
+      # Extract the artifact from its download folder (./${{env.ARTIFACT_NAME_VSIX}}) to the repo root,
+      # then delete the download folder and compile the source code.
+      - name: Prepare for smoke tests
+        run: |
+           mv ${{env.ARTIFACT_NAME_VSIX}}/* .
+           rm -r ${{env.ARTIFACT_NAME_VSIX}}
+           npx tsc -p ./
+        shell: bash
+
+      - name: Run smoke tests
+        env:
+          DISPLAY: 10
+        uses: GabrielBB/xvfb-action@v1.0
+        with:
+          run: node ./out/test/smokeTest.js
