test_plan.md

Test plan

Environment

OS: XXX (Windows, macOS, latest Ubuntu LTS)
- Shell: XXX (Command Prompt, PowerShell, bash, fish)
Python
- Distribution: XXX (CPython, miniconda)
- Version: XXX (2.7, latest 3.x)
VS Code: XXX (Insiders)

Tests

ALWAYS:

Check the Output window under Python for logged errors
Have Developer Tools open to detect any errors
Consider running the tests in a multi-folder workspace
Focus on in-development features (i.e. experimental debugger and language server)

Scenarios

Environment

Interpreters

Terminal

Sample file:

import requests
request = requests.get("https://drive.google.com/uc?export=download&id=1_9On2-nsBQIw3JiY43sWbrF8EjrqrR4U")
with open("survey2017.zip", "wb") as file:
    file.write(request.content)
import zipfile
with zipfile.ZipFile('survey2017.zip') as zip:
    zip.extractall('survey2017')
import shutil, os
shutil.move('survey2017/survey_results_public.csv','survey2017.csv')
shutil.rmtree('survey2017')
os.remove('survey2017.zip')

Shift+Enter to send selected code in sample file to terminal works

Virtual environments

ALWAYS:

Environment files

Sample files:

# example.py
import os
print('Hello,', os.environ.get('WHO'), '!')

# .env
WHO=world
PYTHONPATH=some/path/somewhere
SPAM='hello ${WHO}'

ALWAYS:

Make sure to use Reload Window between tests to reset your environment
Note that environment files only apply under the debugger and Jedi
Environment variables in a .env file are exposed when running under the debugger
"python.envFile" allows for specifying an environment file manually (e.g. Jedi picks up PYTHONPATH changes)
envFile in a launch.json configuration works
simple variable substitution works

Debugging

pythonPath setting in your launch.json overrides your python.pythonPath default setting

Linting

ALWAYS:

Check under the Problems tab to see e.g. if a linter is raising errors

Language server

LS is downloaded using HTTP (no SSL) when the "http.proxyStrictSSL" setting is false
An item with a cloud icon appears in the status bar indicating progress while downloading the language server
Installing requests in virtual environment is detected
- Import of requests without package installed is flagged as unresolved
- Create a virtual environment
- Install requests into the virtual environment

Pylint/default linting

[Prompting to install Pylint is covered under Environments above]

For testing the disablement of the default linting rules for Pylint:

# pylintrc
[MESSAGES CONTROL]
enable=bad-names

# example.py
foo = 42  # Marked as a blacklisted name.

Installation via the prompt installs Pylint as appropriate
- Uses --user for system-install of Python
- Installs into a virtual environment environment directly
Pylint works
"python.linting.pylintUseMinimalCheckers": false turns off the default rules w/ no pylintrc file present
The existence of a pylintrc file turns off the default rules

Other linters

Note:

Editing

IntelliSense

Please also test for general accuracy on the most "interesting" code you can find.

"python.autoComplete.extraPaths" works
"python.autocomplete.addBrackets": true causes auto-completion of functions to append ()
Auto-completions works

Formatting

Sample file:

# There should be _some_ change after running `Format Document`.
import os,sys;
def foo():pass

Refactoring

Extract Variable works
- You are prompted to install rope if it is not already available
Extract method works
- You are prompted to install rope if it is not already available
Sort Imports works

Debugging

Unit testing

`unittest`

import unittest

MODULE_SETUP = False


def setUpModule():
    global MODULE_SETUP
    MODULE_SETUP = True


class PassingSetupTests(unittest.TestCase):
    CLASS_SETUP = False
    METHOD_SETUP = False

    @classmethod
    def setUpClass(cls):
        cls.CLASS_SETUP = True

    def setUp(self):
        self.METHOD_SETUP = True

    def test_setup(self):
        self.assertTrue(MODULE_SETUP)
        self.assertTrue(self.CLASS_SETUP)
        self.assertTrue(self.METHOD_SETUP)


class PassingTests(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42, 42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")


class FailingTests(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42, -13)

    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

Run All Unit Tests triggers the prompt to configure the test runner
Tests are discovered (as shown by code lenses on each test)
- Code lens for a class runs all tests for that class
- Code lens for a method runs just that test
  - Run Test works
  - Debug Test works
  - Module/suite setup methods are also run (run the test_setup method to verify)
while debugging tests, an uncaught exception in a test does not cause debugpy to raise SystemExit exception.

`pytest`

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

Run All Unit Tests triggers the prompt to configure the test runner
- pytest gets installed
Tests are discovered (as shown by code lenses on each test)
- Run Test works
- Debug Test works
A Diagnostic is shown in the problems pane for each failed/skipped test
- The Diagnostics are organized according to the file the test was executed from (not necessarily the file it was defined in)
- The appropriate DiagnosticRelatedInformation is shown for each Diagnostic
- The DiagnosticRelatedInformation reflects the traceback for the test

`nose`

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

Run All Unit Tests triggers the prompt to configure the test runner
- Nose gets installed
Tests are discovered (as shown by code lenses on each test)
- Run Test works
- Debug Test works

General

Data Science

P0 Test Scenarios

P1 Test Scenarios

Connect to a remote server
1. Open up a valid python command prompt that can run jupyter notebook (a default Anaconda prompt works well)
2. Run jupyter notebook to start up a local Jupyter server
3. In the command window that launched Jupyter look for the server / token name like so: http://localhost:8888/?token=bf9eae43641cd75015df9104f814b8763ef0e23ffc73720d
4. Run the command Python: Select Jupyter server URI then Type in the URI to connect to a running jupyter server
5. Input the server / token name here
6. Now run the cells in the manualTestFile.py
7. Verify that you see the server name in the initial connection message
8. Verify the outputs of the cells
Interactive Window commands
- Verify per-cell commands
  1. Expand and collapse the input area of a cell
  2. Use the X button to remove a cell
  3. Use the Goto Code button to jump to the part of the .py file that submitted the code
- Verify top menu commands
  1. Use X to delete all cells
  2. Undo the delete action with Undo
  3. Redo the delete action with Redo
  4. In manualTestFile.py modify the trange command in the progress bar from 100 to 2000. Run the Cell. As the cell is running hit the Interrupt iPython Kernel button
  5. The progress bar should be interrupted and you should see a KeyboardInterrupt error message in the output
  6. Test the Restart iPython kernel command. Kernel should be restarted and you should see a status output message for the kernel restart
  7. Use the expand all input and collapse all input commands to collapse all cell inputs
Verify theming works
1. Start Python Interactive window
2. Add a cell with some comments
3. Switch VS Code theme to something else
4. Check that the cell you just added updates the comment color
5. Switch back and forth between a 'light' and a 'dark' theme
6. Check that the cell switches colors
7. Check that the buttons on the top change to their appropriate 'light' or 'dark' versions
8. Enable the 'ignoreVscodeTheme' setting
9. Close the Python Interactive window and reopen it. The theme in just the 'Python Interactive' window should be light
10. Switch to a dark theme. Make sure the interactive window remains in the light theme.
Verify code lenses
1. Check that Run Cell Run Above and Run Below all do the correct thing
Verify context menu navigation commands
1. Check the Run Current Cell and Run Current Cell And Advance context menu commands
2. If run on the last cell of the file Run Current Cell And Advance should create a new empty cell and advance to it
Verify command palette commands
1. Close the Interactive Window then pick Python: Show Interactive Window
2. Restart the kernel and pick Python: Run Current File In Python Interactive Window it should run the whole file again
Verify shift-enter
1. Move to the top cell in the .py file
2. Shift-enter should run each cell and advance to the next
3. Shift-enter on the final cell should create a new cell and move to it
Verify file without cells
1. Open the manualTestFileNoCells.py file
2. Select a chunk of code, shift-enter should send it to the terminal
3. Open VSCode settings, change Send Selection To Interactive Window to true
4. Select a chunk of code, shift-enter should send that selection to the Interactive Windows
5. Move your cursor to a line, but don't select anything. Shift-enter should send that line to the Interactive Window
Multiple installs
1. Close and re-open VSCode to make sure that all jupyter servers are closed
2. Also make sure you are set to locally launch Jupyter and not to connect to an existing URI
3. In addition to your main testing environment install a new python or miniconda install (conda won't work as it has Jupyter by default)
4. In VS code change the python interpreter to the new install
5. Try Run Cell
6. You should get a message that Jupyter was not found and that it is defaulting back to launch on the python instance that has Jupyter
LiveShare Support
1. Install the LiveShare VSCode Extension
2. Open manualTestFile.py in VSCode
3. Run the first cell in the file
4. Switch to the Live Share tab in VS Code and start a session
  - Verify server start
    1. Jupyter server instance should appear in the live share tab
5. Open another window of VSCode
6. Connect the second instance of VSCode as a Guest to the first Live Share session
7. After the workspace opens, open the manualTestFile.py on the Guest instance
8. On the Guest instance run a cell from the file, both via the codelens and via the command palette Run Cell command
  - Verify results
    1. Output should show up on the Guest Interactive Window
    2. Same output should show up in the Host Interactive Window
9. On the Host instance run a cell from the file, both via the codelens and via the command palette
  - Verify results
    1. Output should show up on the Guest Interactive Window
    2. Same output should show up in the Host Interactive Window
10. Export the file to a notebook
11. Open the notebook editor on the host
12. Run a cell on the host
13. Verify the editor opens on the guest and the cell is run there too
Jupyter Hub support
1. Windows install instructions
  1. Install Docker Desktop onto a machine
  2. Create a folder with a file 'Dockerfile' in it.
  3. Mark the file to look like so:
```
ARG BASE_CONTAINER=jupyterhub/jupyterhub
FROM $BASE_CONTAINER

USER root

USER $NB_UID
```
  1. From a command prompt (in the same folder as the Dockerfile), run docker build -t jupyterhubcontainer:1.0 .
  2. Run docker container create --name jupyterhub jupyterhubcontainer:1.0 jupyterhub
  3. From the docker desktop app, start the jupyterhub container.
  4. From the docker desktop app, run the CLI
2. OR Mac / Linux install instructions
  1. Install docker
  2. From the terminal docker run -p 8000:8000 -d --name jupyterhub jupyterhub/jupyterhub jupyterhub
  3. Open a terminal in the docker container with docker exec -it jupyterhub bash
  4. From that terminal run python3 -m pip install notebook
3. From the new command prompt, run adduser testuser
4. Follow the series of prompts to add a password for this user
5. Open VS code
6. Open a folder with a python file in it.
7. Run the Python: Specify local or remote Jupyter server for connections command.
8. Pick 'Existing'
9. Enter http://localhost:8000 (assuming the jupyter hub container was successful in launching)
10. Reload VS code and reopen this folder.
11. Run a cell in a python file. [ ] Verify results 1. Verify you are asked first for a user name and then a password. 1. Verify a cell runs once you enter the user name and password 1. Verify that the python that is running in the interactive window is from the docker container (if on windows it should show a linux path)

P2 Test Scenarios

Directory change
- Verify directory change in export
  1. Follow the previous steps for export, but export the ipynb to a directory outside of the current workspace
  2. Open the file in the browser, you should get an initial cell added to change directory back to your workspace directory
- Verify directory change in import
  1. Follow the previous steps for import, but import an ipynb that is located outside of your current workspace
  2. Open the file in the editor. There should be python code at the start to change directory to the previous location of the .ipynb file
Interactive Window input history history
1. Start up an Interactive Window session
2. Input several lines into the Interactive Window terminal
3. Press up to verify that those previously entered lines show in the Interactive Window terminal history
Extra themes 1. Try several of the themes that come with VSCode that are not the default Dark+ and Light+

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test plan

Environment

Tests

Environment

Interpreters

Terminal

Virtual environments

Environment files

Debugging

Linting

Language server

Pylint/default linting

Other linters

Editing

IntelliSense

Formatting

Refactoring

Debugging

Unit testing

`unittest`

`pytest`

`nose`

General

Data Science

P0 Test Scenarios

P1 Test Scenarios

P2 Test Scenarios

FilesExpand file tree

test_plan.md

Latest commit

History

test_plan.md

File metadata and controls

Test plan

Environment

Tests

Interpreters

Terminal

Virtual environments

Language server

Pylint/default linting

Other linters

General

P0 Test Scenarios

P1 Test Scenarios

P2 Test Scenarios