Skip to content

Commit 9315191

Browse files
DonJayamanneDonJayamanne
authored
Fixes to UI Tests related to LS (#7659)
* Fix LS UI Tests * Warn instead of throwing exception * Create file * Increase delays * Added comments
1 parent b7ad1d1 commit 9315191

10 files changed

Lines changed: 144 additions & 140 deletions

File tree

‎uitests/bootstrap/extension/extension.js‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ const vscode = require('vscode');
88
const fs = require('fs');
99
const path = require('path');
1010
const util = require('util');
11-
11+
const lsOutputOpenedFile = path.join(__dirname, '..', 'lsoutputdisplayed.log');
1212
let activated = false;
1313
async function sleep(timeout) {
1414
return new Promise(resolve => setTimeout(resolve, timeout));
@@ -91,6 +91,9 @@ function activate(context) {
9191
continue;
9292
}
9393
}
94+
// Creating this file indicates the fact that the command was executed successfully.
95+
// The existence of this file is checked by the ui tests.
96+
await new Promise(resolve => fs.writeFile(lsOutputOpenedFile, '', resolve));
9497
});
9598
vscode.commands.registerCommand('smoketest.runInTerminal', async () => {
9699
const filePath = path.join(__dirname, '..', 'commands.txt');

‎uitests/features/languageServer/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Notes:
2+
3+
* Why do have delays of >= 120 seconds when using LS.
4+
* 1. This is to provide LS sufficient time to analyze the entire workspace.
5+
* E.g. when using a `conda` environment, LS takes a long time to complete analysis.
6+
* This results in delays in responsiveness of LS, i.e. features such as `intellisense`, `code navigation`, etc might not be ready or won't respond in time.
7+
* To determine whether the LS has completed analysis, we ensure the statusbar item `Analyzing in background` is no longer visible.
8+
* 2. Also, it takes time to download & extract the Language Server

‎uitests/features/languageServer/basic.feature‎

Lines changed: 13 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,20 @@
22
@code:code/languageServer/basic
33
Feature: Language Server
44
@smoke
5-
Scenario Outline: Check output of 'Python' output panel when starting VS Code with Jedi <jedi_enable>d
6-
When I <jedi_enable> the workspace setting "python.jediEnabled"
5+
Scenario: Check output of 'Python' output panel when starting VS Code with Jedi enabled
6+
When I enable the workspace setting "python.jediEnabled"
77
And I wait for the Python extension to activate
88
And I select the command "Python: Show Output"
9-
Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
9+
Then the text "Jedi Python language engine" will be displayed in the output panel within 10 seconds
1010

11-
Examples:
12-
| jedi_enable | time_to_activate | first_text_in_ooutput_panel |
13-
| enable | 10 | Jedi Python language engine |
14-
| disable | 120 | Microsoft Python language server |
11+
@smoke
12+
Scenario: Check output of 'Python' and 'Python Language Server' output panel when starting VS Code with Language Server enabled
13+
When I disable the workspace setting "python.jediEnabled"
14+
And I wait for the Python extension to activate
15+
And I select the command "Python: Show Output"
16+
Then the text "Microsoft Python language server" will be displayed in the output panel within 120 seconds
17+
When I select the command "Python: Show Language Server Output"
18+
Then the text "Initializing for" will be displayed in the output panel within 120 seconds
1519

1620
Scenario Outline: Language Server is downloaded with http.proxyStrictSSL setting <enabled_disabled>
1721
When I open VS Code for the first time
@@ -26,45 +30,5 @@ Feature: Language Server
2630

2731
Examples:
2832
| enabled_disabled | protocol_to_look_for |
29-
| enabled | https:// |
30-
| disabled | http:// |
31-
32-
@autoretry
33-
Scenario Outline: Navigate to definition of a variable when extension has already been activated with Jedi <jedi_enable>d
34-
When I reload VS Code
35-
And I <jedi_enable> the workspace setting "python.jediEnabled"
36-
And I wait for the Python extension to activate
37-
And I select the command "Python: Show Output"
38-
Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
39-
# Because LS is slow.
40-
And wait for <time_to_activate> seconds
41-
When I open the file "my_sample.py"
42-
And I go to line 3, column 10
43-
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
44-
And I wait for 10 seconds
45-
When I select the command "Go to Definition"
46-
Then the cursor is on line 1
47-
48-
Examples:
49-
| jedi_enable | time_to_activate | first_text_in_ooutput_panel |
50-
| enable | 10 | Jedi Python language engine |
51-
| disable | 120 | Microsoft Python language server |
52-
53-
# @autoretry @wip
54-
# Scenario Outline: Navigate to definition of a variable after opening a file with Jedi <jedi_enabled>
55-
# Given the workspace setting "python.jediEnabled" is <jedi_enabled>
56-
# When I open the file "my_sample.py"
57-
# And I wait for the Python extension to activate
58-
# And I select the command "Python: Show Output"
59-
# Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
60-
# And the text "<second_text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
61-
# When I go to line 3, column 10
62-
# # Wait for intellisense to kick in (sometimes slow in jedi & ls)
63-
# And I wait for 10 seconds
64-
# And I select the command "Go to Definition"
65-
# Then the cursor is on line 1
66-
67-
# Examples:
68-
# | jedi_enabled | time_to_activate | first_text_in_ooutput_panel | second_text_in_output_panel |
69-
# | enabled | 10 | Jedi Python language engine | Jedi Python language engine |
70-
# | disabled | 120 | Microsoft Python language server | Initializing for |
33+
| enabled | Downloading https:// |
34+
| disabled | Downloading http:// |

‎uitests/features/languageServer/goToDefinition.feature‎

Lines changed: 28 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,64 +1,47 @@
11
@ls
22
@code:code/languageServer/basic
33
Feature: Language Server
4-
Scenario Outline: When <reload_or_start_vs_for_first_time> with Jedi <jedi_enable>d then output contains <text_in_output_panel>
5-
When <reload_or_start_vs_for_first_time>
6-
And I <jedi_enable> the workspace setting "python.jediEnabled"
7-
And I wait for the Python extension to activate
8-
And I select the command "<output_panel_command>"
9-
Then the text "<text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
10-
11-
Examples:
12-
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | text_in_output_panel | output_panel_command |
13-
| enable | I open VS Code for the first time | 5 | Jedi Python language engine | Python: Show Output |
14-
| enable | I reload VS Code | 5 | Jedi Python language engine | Python: Show Output |
15-
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | Python: Show Output |
16-
| disable | I open VS Code for the first time | 120 | Downloading | Python: Show Language Server Output |
17-
| disable | I reload VS Code | 120 | Microsoft Python language server | Python: Show Output |
18-
194
@autoretry
20-
Scenario Outline: When <reload_or_start_vs_for_first_time> with Jedi <jedi_enable>d then navigate to definition of a variable
5+
Scenario Outline: When <reload_or_start_vs_for_first_time> then navigate to the definition of a variable after opening file with Jedi enabled
216
When <reload_or_start_vs_for_first_time>
22-
And I <jedi_enable> the workspace setting "python.jediEnabled"
7+
And I enable the workspace setting "python.jediEnabled"
238
And I wait for the Python extension to activate
24-
And I select the command "<output_panel_command>"
25-
Then the text "<text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
26-
# Because LS is slow.
27-
And wait for <time_to_activate> seconds
9+
And I select the command "Python: Show Output"
10+
Then the text "Jedi Python language engine" will be displayed in the output panel within 10 seconds
11+
# Sometimes LS & Jedi are slow.
12+
And wait for 10 seconds
2813
When I open the file "my_sample.py"
2914
And I go to line 3, column 10
3015
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
31-
And I wait for 5 seconds
32-
And I select the command "Go to Definition"
16+
And I wait for 10 seconds
17+
When I select the command "Go to Definition"
3318
Then the cursor is on line 1
3419

3520
Examples:
36-
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | text_in_output_panel | output_panel_command |
37-
| enable | I open VS Code for the first time | 5 | Jedi Python language engine | Python: Show Output |
38-
| enable | I reload VS Code | 5 | Jedi Python language engine | Python: Show Output |
39-
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | Python: Show Output |
40-
| disable | I open VS Code for the first time | 120 | Downloading | Python: Show Language Server Output |
41-
| disable | I reload VS Code | 120 | Microsoft Python language server | Python: Show Output |
21+
| reload_or_start_vs_for_first_time |
22+
| I open VS Code for the first time |
23+
| I reload VS Code |
4224

4325
@autoretry
44-
Scenario Outline: When I open VS Code for the first time with Jedi <jedi_enable>d, open a file then navigate to definition of a variable
45-
When I open VS Code for the first time
46-
And I <jedi_enable> the workspace setting "python.jediEnabled"
47-
And I select the command "Python: Show Output"
26+
Scenario Outline: When <reload_or_start_vs_for_first_time> then navigate to the definition of a variable after opening file with Language Server enabled
27+
When <reload_or_start_vs_for_first_time>
28+
And I disable the workspace setting "python.jediEnabled"
4829
And I wait for the Python extension to activate
49-
And I open the file "my_sample.py"
50-
And I select the command "<output_panel_command>"
51-
Then the text "<text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
52-
# Because LS is slow.
53-
And wait for <time_to_activate> seconds
54-
When I go to line 3, column 10
30+
And I select the command "Python: Show Output"
31+
Then the text "Microsoft Python language server" will be displayed in the output panel within 120 seconds
32+
When I select the command "Python: Show Language Server Output"
33+
Then the text "Initializing for" will be displayed in the output panel within 120 seconds
34+
# Sometimes LS & Jedi are slow.
35+
When I wait for 10 seconds
36+
Then the status bar item containing the text "Analyzing in background" will be hidden within 120 seconds
37+
When I open the file "my_sample.py"
38+
And I go to line 3, column 10
5539
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
56-
And I wait for 5 seconds
57-
And I select the command "Go to Definition"
40+
And I wait for 10 seconds
41+
When I select the command "Go to Definition"
5842
Then the cursor is on line 1
5943

6044
Examples:
61-
| jedi_enable | time_to_activate | text_in_output_panel | output_panel_command |
62-
| enable | 5 | Jedi Python language engine | Python: Show Output |
63-
| disable | 120 | Microsoft Python language server | Python: Show Output |
64-
| disable | 120 | Downloading | Python: Show Language Server Output |
45+
| reload_or_start_vs_for_first_time |
46+
| I open VS Code for the first time |
47+
| I reload VS Code |

‎uitests/features/languageServer/intellisense.feature‎

Lines changed: 28 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,19 @@ Feature: Language Server
77
And I <jedi_enable> the workspace setting "python.jediEnabled"
88
And I wait for the Python extension to activate
99
And I select the command "Python: Show Output"
10-
Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
11-
And wait for <time_to_activate> seconds
10+
Then the text "<first_text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
11+
And I select the command "<second_output_panel_command>"
12+
Then the text "<last_text_in_second_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
13+
# Sometimes LS & Jedi are slow.
14+
And wait for 10 seconds
15+
# Though this only applies to Language Server, there's no harm in testing this in Jedi (it won't exist).
16+
# One less column in the example section.
17+
And the status bar item containing the text "Analyzing in background" will be hidden within 120 seconds
1218
# Get more realestate on UI (hide what we don't need).
1319
And select the command "View: Close Panel"
1420
When I open the file "intelli_sample.py"
1521
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
16-
And I wait for <wait_time> seconds
22+
And wait for 10 seconds
1723
And I go to line 3, column 13
1824
And I press ctrl+space
1925
Then auto completion list contains the item "excepthook"
@@ -40,20 +46,26 @@ Feature: Language Server
4046
And auto completion list contains the item "count"
4147

4248
Examples:
43-
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | first_text_in_ooutput_panel | wait_time |
44-
| enable | I open VS Code for the first time | 5 | Jedi Python language engine | 5 |
45-
| enable | I reload VS Code | 5 | Jedi Python language engine | 5 |
46-
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | 5 |
47-
| disable | I reload VS Code | 120 | Microsoft Python language server | 5 |
49+
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | first_text_in_output_panel | last_text_in_second_output_panel | wait_time | second_output_panel_command |
50+
| enable | I open VS Code for the first time | 10 | Jedi Python language engine | Jedi Python language engine | 10 | Python: Show Output |
51+
| enable | I reload VS Code | 10 | Jedi Python language engine | Jedi Python language engine | 10 | Python: Show Output |
52+
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | Microsoft Python language server | 10 | Python: Show Language Server Output |
53+
| disable | I reload VS Code | 120 | Microsoft Python language server | Microsoft Python language server | 10 | Python: Show Language Server Output |
4854

4955
@autoretry
5056
Scenario Outline: When <reload_or_start_vs_for_first_time> with Jedi <jedi_enable>d then intellisense works for untitled files
5157
When <reload_or_start_vs_for_first_time>
5258
And I <jedi_enable> the workspace setting "python.jediEnabled"
5359
And I wait for the Python extension to activate
5460
And I select the command "Python: Show Output"
55-
Then the text "<first_text_in_ooutput_panel>" will be displayed in the output panel within <time_to_activate> seconds
56-
And wait for <time_to_activate> seconds
61+
Then the text "<first_text_in_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
62+
And I select the command "<second_output_panel_command>"
63+
Then the text "<last_text_in_second_output_panel>" will be displayed in the output panel within <time_to_activate> seconds
64+
# Sometimes LS & Jedi are slow.
65+
And wait for 10 seconds
66+
# Though this only applies to Language Server, there's no harm in testing this in Jedi (it won't exist).
67+
# One less column in the example section.
68+
And the status bar item containing the text "Analyzing in background" will be hidden within 120 seconds
5769
# Get more realestate on UI (hide what we don't need).
5870
And select the command "View: Close Panel"
5971
When I create a new file with the following content
@@ -64,16 +76,16 @@ Feature: Language Server
6476
"""
6577
And I change the language of the file to "Python"
6678
# Wait for intellisense to kick in (sometimes slow in jedi & ls)
67-
And I wait for <wait_time> seconds
79+
And wait for 10 seconds
6880
And I go to line 3, column 13
6981
And I press ctrl+space
7082
Then auto completion list contains the item "excepthook"
7183
And auto completion list contains the item "exec_prefix"
7284
And auto completion list contains the item "executable"
7385

7486
Examples:
75-
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | first_text_in_ooutput_panel | wait_time |
76-
| enable | I open VS Code for the first time | 5 | Jedi Python language engine | 5 |
77-
| enable | I reload VS Code | 5 | Jedi Python language engine | 5 |
78-
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | 5 |
79-
| disable | I reload VS Code | 120 | Microsoft Python language server | 5 |
87+
| jedi_enable | reload_or_start_vs_for_first_time | time_to_activate | first_text_in_output_panel | last_text_in_second_output_panel | wait_time | second_output_panel_command |
88+
| enable | I open VS Code for the first time | 10 | Jedi Python language engine | Jedi Python language engine | 10 | Python: Show Output |
89+
| enable | I reload VS Code | 10 | Jedi Python language engine | Jedi Python language engine | 10 | Python: Show Output |
90+
| disable | I open VS Code for the first time | 120 | Microsoft Python language server | Microsoft Python language server | 10 | Python: Show Language Server Output |
91+
| disable | I reload VS Code | 120 | Microsoft Python language server | Microsoft Python language server | 10 | Python: Show Language Server Output |

0 commit comments

Comments
 (0)