A/B testing issue: #5042

Environment data

• VS Code version: 1.35.1
• Extension version (available under the Extensions sidebar): commit 6b1b8a3
• OS and version: Mac (OSX 10.14.5)
• Python version (& distribution if applicable, e.g. Anaconda): 3.7
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Relevant/affected Python packages and their versions: N/A
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Jedi

Expected behaviour

Opting out of telemetry opts out of A/B testing, even if the local experiments.json is populated.

Actual behaviour

Opting out of telemetry doesn't prevent the local experiments.json file from being read and enabling experiments for the user.

Steps to reproduce:

1. Update experiments.json with:

[
    {
        "name": "LS - control",
        "salt": "LS",
        "min": 0,
        "max": 10
    },
    {
        "name": "LS - enabled",
        "salt": "LS",
        "min": 85,
        "max": 100
    },
    {
        "name": "AlwaysDisplayTestExplorer - enabled",
        "salt": "LS",
        "min": 70,
        "max": 90
    },
    {
        "name": "AlwaysDisplayTestExplorer - control",
        "salt": "LS",
        "min": 40,
        "max": 50
    }
]

2. Check your hash (see @karrtikr 's comment) and salt and see which experiment(s) you belong to (in my case I had LS - enabled and AlwaysDisplayTestExplorer - enabled)
3. Go to settings and un-check 'Telemetry: Enable Telemetry' (or set telemetry.enableTelemetry to false)
4. Set a breakpoint at line 106 of experiments.ts
5. Start debugging the extension inside a container and notice that the breakpoint is being hit

Logs

Output for Python in the Output panel after disabling telemetry:

User belongs to experiment group 'LS - enabled'
User belongs to experiment group 'AlwaysDisplayTestExplorer - enabled'
Starting Microsoft Python language server.