Log experiments run via ExP in the output panel#12742
Conversation
| } | ||
|
|
||
| private logExperiments() { | ||
| const experiments = this.globalState.get<{ features: string[] }>(EXP_MEMENTO_KEY, { features: [] }); |
There was a problem hiding this comment.
Does this return all experiments the user belongs to? Or all experiments running on behalf of the Python extension.
We should be logging the prior one, right? The news entry and naming is making it seem like we're logging all the experiments, and not just the one the user belongs to.
There was a problem hiding this comment.
If the user doesn't belong to an experiment it won't be in storage, so by default it returns all experiments and control groups the user belongs to. Then because we don't want to log experiments that are not run by the Python extension (agreed on with Luciana) we filter them out.
There was a problem hiding this comment.
I don't think the news entry nor the naming are ambiguous (we're logging experiments inside the Python extension, why would we log experiments that are not from the Python extension), but I updated the issue content provide an answer to your question.
There was a problem hiding this comment.
This isn't a public API (accessing the stored experiments from the global storage), shouldn't we ask for a public API instead.
I.e. the storage key isn't public.
There was a problem hiding this comment.
We asked, and they said they were working on it. If I remember correctly other teams also asked for this feature, so hopefully there will be an "official" way to do it soon. In the meantime, they are the ones who provided us with this workaround.
There was a problem hiding this comment.
we're logging experiments inside the Python extension, why would we log experiments that are not from the Python extension
What I was trying to say is, by looking at the news entry which says
Log experiments run via ExP in the output panel
It seems we're logging all experiments registered in the Python extension - whether the user is in it or not.
There was a problem hiding this comment.
Sure, I can amend it to be "Log ExP experiments the user belongs to in the output panel.".
There was a problem hiding this comment.
Great, I also recommend adding a doc comment in the method logExperiments(), which says it logs the experiment the user belongs to.
There was a problem hiding this comment.
The behaviour is similar as the existing experiment manager, which already only logs the experiments the user belongs to. Instead I'll clarify the existing comment to say that we filter out experiments that are not run by the Python extension.
|
Kudos, SonarCloud Quality Gate passed!
|
For #12656
🕯️ We need to filter the experiments we log since the ExP platform is also used by other teams. After talking with @luabud we agreed to prefix all future experiment names with
python(will update the documentation). The start page experiment doesn't follow this convention, but it should be phased out soon so it's not a problem.package-lock.jsonhas been regenerated by runningnpm install(if dependencies have changed).