Skip to content

Add torch.profiler submodule telemetry#15830

Merged
joyceerhl merged 1 commit into
microsoft:mainfrom
joyceerhl:dev/joyceerhl/torch-profiler-telemetry
Apr 5, 2021
Merged

Add torch.profiler submodule telemetry#15830
joyceerhl merged 1 commit into
microsoft:mainfrom
joyceerhl:dev/joyceerhl/torch-profiler-telemetry

Conversation

@joyceerhl

Copy link
Copy Markdown

For #15825

@joyceerhl joyceerhl added the skip tests Updates to tests unnecessary label Apr 1, 2021
@joyceerhl
joyceerhl marked this pull request as ready for review April 1, 2021 18:23
}
}
if (s && TorchProfilerImportRegEx.test(s)) {
sendTelemetryEvent(EventName.TENSORBOARD_TORCH_PROFILER_IMPORT);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems unnecessary as we already have import telemetry? Why is tensorboard profiler special?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because our existing import telemetry only tracks top level modules, i.e. we know how many people are importing torch, but not the submodules within torch. This is a request from Rong and Jeffrey so they can measure the PyTorch profiler user base size.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suppose that we don't worry about PII like with the package hashing since it's just a single known package and module. Might be worth generalizing at some point, but we can worry about that if we have more submodules that we want to look for in the future. No need to generalize now.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@karrtikr @karthiknadig do you have any concerns about modifying the existing import regex and changing the HASHED_PACKAGE_NAME telemetry event to send both the top-level module name as well as the specific submodules that the user imported?

This would let us reuse an existing event rather than using a separate regex and separate telemetry event, and would be more generalizable to future requests to track imports with submodule granularity. Are there any implications of doing this from the Python extension team's perspective (either PM or dev)?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might impact existing telemetry. I don't have any concerns from the extension perspective. @luabud Do you use this information in an way that might impact analysis in any way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joyceerhl Sorry didn't mean to complicate things now. Was more of a question for the future if we want to do more of this. I'm fine with the one-off for this.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be clear, right now the regular telemetry will gather this:

import torch as pyt

but won't gather this

import torch.profiler as pyt

In fact on the second one, the import is ignored.

Joyce's suggestion would leave the original import in place, but then add a new hash for the 'torch.profiler' one.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elaborating on that, I'm proposing that for something like from torch import profiler or import torch.profiler, we'd have the following two attached properties on the existing telemetry event:

{
    hashedName: string // Existing property representing the hashed top-level package name, e.g. `torch`
    hashedFullyQualifiedName: string // Hash of `torch.profiler`
}

In importTracker.ts we would construct the fully qualified, '.'-delimited import name, hash that, and send it as the second property.

A limitation is that this won't be able to distinguish submodules from functions, e.g. for an import like from torch.utils.tensorboard import SummaryWriter the fully qualified name is torch.utils.tensorboard.SummaryWriter. The PMs would have to account for all possible submodule import paths as distinct hashes in their queries.

@rchiodo rchiodo Apr 2, 2021

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PMs would have to account for all possible submodule import paths as distinct hashes in their queries.

Only if they wanted to look at the hashedFullyQualifedName. Otherwise the current query would remain the same.

@rchiodo rchiodo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this is the second special case for import telemetry, I'd rather we made it more general now, but if there's a rush, seems okay for now.

@joyceerhl
joyceerhl merged commit b55a31a into microsoft:main Apr 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip tests Updates to tests unnecessary

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants