Skip to content

Commit d727f63

Browse files
gokulavasanfacebook-github-bot
gokulavasan
authored and
facebook-github-bot
committed
Address CI test failures in pytorch/data (meta-pytorch#1219)
Summary: ### Changes - Remove torchvision tests from domain_ci. These tests are failing in CI and torchvision stopped maintenance of that part of the codebase (https://github.com/pytorch/vision/blob/main/.github/workflows/prototype-tests-linux-gpu.yml#L47-L49) - Change AIStore branch name - Run pre-commit hook Pull Request resolved: meta-pytorch#1219 Reviewed By: kartikayk, huydhn, ejguan Differential Revision: D52903377 Pulled By: gokulavasan fbshipit-source-id: 1deaee3b88b8c062cedd718e7c268019fc2b3065
1 parent c3d1c1a commit d727f63

File tree

7 files changed

+16
-78
lines changed

7 files changed

+16
-78
lines changed

‎.github/workflows/aistore_ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ jobs:
4848
pip3 install -r requirements.txt
4949
pip3 install --pre torch -f "${{ steps.pytorch_channel.outputs.value }}"
5050
- name: Run AIStore local deployment
51-
uses: NVIDIA/aistore@master
51+
uses: NVIDIA/aistore@main
5252
- name: Build TorchData
5353
run: |
5454
pip3 install .

‎.github/workflows/domain_ci.yml‎

Lines changed: 0 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -10,55 +10,6 @@ on:
1010
- gh/*/*/base
1111

1212
jobs:
13-
torchvision:
14-
if: ${{ github.repository_owner == 'pytorch' }}
15-
runs-on: ${{ matrix.os }}
16-
strategy:
17-
fail-fast: false
18-
matrix:
19-
os:
20-
- macos-latest
21-
- ubuntu-latest
22-
- windows-latest
23-
python-version:
24-
- 3.8
25-
- 3.9
26-
steps:
27-
- name: Setup Python ${{ matrix.python-version }}
28-
uses: actions/setup-python@v4
29-
with:
30-
python-version: ${{ matrix.python-version }}
31-
32-
- name: Install torch and torchvision from nightlies
33-
run: |
34-
pip install numpy networkx
35-
pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cpu
36-
37-
- name: Check out torchdata repository
38-
uses: actions/checkout@v3
39-
40-
- name: Install torchdata
41-
run: |
42-
pip install -r requirements.txt
43-
pip install .
44-
45-
- name: Install test requirements
46-
run: pip install pytest pytest-mock scipy iopath pycocotools h5py
47-
48-
- name: Extract torchvision ref
49-
id: torchvision
50-
run: echo "ref=$(python -c 'import torchvision; print(torchvision.version.git_version)')" >> $GITHUB_OUTPUT
51-
52-
- name: Check out torchvision repository
53-
uses: actions/checkout@v3
54-
with:
55-
repository: pytorch/vision
56-
ref: ${{ steps.torchvision.outputs.ref }}
57-
path: vision
58-
59-
- name: Run torchvision builtin datasets tests
60-
run: pytest --no-header -v vision/test/test_prototype_datasets_builtin.py
61-
6213
torchtext:
6314
if: ${{ github.repository_owner == 'pytorch' }}
6415
runs-on: ${{ matrix.os }}

‎README.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,11 @@
44
[**What are DataPipes?**](#what-are-datapipes) | [**Beta Usage and Feedback**](#beta-usage-and-feedback) |
55
[**Contributing**](#contributing) | [**Future Plans**](#future-plans)
66

7-
**:warning: As of July 2023, we have paused active development on TorchData and have paused new releases. We have learnt a lot from building it and hearing from users, but also believe we need to re-evaluate the technical design and approach given how much the industry has changed since we began the project. During the rest of 2023 we will be re-evaluating our plans in this space. Please reach out if you suggestions or comments (please use [#1196](https://github.com/pytorch/data/issues/1196) for feedback).**
7+
**:warning: As of July 2023, we have paused active development on TorchData and have paused new releases. We have learnt
8+
a lot from building it and hearing from users, but also believe we need to re-evaluate the technical design and approach
9+
given how much the industry has changed since we began the project. During the rest of 2023 we will be re-evaluating our
10+
plans in this space. Please reach out if you suggestions or comments (please use
11+
[#1196](https://github.com/pytorch/data/issues/1196) for feedback).**
812

913
`torchdata` is a library of common modular data loading primitives for easily constructing flexible and performant data
1014
pipelines.

‎examples/dataloader2/train_loop.py‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ def forward(self, x: torch.Tensor) -> torch.Tensor:
2626
"""
2727
Simple model forward function
2828
"""
29-
return self.a + self.b * x + self.c * x**2 + self.d * x**3
29+
return self.a + self.b * x + self.c * x ** 2 + self.d * x ** 3
3030

3131

3232
def main() -> None:
@@ -69,10 +69,7 @@ def main() -> None:
6969
running_loss += loss.item()
7070
# Print the loss every 2000 mini-batches.
7171
if step % 2000 == 1999:
72-
print(
73-
"[epoch: %d, %5d] loss: %.3f"
74-
% (epoch + 1, step + 1, running_loss / 2000)
75-
)
72+
print("[epoch: %d, %5d] loss: %.3f" % (epoch + 1, step + 1, running_loss / 2000))
7673
running_loss = 0.0
7774

7875
print("Finished Training")

‎examples/dataloader2/train_loop_distributed_reading_service.py‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ def forward(self, x: torch.Tensor) -> torch.Tensor:
3131
"""
3232
Simple model forward function
3333
"""
34-
return self.a + self.b * x + self.c * x**2 + self.d * x**3
34+
return self.a + self.b * x + self.c * x ** 2 + self.d * x ** 3
3535

3636

3737
def main() -> None:
@@ -84,10 +84,7 @@ def main() -> None:
8484

8585
running_loss += loss.item()
8686
if step % 2000 == 1999:
87-
print(
88-
"[epoch: %d, %5d] loss: %.3f"
89-
% (epoch + 1, step + 1, running_loss / 2000)
90-
)
87+
print("[epoch: %d, %5d] loss: %.3f" % (epoch + 1, step + 1, running_loss / 2000))
9188
running_loss = 0.0
9289

9390
print("Finished Training")

‎examples/dataloader2/train_loop_reading_service.py‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ def forward(self, x: torch.Tensor) -> torch.Tensor:
2727
"""
2828
Simple model forward function
2929
"""
30-
return self.a + self.b * x + self.c * x**2 + self.d * x**3
30+
return self.a + self.b * x + self.c * x ** 2 + self.d * x ** 3
3131

3232

3333
def main() -> None:
@@ -63,10 +63,7 @@ def main() -> None:
6363

6464
running_loss += loss.item()
6565
if step % 2000 == 1999:
66-
print(
67-
"[epoch: %d, %5d] loss: %.3f"
68-
% (epoch + 1, step + 1, running_loss / 2000)
69-
)
66+
print("[epoch: %d, %5d] loss: %.3f" % (epoch + 1, step + 1, running_loss / 2000))
7067
running_loss = 0.0
7168

7269
print("Finished Training")

‎examples/dataloader2/train_loop_torchtext.py‎

Lines changed: 4 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,7 @@
2424

2525

2626
XLMR_VOCAB_PATH = r"https://download.pytorch.org/models/text/xlmr.vocab.pt"
27-
XLMR_SPM_MODEL_PATH = (
28-
r"https://download.pytorch.org/models/text/xlmr.sentencepiece.bpe.model"
29-
)
27+
XLMR_SPM_MODEL_PATH = r"https://download.pytorch.org/models/text/xlmr.sentencepiece.bpe.model"
3028

3129
DEVICE = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
3230

@@ -70,9 +68,7 @@ def evaluate() -> None:
7068
counter = 0
7169
with torch.no_grad():
7270
for batch in eval_dataloader:
73-
input = F.to_tensor(batch["token_ids"], padding_value=PADDING_IDX).to(
74-
DEVICE
75-
)
71+
input = F.to_tensor(batch["token_ids"], padding_value=PADDING_IDX).to(DEVICE)
7672
target = torch.tensor(batch["target"]).to(DEVICE)
7773
loss, predictions = eval_step(input, target)
7874
total_loss += loss
@@ -101,9 +97,7 @@ def main() -> None:
10197
eval_dataloader = DataLoader2(datapipe=eval_datapipe)
10298
print("Created eval dataloader")
10399

104-
classifier_head = torchtext.models.RobertaClassificationHead(
105-
num_classes=NUM_CLASSES, input_dim=INPUT_DIM
106-
)
100+
classifier_head = torchtext.models.RobertaClassificationHead(num_classes=NUM_CLASSES, input_dim=INPUT_DIM)
107101
model = torchtext.models.XLMR_BASE_ENCODER.get_model(head=classifier_head)
108102
model.to(DEVICE)
109103

@@ -112,9 +106,7 @@ def main() -> None:
112106

113107
for epoch in range(NUM_EPOCHS):
114108
for step, batch in enumerate(train_dataloader):
115-
input = F.to_tensor(batch["token_ids"], padding_value=PADDING_IDX).to(
116-
DEVICE
117-
)
109+
input = F.to_tensor(batch["token_ids"], padding_value=PADDING_IDX).to(DEVICE)
118110
target = torch.tensor(batch["target"]).to(DEVICE)
119111
train_step(input, target)
120112

0 commit comments

Comments
 (0)