Handle empty Android benchmark results (#5916)

huydhn · facebook-github-bot · commit 84f5a561e531 · 2024-10-06T00:12:48.000-07:00
Summary: The script didn't handle invalid JSON well when the benchmark results is empty https://github.com/pytorch/executorch/actions/runs/11197448597/job/31128464106. It's better to upload whatever it finds instead of crashing in this case. Looking a bit close at the benchmark jobs, there are some cases where the `benchmark_results.json` file is empty. I'm not sure why yet, but it can be fixed in another PR: * https://github.com/pytorch/executorch/actions/runs/11197448597/job/31127998979#step:15:286 (stories110M, qnn) * https://github.com/pytorch/executorch/actions/runs/11197448597/job/31127999221#step:15:946 (vit, xnnpack) The test spec already waits for 3 minutes to see if the file is there before giving up Pull Request resolved: #5916 Reviewed By: guangy10 Differential Revision: D63950793 Pulled By: huydhn fbshipit-source-id: 160d1465395c025e028b0e2cb69b5837a8f0208a
diff --git a/.github/scripts/extract_benchmark_results.py b/.github/scripts/extract_benchmark_results.py
@@ -129,6 +129,10 @@ def extract_android_benchmark_results(
     except error.HTTPError:
         warning(f"Fail to {artifact_type} {artifact_s3_url}")
         return []
+    except json.decoder.JSONDecodeError:
+        # This is to handle the case where there is no benchmark results
+        warning(f"Fail to load the benchmark results from {artifact_s3_url}")
+        return []
 
 
 def extract_job_id(artifacts_filename: str) -> int:
