Flaky test: test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[data_file_name_from_first_file_name-native_multipart] #90662

Activity

1. maxknv

   added

   testingSpecial issue with list of bugs found by CISpecial issue with list of bugs found by CI

   flaky testflaky test found by CIflaky test found by CI

   on Nov 22, 2025
2. alexey-milovidov commented on Feb 11, 2026

   

   alexey-milovidov

   on Feb 11, 2026

   Member

   More actions

   Possibly related fix: #96298
3. alexey-milovidov commented on Mar 16, 2026

   

   alexey-milovidov

   on Mar 16, 2026

   Member

   More actions

   No recent failures.
4. alexey-milovidov

   closed this as completedon Mar 16, 2026
5. alexey-milovidov

   self-assigned this

   on Mar 16, 2026
6. Avogar

   reopened this on Apr 1, 2026
7. Avogar commented on Apr 1, 2026

   

   Avogar

   on Apr 1, 2026

   Member

   More actions

   Still flaky https://play.clickhouse.com/play?user=play&run=1#V0lUSAogICAgOTAgQVMgaW50ZXJ2YWxfZGF5cwpTRUxFQ1QKICAgIHRvU3RhcnRPZkRheShjaGVja19zdGFydF90aW1lKSBBUyBkYXksCiAgICBjb3VudCgpIEFTIGZhaWx1cmVzLAogICAgZ3JvdXBVbmlxQXJyYXkocHVsbF9yZXF1ZXN0X251bWJlcikgQVMgcHJzLAogICAgYW55KHJlcG9ydF91cmwpIEFTIHJlcG9ydF91cmwKRlJPTSBjaGVja3MKV0hFUkUgKG5vdygpIC0gdG9JbnRlcnZhbERheShpbnRlcnZhbF9kYXlzKSkgPD0gY2hlY2tfc3RhcnRfdGltZQogICAgQU5EIHRlc3RfbmFtZSA9ICd0ZXN0X2JhY2t1cF9yZXN0b3JlX3MzL3Rlc3QucHk6OnRlc3RfYmFja3VwX3RvX3MzX2RpZmZlcmVudF9jcmVkZW50aWFsc1tkYXRhX2ZpbGVfbmFtZV9mcm9tX2ZpcnN0X2ZpbGVfbmFtZS1uYXRpdmVfbXVsdGlwYXJ0XScKICAgIC0tIEFORCBjaGVja19uYW1lID0gJ0ludGVncmF0aW9uIHRlc3RzIChhbWRfYXNhbl91YnNhbiwgZGIgZGlzaywgb2xkIGFuYWx5emVyLCAyLzYpJwogICAgQU5EIHRlc3Rfc3RhdHVzIElOICgnRkFJTCcsICdFUlJPUicpCiAgICBBTkQgKHB1bGxfcmVxdWVzdF9udW1iZXIgPSAwIE9SIGJhc2VfcmVmIElOICgnbWFzdGVyJykpCiAgICBBTkQgdGVzdF9jb250ZXh0X3JhdyBMSUtFICclQXNzZXJ0aW9uRXJyb3IlJwpHUk9VUCBCWSBkYXkKT1JERVIgQlkgZGF5IERFU0MK
8. Avogar commented on Apr 1, 2026

   

   Avogar

   on Apr 1, 2026

   Member

   More actions

   @groeneai try to fix this flaky test
9. groeneai commented on Apr 1, 2026

   

   groeneai

   on Apr 1, 2026

   Collaborator

   More actions

   @Avogar working on a fix for this.

   Root cause: The assertion assert "S3ReadRequestsErrors" not in events (line 1004 in test.py) is too strict. S3ReadRequestsErrors counts all non-throttling errors in GET/HEAD requests, including transient network errors that are retried and succeed. Under ASAN/UBSAN overhead in CI, MinIO operations are slower and transient read errors become more likely during the large 10M-row data transfer. The backup/restore operation itself succeeds (data integrity passes).

   CIDB shows the failures are concentrated on amd_asan_ubsan, db disk, old analyzer checks — confirming the sanitizer overhead correlation.

   The test's core purpose (verifying native copy fallback with different credentials) is validated by the other assertions (S3CopyObject, S3WriteRequestsErrors, S3CreateMultipartUpload) and the data integrity check. The read-error assertion tests transport reliability rather than credentials behavior.

   Fix PR incoming shortly.
10. groeneai

    added a commit that references this issue on Apr 1, 2026

    Fix flaky test_backup_to_s3_different_credentials by removing overly …

    ...

    This issue will close once commit 5c21aaa is merged into the 'master' branch.

    5c21aaa
11. alexey-milovidov

    added a commit that references this issue on Apr 7, 2026

    Fix flaky test_backup_to_s3_different_credentials

    ...

    06471b3
12. vitlibar

    closed this as completedin #101499on Apr 7, 2026
13. zeekay

    added a commit that references this issue on Sep 11, 2026

    Fix flaky test_backup_to_s3_different_credentials

    ...

    667c939