Skip to content

Commit bdcf3e2

Browse files
ajarrajarr
committed
mgr/rbd_support: Fix "start-time" arg behavior
The "start-time" argument, optionally passed when adding or removing an mirror image snapshot schedule or a trash purge schedule, does not behave as intended. It is meant to schedule an initial operation at a specific time of day in a given time zone. Instead, it offsets the schedule’s anchor time. By default, the scheduler uses the UNIX epoch as the anchor to calculate recurring schedule times, and "start-time" simply shifts this anchor away from UTC, which can confuse users. For example: ``` $ # current time $ date --universal Wed Dec 10 05:55:21 PM UTC 2025 $ rbd mirror snapshot schedule add -p data --image img1 1h 19:00Z $ rbd mirror snapshot schedule ls -p data --image img1 every 15m starting at 19:00:00+00:00 ``` A user might assume that the scheduler will run the first snapshot each day at 19:00 UTC and then run snapshots every 15 minutes. Instead, the scheduler runs the first snapshot at 18:00 UTC and then continues at the configured interval: ``` $ rbd mirror snapshot schedule status -p data --image img1 SCHEDULE TIME IMAGE 2025-12-10 18:00:00 data/img1 ``` Additionally, the "start-time" argument accepts a full ISO 8601 timestamp but silently ignores everything except hour, minute, and time zone. Even time zone handling is incorrect: specifying "23:00-01:00" with an interval of "1d" results in a snapshot taken once per day at 22:00 UTC rather than 00:00 UTC, because only utcoffset.seconds is used while utcoffset.days is ignored. Fix: Similar to the handling of the "start" argument in the FS snap-schedule manager module, require "start-time" to use an ISO 8601 date-time format with a mandatory date component. Time and time zone are optional and default to 00:00 and UTC respectively. The "start-time" now defines the anchor time used to compute recurring schedule times. The default anchor remains the UNIX epoch. Existing on-disk schedules with legacy-format "start-time" values are updated to include the date Jan 1, 1970. The `snap schedule ls` output now displays "start-time" with date and time in the format "%Y-%m-%d %H:%M:00". The display time is in UTC. Fixes: https://tracker.ceph.com/issues/74192 Signed-off-by: Ramana Raja <rraja@redhat.com>
1 parent bacb971 commit bdcf3e2

7 files changed

Lines changed: 163 additions & 118 deletions

File tree

‎PendingReleaseNotes‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,11 @@
1515
and the scrubbing process is delayed between each read in order to avoid monopolizing
1616
the I/O capacity of the OSD.
1717
The default stride size (``osd_deep_scrub_stride``) was 512 KBytes, and is now 4 MBytes.
18+
* RBD: Fixed incorrect behavior of the "start-time" argument for mirror
19+
snapshot and trash purge schedules, where it previously offset the schedule
20+
anchor instead of defining it. The argument now requires an ISO 8601
21+
date-time. The `schedule ls` output displays the start time in UTC, including
22+
the date and time in the format "%Y-%m-%d %H:%M:00".
1823

1924
>=20.0.0
2025

‎doc/man/8/rbd.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1037,7 +1037,7 @@ To restore an image from trash and rename it::
10371037

10381038
To create a mirror snapshot schedule for an image::
10391039

1040-
rbd mirror snapshot schedule add --pool mypool --image myimage 12h 14:00:00-05:00
1040+
rbd mirror snapshot schedule add --pool mypool --image myimage 12h 2020-01-14T11:30+05:30
10411041

10421042
Availability
10431043
============

‎doc/rbd/rbd-mirroring.rst‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -428,10 +428,11 @@ image name; interval; and optional start time::
428428
rbd mirror snapshot schedule add [--pool {pool-name}] [--image {image-name}] {interval} [{start-time}]
429429

430430
The ``interval`` can be specified in days, hours, or minutes using ``d``, ``h``,
431-
``m`` suffix respectively. The optional ``start-time`` can be specified using
432-
the ISO 8601 time format. For example::
431+
``m`` suffix respectively. The optional ``start-time`` must be specified in
432+
the ISO 8601 time format. If no UTC offset is provided, UTC is assumed. For
433+
example::
433434

434-
$ rbd --cluster site-a mirror snapshot schedule add --pool image-pool 24h 14:00:00-05:00
435+
$ rbd --cluster site-a mirror snapshot schedule add --pool image-pool 24h 2020-01-14T11:30+05:30
435436
$ rbd --cluster site-a mirror snapshot schedule add --pool image-pool --image image1 6h
436437

437438
To remove a mirror-snapshot schedules with ``rbd``, specify the
@@ -441,12 +442,13 @@ corresponding ``add`` schedule command.
441442
To list all snapshot schedules for a specific level (global, pool, or image)
442443
with ``rbd``, specify the ``mirror snapshot schedule ls`` command along with
443444
an optional pool or image name. Additionally, the ``--recursive`` option can
444-
be specified to list all schedules at the specified level and below. For
445-
example::
445+
be specified to list all schedules at the specified level and below.
446+
447+
Schedule start times are always displayed in UTC. For example::
446448

447449
$ rbd --cluster site-a mirror snapshot schedule ls --pool image-pool --recursive
448450
POOL NAMESPACE IMAGE SCHEDULE
449-
image-pool - - every 1d starting at 14:00:00-05:00
451+
image-pool - - every 1d starting at 2020-01-14 06:00:00
450452
image-pool image1 every 6h
451453

452454
To view the status for when the next snapshots will be created for
@@ -456,11 +458,12 @@ image name::
456458

457459
rbd mirror snapshot schedule status [--pool {pool-name}] [--image {image-name}]
458460

459-
For example::
461+
The next schedule time is always displayed in UTC. For example::
460462

461463
$ rbd --cluster site-a mirror snapshot schedule status
462464
SCHEDULE TIME IMAGE
463-
2020-02-26 18:00:00 image-pool/image1
465+
2026-01-24 06:00:00 image-pool/image1
466+
464467

465468
Disable Image Mirroring
466469
-----------------------

‎qa/workunits/rbd/cli_generic.sh‎

Lines changed: 57 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1167,12 +1167,21 @@ test_trash_purge_schedule() {
11671167
expect_fail rbd trash purge schedule remove -p rbd dummy
11681168
expect_fail rbd trash purge schedule remove -p rbd 1d dummy
11691169

1170-
rbd trash purge schedule add -p rbd 1d 01:30
1170+
rbd trash purge schedule add -p rbd 1h 2100-01-01T19:00Z
1171+
test "$(rbd trash purge schedule ls -p rbd)" = 'every 1h starting at 2100-01-01 19:00:00'
1172+
for i in `seq 12`; do
1173+
rbd trash purge schedule status -p rbd | grep '2100-01-01 19:00:00' && break
1174+
sleep 10
1175+
done
1176+
test "$(rbd trash purge schedule status -p rbd --format xml |
1177+
xmlstarlet sel -t -v '//scheduled/item/schedule_time')" = '2100-01-01 19:00:00'
1178+
rbd trash purge schedule rm -p rbd
11711179

1172-
rbd trash purge schedule ls -p rbd | grep 'every 1d starting at 01:30'
1180+
rbd trash purge schedule add -p rbd 1d 2020-01-14T07:00+05:30
1181+
rbd trash purge schedule ls -p rbd | grep 'every 1d starting at 2020-01-14 01:30:00'
11731182
expect_fail rbd trash purge schedule ls
1174-
rbd trash purge schedule ls -R | grep 'every 1d starting at 01:30'
1175-
rbd trash purge schedule ls -R -p rbd | grep 'every 1d starting at 01:30'
1183+
rbd trash purge schedule ls -R | grep 'every 1d starting at 2020-01-14 01:30:00'
1184+
rbd trash purge schedule ls -R -p rbd | grep 'every 1d starting at 2020-01-14 01:30:00'
11761185
expect_fail rbd trash purge schedule ls -p rbd2
11771186
test "$(rbd trash purge schedule ls -p rbd2 -R --format json)" = "[]"
11781187

@@ -1193,18 +1202,18 @@ test_trash_purge_schedule() {
11931202
test "$(rbd trash purge schedule status -p rbd --format xml |
11941203
xmlstarlet sel -t -v '//scheduled/item/pool')" = 'rbd'
11951204

1196-
rbd trash purge schedule add 2d 00:17
1197-
rbd trash purge schedule ls | grep 'every 2d starting at 00:17'
1198-
rbd trash purge schedule ls -R | grep 'every 2d starting at 00:17'
1205+
rbd trash purge schedule add 2d 2020-01-14T05:47+05:30
1206+
rbd trash purge schedule ls | grep 'every 2d starting at 2020-01-14 00:17:00'
1207+
rbd trash purge schedule ls -R | grep 'every 2d starting at 2020-01-14 00:17:00'
11991208
expect_fail rbd trash purge schedule ls -p rbd2
1200-
rbd trash purge schedule ls -p rbd2 -R | grep 'every 2d starting at 00:17'
1201-
rbd trash purge schedule ls -p rbd2/ns1 -R | grep 'every 2d starting at 00:17'
1209+
rbd trash purge schedule ls -p rbd2 -R | grep 'every 2d starting at 2020-01-14 00:17:00'
1210+
rbd trash purge schedule ls -p rbd2/ns1 -R | grep 'every 2d starting at 2020-01-14 00:17:00'
12021211
test "$(rbd trash purge schedule ls -R -p rbd2/ns1 --format xml |
12031212
xmlstarlet sel -t -v '//schedules/schedule/pool')" = "-"
12041213
test "$(rbd trash purge schedule ls -R -p rbd2/ns1 --format xml |
12051214
xmlstarlet sel -t -v '//schedules/schedule/namespace')" = "-"
12061215
test "$(rbd trash purge schedule ls -R -p rbd2/ns1 --format xml |
1207-
xmlstarlet sel -t -v '//schedules/schedule/items/item/start_time')" = "00:17:00"
1216+
xmlstarlet sel -t -v '//schedules/schedule/items/item/start_time')" = "2020-01-14 00:17:00"
12081217

12091218
for i in `seq 12`; do
12101219
rbd trash purge schedule status --format xml |
@@ -1222,18 +1231,18 @@ test_trash_purge_schedule() {
12221231
xmlstarlet sel -t -v '//scheduled/item/pool'))" = 'rbd2 rbd2'
12231232

12241233
test "$(echo $(rbd trash purge schedule ls -R --format xml |
1225-
xmlstarlet sel -t -v '//schedules/schedule/items'))" = "2d00:17:00 1d01:30:00"
1234+
xmlstarlet sel -t -v '//schedules/schedule/items/item'))" = "2d2020-01-14 00:17:00 1d2020-01-14 01:30:00"
12261235

12271236
rbd trash purge schedule add 1d
1228-
rbd trash purge schedule ls | grep 'every 2d starting at 00:17'
1237+
rbd trash purge schedule ls | grep 'every 2d starting at 2020-01-14 00:17:00'
12291238
rbd trash purge schedule ls | grep 'every 1d'
12301239

12311240
rbd trash purge schedule ls -R --format xml |
1232-
xmlstarlet sel -t -v '//schedules/schedule/items' | grep '2d00:17'
1241+
xmlstarlet sel -t -v '//schedules/schedule/items' | grep '2d2020-01-14 00:17:00'
12331242

12341243
rbd trash purge schedule rm 1d
1235-
rbd trash purge schedule ls | grep 'every 2d starting at 00:17'
1236-
rbd trash purge schedule rm 2d 00:17
1244+
rbd trash purge schedule ls | grep 'every 2d starting at 2020-01-14 00:17:00'
1245+
rbd trash purge schedule rm 2d 2020-01-14T00:17:00
12371246
expect_fail rbd trash purge schedule ls
12381247

12391248
for p in rbd2 rbd2/ns1; do
@@ -1272,9 +1281,17 @@ test_trash_purge_schedule() {
12721281
expect_fail rbd trash purge schedule remove -p rbd 1d dummy
12731282
expect_fail rbd trash purge schedule remove dummy
12741283
expect_fail rbd trash purge schedule remove 1d dummy
1275-
rbd trash purge schedule ls -p rbd | grep 'every 1d starting at 01:30'
1284+
expect_fail rbd trash purge schedule add -p rbd 30m 00:15
1285+
expect_fail rbd trash purge schedule add -p rbd 30m 00:15+05:30
1286+
expect_fail rbd trash purge schedule add -p rbd 30m 2020-13-14T00:15+05:30
1287+
expect_fail rbd trash purge schedule add -p rbd 30m 2020-01-32T00:15+05:30
1288+
expect_fail rbd trash purge schedule add -p rbd 30m 2020-01-14T25:15+05:30
1289+
expect_fail rbd trash purge schedule add -p rbd 30m 2020-01-14T00:60+05:30
1290+
expect_fail rbd trash purge schedule add -p rbd 30m 2020-01-14T00:15+24:00
1291+
1292+
rbd trash purge schedule ls -p rbd | grep 'every 1d starting at 2020-01-14 01:30:00'
12761293
rbd trash purge schedule ls | grep 'every 2m'
1277-
rbd trash purge schedule remove -p rbd 1d 01:30
1294+
rbd trash purge schedule remove -p rbd 1d 2020-01-14T01:30
12781295
rbd trash purge schedule remove 2m
12791296
test "$(rbd trash purge schedule ls -R --format json)" = "[]"
12801297

@@ -1352,6 +1369,16 @@ test_mirror_snapshot_schedule() {
13521369
expect_fail rbd mirror snapshot schedule remove -p rbd2/ns1 --image test1 dummy
13531370
expect_fail rbd mirror snapshot schedule remove -p rbd2/ns1 --image test1 1h dummy
13541371

1372+
rbd mirror snapshot schedule add -p rbd2/ns1 1h 2100-01-01T19:00Z
1373+
test "$(rbd mirror snapshot schedule ls -p rbd2/ns1)" = 'every 1h starting at 2100-01-01 19:00:00'
1374+
for i in `seq 12`; do
1375+
rbd mirror snapshot schedule status -p rbd2/ns1 | grep '2100-01-01 19:00:00' && break
1376+
sleep 10
1377+
done
1378+
test "$(rbd mirror snapshot schedule status -p rbd2/ns1 --format xml |
1379+
xmlstarlet sel -t -v '//scheduled_images/image/schedule_time')" = '2100-01-01 19:00:00'
1380+
rbd mirror snapshot schedule rm -p rbd2/ns1
1381+
13551382
rbd mirror snapshot schedule add -p rbd2/ns1 --image test1 1m
13561383
expect_fail rbd mirror snapshot schedule ls
13571384
rbd mirror snapshot schedule ls -R | grep 'rbd2 *ns1 *test1 *every 1m'
@@ -1403,15 +1430,15 @@ test_mirror_snapshot_schedule() {
14031430
done
14041431
rbd mirror snapshot schedule status | grep 'rbd2/ns1/test1'
14051432

1406-
rbd mirror snapshot schedule add 1h 00:15
1407-
test "$(rbd mirror snapshot schedule ls)" = 'every 1h starting at 00:15:00'
1408-
rbd mirror snapshot schedule ls -R | grep 'every 1h starting at 00:15:00'
1433+
rbd mirror snapshot schedule add 1h 2020-01-14T04:30+05:30
1434+
test "$(rbd mirror snapshot schedule ls)" = 'every 1h starting at 2020-01-13 23:00:00'
1435+
rbd mirror snapshot schedule ls -R | grep 'every 1h starting at 2020-01-13 23:00:00'
14091436
rbd mirror snapshot schedule ls -R | grep 'rbd2 *ns1 *test1 *every 1m'
14101437
expect_fail rbd mirror snapshot schedule ls -p rbd2
1411-
rbd mirror snapshot schedule ls -p rbd2 -R | grep 'every 1h starting at 00:15:00'
1438+
rbd mirror snapshot schedule ls -p rbd2 -R | grep 'every 1h starting at 2020-01-13 23:00:00'
14121439
rbd mirror snapshot schedule ls -p rbd2 -R | grep 'rbd2 *ns1 *test1 *every 1m'
14131440
expect_fail rbd mirror snapshot schedule ls -p rbd2/ns1
1414-
rbd mirror snapshot schedule ls -p rbd2/ns1 -R | grep 'every 1h starting at 00:15:00'
1441+
rbd mirror snapshot schedule ls -p rbd2/ns1 -R | grep 'every 1h starting at 2020-01-13 23:00:00'
14151442
rbd mirror snapshot schedule ls -p rbd2/ns1 -R | grep 'rbd2 *ns1 *test1 *every 1m'
14161443
test "$(rbd mirror snapshot schedule ls -p rbd2/ns1 --image test1)" = 'every 1m'
14171444

@@ -1424,7 +1451,14 @@ test_mirror_snapshot_schedule() {
14241451
expect_fail rbd mirror snapshot schedule remove 1h dummy
14251452
expect_fail rbd mirror snapshot schedule remove -p rbd2/ns1 --image test1 dummy
14261453
expect_fail rbd mirror snapshot schedule remove -p rbd2/ns1 --image test1 1h dummy
1427-
test "$(rbd mirror snapshot schedule ls)" = 'every 1h starting at 00:15:00'
1454+
expect_fail rbd mirror snapshot schedule add 30m 04:30
1455+
expect_fail rbd mirror snapshot schedule add 30m 04:30+05:30
1456+
expect_fail rbd mirror snapshot schedule add 30m 2020-13-14T04:30+05:30
1457+
expect_fail rbd mirror snapshot schedule add 30m 2020-01-32T04:30+05:30
1458+
expect_fail rbd mirror snapshot schedule add 30m 2020-01-14T25:30+05:30
1459+
expect_fail rbd mirror snapshot schedule add 30m 2020-01-14T04:60+05:30
1460+
expect_fail rbd mirror snapshot schedule add 30m 2020-01-14T04:30+24:00
1461+
test "$(rbd mirror snapshot schedule ls)" = 'every 1h starting at 2020-01-13 23:00:00'
14281462
test "$(rbd mirror snapshot schedule ls -p rbd2/ns1 --image test1)" = 'every 1m'
14291463

14301464
rbd rm rbd2/ns1/test1

‎src/pybind/mgr/rbd_support/mirror_snapshot_schedule.py‎

Lines changed: 15 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
import rbd
55
import traceback
66

7-
from datetime import datetime
7+
from datetime import datetime, timezone
88
from threading import Condition, Lock, Thread
99
from typing import Any, Dict, List, NamedTuple, Optional, Set, Tuple, Union
1010

@@ -338,7 +338,7 @@ def __init__(self, module: Any) -> None:
338338
self.condition = Condition(self.lock)
339339
self.module = module
340340
self.log = module.log
341-
self.last_refresh_images = datetime(1970, 1, 1)
341+
self.last_refresh_images = datetime(1970, 1, 1, tzinfo=timezone.utc)
342342
self.create_snapshot_requests = CreateSnapshotRequests(self)
343343

344344
self.stop_thread = False
@@ -370,7 +370,7 @@ def run(self) -> None:
370370
pool_id, namespace, image_id = image_spec
371371
self.create_snapshot_requests.add(pool_id, namespace, image_id)
372372
with self.lock:
373-
self.enqueue(datetime.now(), pool_id, namespace, image_id)
373+
self.enqueue(datetime.now(timezone.utc), pool_id, namespace, image_id)
374374

375375
except (rados.ConnectionShutdown, rbd.ConnectionShutdown):
376376
self.log.exception("MirrorSnapshotScheduleHandler: client blocklisted")
@@ -381,7 +381,7 @@ def run(self) -> None:
381381

382382
def init_schedule_queue(self) -> None:
383383
# schedule_time => image_spec
384-
self.queue: Dict[str, List[ImageSpec]] = {}
384+
self.queue: Dict[datetime, List[ImageSpec]] = {}
385385
# pool_id => {namespace => image_id}
386386
self.images: Dict[str, Dict[str, Dict[str, str]]] = {}
387387
self.schedules = Schedules(self)
@@ -393,7 +393,7 @@ def load_schedules(self) -> None:
393393
self.schedules.load(namespace_validator, image_validator)
394394

395395
def refresh_images(self) -> float:
396-
elapsed = (datetime.now() - self.last_refresh_images).total_seconds()
396+
elapsed = (datetime.now(timezone.utc) - self.last_refresh_images).total_seconds()
397397
if elapsed < self.REFRESH_DELAY_SECONDS:
398398
return self.REFRESH_DELAY_SECONDS - elapsed
399399

@@ -405,7 +405,7 @@ def refresh_images(self) -> float:
405405
self.log.debug("MirrorSnapshotScheduleHandler: no schedules")
406406
self.images = {}
407407
self.queue = {}
408-
self.last_refresh_images = datetime.now()
408+
self.last_refresh_images = datetime.now(timezone.utc)
409409
return self.REFRESH_DELAY_SECONDS
410410

411411
images: Dict[str, Dict[str, Dict[str, str]]] = {}
@@ -421,7 +421,7 @@ def refresh_images(self) -> float:
421421
self.refresh_queue(images)
422422
self.images = images
423423

424-
self.last_refresh_images = datetime.now()
424+
self.last_refresh_images = datetime.now(timezone.utc)
425425
return self.REFRESH_DELAY_SECONDS
426426

427427
def load_pool_images(self,
@@ -473,13 +473,10 @@ def load_pool_images(self,
473473
pool_name, e))
474474

475475
def rebuild_queue(self) -> None:
476-
now = datetime.now()
477-
478476
# don't remove from queue "due" images
479-
now_string = datetime.strftime(now, "%Y-%m-%d %H:%M:00")
480-
477+
now = datetime.now(timezone.utc)
481478
for schedule_time in list(self.queue):
482-
if schedule_time > now_string:
479+
if schedule_time > now:
483480
del self.queue[schedule_time]
484481

485482
if not self.schedules:
@@ -494,7 +491,7 @@ def rebuild_queue(self) -> None:
494491

495492
def refresh_queue(self,
496493
current_images: Dict[str, Dict[str, Dict[str, str]]]) -> None:
497-
now = datetime.now()
494+
now = datetime.now(timezone.utc)
498495

499496
for pool_id in self.images:
500497
for namespace in self.images[pool_id]:
@@ -536,13 +533,11 @@ def dequeue(self) -> Tuple[Optional[ImageSpec], float]:
536533
if not self.queue:
537534
return None, 1000.0
538535

539-
now = datetime.now()
540-
schedule_time = sorted(self.queue)[0]
536+
now = datetime.now(timezone.utc)
537+
schedule_time = min(self.queue)
541538

542-
if datetime.strftime(now, "%Y-%m-%d %H:%M:%S") < schedule_time:
543-
wait_time = (datetime.strptime(schedule_time,
544-
"%Y-%m-%d %H:%M:%S") - now)
545-
return None, wait_time.total_seconds()
539+
if now < schedule_time:
540+
return None, (schedule_time - now).total_seconds()
546541

547542
images = self.queue[schedule_time]
548543
image = images.pop(0)
@@ -616,7 +611,7 @@ def status(self, level_spec: LevelSpec) -> Tuple[int, str, str]:
616611
continue
617612
image_name = self.images[pool_id][namespace][image_id]
618613
scheduled_images.append({
619-
'schedule_time': schedule_time,
614+
'schedule_time': schedule_time.strftime("%Y-%m-%d %H:%M:00"),
620615
'image': image_name
621616
})
622617
return 0, json.dumps({'scheduled_images': scheduled_images},

0 commit comments

Comments
 (0)