mon: answer a resent osd pool create once the pool exists - #73741

Implement completion messaging from OSD PGs to the monitor when pool
migration completes:

- Add MOSDPGMigratedPool message to notify OSDMonitor when a PG has finished
  migrating all objects to the target pool.
- Register MSG_OSD_PG_MIGRATED_POOL in msg infrastructure.
- Implement OSDService::send_pg_migrated_pool() helper.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Implement pool migration peering states and inter-OSD reservation messaging:

- Add MigratingSource, MigratingTarget, WaitLocalPoolMigrationReserved,
  WaitRemotePoolMigrationReserved, NotMigrating, and
  RepWaitPoolMigrationReserved states to PeeringState.
- Add MPoolMigrationReserve peering message to negotiate and track pool
  migration reservations across peer shards.
- Extend PeeringListener and PG interfaces with pool migration reservation
  and suspension callbacks, add HBHandle* parameter to on_activate_complete
  and on_activate_committed, and add num_objects parameter to
  try_reserve_recovery_space.
- Add OSD dispatching and perf counters for pool migration states and
  stats invalidation (rs_stats_invalidated).
- Increase BOOST_MPL_LIMIT_LIST_SIZE from 30 to 40 in CMakeLists.txt to
  accommodate expanded PeeringState reaction lists.
- Add last_event_desc and last_exited_state diagnostic recording to
  PeeringState Crashed state handler.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>
Co-authored-by: Connor Fawcett <connorfa@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Update Crimson OSD and PG interfaces to support pool migration:

- Add stub implementations for Crimson PeeringListener and PGBackend interface
  methods for pool migration events and watermark tracking.
- Add MOSDPGMigratedPool dispatch and sending support in Crimson shard_services.
- Add stub handlers in Crimson PGRecovery for pool migration reservation states.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>
Co-authored-by: Tom Sollers <tom.sollers@ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Add the protocol by which a source PG reserves space on a target PG
before copying objects:

- Add CEPH_OSD_OP_PG_POOL_MIGRATION_RESERVE/RELEASE and the
  pool_migration_reserve (num_bytes, num_objects) op payload in rados.h.
- Source: request a reservation from the target PG, sized by the number
  of remaining target PGs. Retry after a delay on -EBUSY (stale epoch),
  stop the migration on -ENOSPC or any other error, and send a release
  when finished. A generation counter invalidates stale async callbacks.
- Target: handle RESERVE/RELEASE in do_pg_op, hold the request until the
  local reservation is granted and then reply, and fail queued requests
  if the reservation is suspended. A granted reservation removes stale
  objects left by an interrupted migration (pool_migration_target_delete).
- Add helpers to map an object hash to its source and target PG and to
  count the remaining target PGs.
- Cancel or clear reservations in on_change and on_shutdown.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>
Co-authored-by: Tom Sollers <tom.sollers@ibm.com>
Co-authored-by: Connor Fawcett <connorfa@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Add the source-side code that runs when a pool migration copy finishes
or fails, and the scanning that selects objects to migrate. Nothing in
this commit starts a copy: recover_pool_migration() is a stub, and the
copy_from itself is added by the object copy engine commit.

- Add C_Migrate, the completion for a migration copy_from. On success it
  deletes the object from the source pool
  (pool_migration_source_start_delete[_head], pool_migration_source_delete),
  deleting a head only after all of its clones. While quiescing, every
  completion is treated as a failure so that in-flight copies drain.
- Add handle_pool_migration_copy_failure, which enters quiesce mode for
  retryable (-EBUSY) and fatal (-ENOENT, -EIO, -EPERM) errors, and
  stop_pool_migration_unfound/error and on_pool_migration_source_suspended
  to report the result once the quiesce completes.
- Add scan_range_migration and update_range(PoolMigrationInterval*) to
  list the objects in the current migration range and keep the list up
  to date from the PG log.
- Add a stub recover_pool_migration and hook it into start_recovery_ops,
  including PG_STATE_MIGRATING handling and the DoPoolMigration and
  PoolMigrationDone peering events. DoPoolMigration is also queued after
  recovery and backfill complete and after mark_all_unfound_lost.
- In do_op_impl for a migration source pool, fail clients that lack
  CEPH_FEATUREMASK_SERVER_VAMPIRE with -EIO, and redirect ops for
  objects below the migration watermark to the target pool.
- Treat the watermark object (at the start of a new interval) and
  in-flight migration objects as degraded.
- Fail a CEPH_OSD_COPY_FROM_FLAG_POOL_MIGRATION copy_from with -EBUSY
  unless the target holds reservations.
- Only drop the recovery read in _clear_recovery_state when the
  recovery_read_marker is set.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>
Co-authored-by: Tom Sollers <tom.sollers@ibm.com>
Co-authored-by: Connor Fawcett <connorfa@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Add backend support in ECBackend and ReplicatedBackend for migration watermarks:

- Introduce PoolMigrationInterval (in recovery_types.h) to track migration
  watermark, object ranges, and progress across peering intervals.
- Implement consider_updating_migration_watermark() across backend pipelines to
  advance watermarks as objects are modified or deleted.
- Propagate watermark updates across MOSDRepOp (bumped to v4) and ECSubWrite
  (bumped to v6) messages to replicas.
- Add cancel_recovery_op() to PGBackend interface and backend implementations
  (ECBackend, ECBackendL, ReplicatedBackend, ECSwitch) to keep in-flight
  recovery ops synchronized with recovery cancellations.
- Add OI_ATTR existence check in ReplicatedBackend calc_head_subsets and
  calc_clone_subsets.
- Add fmt formatter support for PoolMigrationInterval in recovery_types.h.
- Ensure backend recovery and clone trims remain synchronized with active
  migration ranges.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Connor Fawcett <connorfa@uk.ibm.com>
Co-authored-by: Tom Sollers <tom.sollers@ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Add the watermark and range-scan logic that decides which objects a
source PG migrates next:

- Add next_pool_migration() and earliest_pool_migration() to find the
  lowest-hash object still to migrate, and
  consider_updating_migration_watermark() to advance the watermark past
  objects that have been migrated or deleted.
- Rework scan_range_migration() and update_range(PoolMigrationInterval*)
  to scan from the watermark, and to leave alone log updates for objects
  below it.
- Split on_activate_complete() into _on_activate_committed(), which
  every shard runs, and the primary-only on_activate_complete(). Both
  now take an HBHandle, and on_activate_committed() is added.
  _on_activate_committed() scans the first migration range, merges
  missing objects into it, sets the watermark from earliest_pool_migration()
  and builds the target PG list (initialize_pool_migration_target_pg_list).
- In on_activate_complete(), queue DoPoolMigration when the PG needs
  migration and nothing else (recovery or backfill) is pending.
- Make recover_pool_migration() initialise the migration interval, and
  keep it up to date with update_range() before choosing the next object.
- Erase an object from pool_migrations_in_flight when it is recovered,
  and assert that none are left in _clear_recovery_state().
- Add the on_pool_migration_source_starting() override.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Connor Fawcett <connorfa@uk.ibm.com>
Co-authored-by: Tom Sollers <tom.sollers@ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Implement the core object copy engine for pool migration in PrimaryLogPG:

- Implement migration_copy_from_check() and cancel_pool_migration_copy_ops()
  to transfer object data, attributes, and omap to the target pool.
- Handle copy-from completion, failure retries, and out-of-order copies.
- Support migrating whiteout head objects (adding whiteout flag to
  object_copy_data_t, creating whiteouts on target, updating whiteout stats,
  and passing pool_migration flag to _delete_oid).
- Add CEPH_OSD_COPY_FROM_FLAG_POOL_MIGRATION flag in rados.h.
- Update PGMap with pool migration throughput and object statistics rate summaries.
- Update SplitOp target calculation for objecter copy-from operations.
- Add osd_debug_inject_copyfrom_single_error config option for error injection.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Connor Fawcett <connorfa@uk.ibm.com>
Co-authored-by: Bill Scales <bill_scales@uk.ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Tom Sollers <tom.sollers@ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Add snapshot context, clone, and watcher migration support to PrimaryLogPG:

- Preserve clone snapsets and snapshot metadata during migration, mirroring
  snapsets to the target pool.
- Source the pool snap context and pool-snaps mode from the migration-chain
  root pool (PGPool), so clone-on-write during migration uses the root's
  snapc, and honour MAP_SNAP_CLONE in execute_ctx to keep the source
  object's snapset.seq instead of fast-forwarding it to the pool seq.
- Refactor trim_object() into add_trim_to_ctx() to allow applying clone trims
  directly to an existing OpContext during migration copy and clone processing.
- Handle clone trimming, pruned clone cleanup, and snapset rebuilds during
  in-flight object copies.
- Add flags2 field to ceph_osd_op.copy_from and define
  CEPH_OSD_COPY_FROM_FLAG_POOL_MIGRATION_HAS_RES and
  CEPH_OSD_COPY_FROM_FLAG2_POOL_MIGRATION_HAS_CLONES in rados.h.
- Migrate registered object watchers to the target pool.
- Update scrubber to tolerate intermediate snapshot states and incomplete clones
  on migration target pools.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Connor Fawcett <connorfa@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Extend the quiesce mechanism (which until now only handled copy
failures) so that in-flight migration copies are drained when the
migration is suspended or the PG re-peers, rather than being discarded.

- Add two quiesce reasons, SUSPEND_NEEDED and NEW_INTERVAL, and the
  pool_migration_quiesce_last_started_reset flag.
  on_pool_migration_source_suspended() sets SUSPEND_NEEDED if copies are
  still in flight, and finishes the suspension once they have drained.
- on_change(): instead of always clearing the quiesce state, set
  NEW_INTERVAL when copies from the previous interval are still in
  flight. An in-progress FATAL_ERROR, RETRY_NEEDED or SUSPEND_NEEDED
  quiesce is preserved.
- _clear_recovery_state(): keep the migrations that still have an
  outstanding copy so they can drain, drop deferred ones that have no
  outstanding op (a pending source delete, or a head waiting for its
  clones), and no longer assert that the in-flight set is empty.
- _on_activate_committed(): while a quiesce is active keep
  pool_migrations_in_flight and re-scan the interval; if everything has
  already drained during teardown, complete the quiesce there. Stop
  resetting the pool_migration_reservations_granted and
  pool_migration_waiting_for_reservations members.
- handle_pool_migration_quiesce_complete(): replace the if/else with a
  switch over all reasons. RETRY_NEEDED clears
  pool_migration_reservations_granted_source before requesting a new
  reservation. SUSPEND_NEEDED rewinds last_pool_migration_started to the
  watermark and completes the suspension. NEW_INTERVAL restarts migration
  with queue_recovery(). The FATAL_ERROR handling is unchanged apart from
  the re-indent.
- C_Migrate::finish() and the source delete completion: complete the
  quiesce when the last in-flight head is removed, and drop objects from
  pool_migrations_in_flight when a source delete finishes after a peering
  reset.
- handle_pool_migration_copy_failure() no longer returns a bool, because
  the caller never needed the result. It also takes a `stale` flag: a
  completion from a previous interval drains the migration accounting but
  must not drop the recovery read lock (which now belongs to recovery after
  the interval change), avoiding a leaked read lock on a clone.
- on_pool_migration_source_suspended(): flush ownerless parked deletes from
  pool_migrations_in_flight and rewind last_pool_migration_started to the
  watermark, so the SUSPEND quiesce can drain instead of hanging.
- recover_pool_migration(): log when quiescing.

Also remove the duplicate declarations of earliest_pool_migration(),
update_migration_watermark() and consider_updating_migration_watermark(),
and of on_pool_migration_source_starting() and
on_pool_migration_source_reserved(), left over from earlier commits, and
a stray blank line in do_op_impl().

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Implement client-side I/O redirection and transparent object listing in Objecter:

- Redirect client reads and writes between source and target pools based on the
  migration watermark and target pool configuration.
- Support cascaded/multi-hop migrations using migration_root.
- Transparently merge object listings and namespace enumerations across migrating
  pools.
- Redirect snapshot-metadata lookups (pool_snap_by_name, pool_snap_get_info
  and pool_snap_list in Objecter, and the equivalents in the neorados RADOS
  client) to the migration-chain root pool.
- Update Client and MDS utility helpers for migration pool lookups.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Tom Sollers <tom.sollers@ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

…olicing, snap/delete routing, listing, status)

Implement the pool migration control plane and scheduling logic in OSDMonitor:

- Add 'ceph osd pool migrate' CLI command with full pool creation options
  and --enable_ec_optimizations.
- Add --show-all option to 'ceph osd pool ls' to list migration source pools.
- Implement migration scheduling and throttle concurrency via
  mon_pool_migration_max_pg_ratio.
- Add policing to prevent pool resize, split, or merge operations during migration.
- Add POOL MIGRATION cluster health warning in PGMap reporting migration progress.
- Process MOSDPGMigratedPool messages and track per-PG completion in PGMap.
- Correctly route snapshot creation, deletion, and pool removal across cascades.
- Filter out migration source pools in RadosClient::pool_list.
- Skip pools with FLAG_NOPGCHANGE in DaemonServer::adjust_pgs autoscaler check.
- Include <cls/sem_set/ops.h> in PGMap.cc.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Connor Fawcett <connorfa@uk.ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Tom Sollers <tom.sollers@ibm.com>
Co-authored-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Add test support for pool migration. The teuthology configuration is
disabled for now (see below), so only the standalone test and
ceph_test_rados change are exercised by default.

ceph_test_rados:
- Add --migrate-pool, --migration-enable-ec-optimizations,
  --initial-migration-delay, --migration-interval and
  --migration-pg-count (TestRados.cc, qa/tasks/rados.py). With
  --migrate-pool, RadosTestContext runs a thread that repeatedly issues
  "osd pool migrate" for the test pool (optionally alternating pg_num),
  waits for the migration to finish, then sleeps for the migration
  interval. Link ceph_test_rados with json_structures for the request
  encoding.
- Replace the std::cerr progress output in RadosModel.h with a LogHelper
  base class, so that output (including TestWatchContext) carries the
  optional timestamp, and list the in-flight op numbers when waiting.
- Fix the watch handling in the ops: on a watched object a rollback now
  marks itself done, a read asserts that the head object exists (not the
  snapshot being read), and the watch callback acks the notify.

Standalone test:
- Add TEST_osd_pool_create_migrate_from to osd-pool-create.sh. It
  migrates an EC pool that has been merged down to 2 PGs and checks that
  pg_num changes on both the source (.migrate-1) and target pools are
  blocked by the mon (nopgchange) and skipped by the mgr.

Teuthology:
- Add qa/pool-migration/ with enabled.yaml and three "none" fragments,
  and link it as a pool-migration facet into the rados basic, singleton,
  thrash and thrash-erasure-code suites and the squid-x parallel upgrade
  suite. enabled.yaml is entirely commented out ("Disabled until pool
  migration is stable"), so no suite runs a migration yet.
- ceph_manager.py: add wait_for_clean(wait_for_migration=...). PGs that
  are migrating, migration_wait or migration_toofull count as clean
  when it is False (the thrashers now pass False), and when it is True,
  migration progress or a paused migration resets the timeout. Add
  helpers for these checks, and compare PG state flags rather than the
  exact string "active+clean". Re-indent several nearby log blocks.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Bill Scales <bill_scales@uk.ibm.com>
Co-authored-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>
Co-authored-by: Tom Sollers <tom.sollers@ibm.com>
Co-authored-by: Jamie Pryde <jamiepry@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Add and update unit tests for pool migration control plane and peering state machine:

- Add test_osd_monitor.cc covering CalcMigratingPgCount* and TargetPgMigrating*
  cases for migration scheduling calculations.
- Update TestPeeringState.cc by removing POOL_MIGRATION conditional guards and
  adding test cases for migration peering state transitions, target reservation
  preemption, and retries.
- Add unit tests in test/librados/list.cc for object listing during migration
  (PoolMigrationRadosLSBasic, PoolMigrationRadosLSMultiple).
- Add MockPeeringListener and MockPGBackend infrastructure for migration
  callback testing.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Bill Scales <bill_scales@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Expose pool migration functionality via the Ceph Manager dashboard REST API:

- Add pool migration endpoint (POST /api/pool/{pool_name}/migrate) to the
  dashboard Pool controller.
- Update OpenAPI specification with migration endpoint schema and parameters.
- Add unit test coverage in test_pool.py.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

Add documentation and update peering state diagrams for pool migration:

- pools.rst: add a "Migrating a Pool" section documenting the
  `ceph osd pool migrate` command and its options. Document the
  `--show-all` flag for `ceph osd pool ls`, which shows hidden pools
  (including .migrate-<n> source pools). Document --enable-ec-optimizations
  and the `allow_ec_optimizations` pool flag.
- ceph.rst (man page): add the `ceph osd pool migrate` subcommand syntax
  and `--show-all` to the `osd pool ls` synopsis.
- ceph_test_rados.rst: document all five pool migration flags
  (--migrate-pool, --migration-enable-ec-optimizations,
  --initial-migration-delay, --migration-interval, --migration-pg-count)
  with a usage example.
- Update gen_state_diagram.py to deduplicate and sort state-transition
  edges so that multiple labels on the same edge are merged rather than
  appearing as separate arrows. Regenerate peering_graph.generated.svg
  to reflect the new pool migration peering states.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Co-authored-by: Connor Fawcett <connorfa@uk.ibm.com>
Assisted-by: Claude Opus 4.8
Assisted-by: IBM Bob 2.3.0

…ers when running ceph_test_rados

Signed-off-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>

Ensure the feature-gated field 'migration_watermark' survives encode/decode

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>

Pool migration endpoints uses a mixed value type dict, so
use cmd_args: Dict[str, Any] = {}

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>

A create of a pool that was still pending answered "already exists" at
once. A client that resent its create got that answer before the pool was
committed, and its next command did not find the pool.

Such a create now waits for the pending update and runs again, as on main.

Fixes: https://tracker.ceph.com/issues/82290
Assisted-by: Claude Code