mon,osd,osdc: EC stretch cluster (stretch pools with num_zones) - #72643

A pool can be stretched over num_zones zones. Each zone holds replica
copies of a replicated pool, or a whole set of k+m shards of an erasure
coded pool, so the pool size is num_zones times the copies per zone.
min_size is kept per zone.

Add num_zones and replica to pg_pool_t in struct version 34. Like
version 33 it is only encoded for peers with SERVER_UMBRELLA; others get
version 32, in which min_size counts every zone, so that encoding writes
min_size times num_zones. A pool decoded from an older version has one
zone, replica equal to size and its min_size unchanged. A replicated
pool with stretch values is also marked legacy_stretch, because only
the map knows whether it belongs to global stretch mode:
OSDMap::settle_legacy_stretch_pools(), run when a map is decoded and
when an incremental is applied, gives it num_zones
peering_crush_bucket_target, replica size / num_zones and min_size
divided by num_zones, rounded up, in a map in stretch mode. Otherwise
the pool keeps one zone, so that an individual stretch pool behaves
exactly as in earlier releases. Pool dumps show both fields.

Add get_zone_size(), get_relative_shard(), which maps a shard to its
position within a zone, and stripe_info_t::get_num_zones(). Without
ec_data_shard_count, get_ec_data_shard_count() now counts the
non-primary shards of one zone only.

The new PgPoolStretchEncodingTest and
get_ec_data_shard_count_multi_zone_fallback tests in
src/test/osd/types.cc cover the encoding and the fallback, and
OSDMapTest.LegacyStretchPoolDecode decodes a map and an incremental in
the earlier format, with an individual three-zone pool and with a pool
of global stretch mode.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

The pg_temp of an optimized EC pool lists the primary-capable shards
first. The vector forms of pgtemp_primaryfirst() and
pgtemp_undo_primaryfirst() do this with a stable partition of the whole
acting set by nonprimary_shards, but the per-shard
pgtemp_undo_primaryfirst(), which the Objecter, the OSD and PeeringState
use to find the shard of the acting primary, computed the shard
arithmetically. That assumed a single zone whose non-primary shards are
1 to k-1.

Walk the shards and count the primary-capable and non-primary ones
instead, so that the per-shard form matches the vector form for any set
of non-primary shards, including a pool whose zones each repeat the k+m
shards. The result is unchanged for a single zone with non-primary
shards 1 to k-1. A position beyond the pool size now aborts. Remove the
per-shard pgtemp_primaryfirst(), which only tests used.

TestOSDMap.cc checks the per-shard form against the vector form, runs
pgtemp_primaryfirst_comprehensive for one to three zones and adds
pgtemp_primaryfirst_stretch_none_holes for pg_temp with
CRUSH_ITEM_NONE holes over several zones.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

An erasure coded pool over several zones holds the same k+m shards in
each zone, so a transaction built for a shard of one zone can be sent
to the matching shard of every other zone, but each copy must name its
own shard.

Add Transaction::remap_shard(), which sets the shard of every object in
the transaction's object table, and of every PG and PG temp collection
in its collection table, to the given shard. Ops and data refer to
these tables by index, so they are unchanged. Other collections keep
their names.

The new Transaction.RemapShard test in test_transaction.cc checks a
remapped copy and that the original is unchanged.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

CrushWrapper::add_simple_stretch_rule() builds a rule that takes a root,
chooses num_failure_domains buckets of the zone type (choose firstn),
then num_replica_per_zone buckets of the OSD failure domain type in each
zone (chooseleaf firstn or indep, or choose when that type is osd, as
add_simple_rule_at() does: chooseleaf of type osd can return an OSD that
is out), so each zone's OSDs form a contiguous block of the result.
Unless forced, it fails when the root does not have exactly that many
zones, or a zone has fewer OSD failure domains with an OSD than the
copies per zone. The free rule id search and device class lookup of
add_simple_rule_at() move into helpers that it shares.

ErasureCodeInterface::create_rule() gains an overload that takes the
number of zones, root, zone and OSD failure domains and device class;
the old form calls it with one zone. For more than one zone ErasureCode
builds an indep stretch rule with k+m OSDs per zone, taking any empty
argument from the profile. For one zone the profile alone decides the
rule, as before. LRC refuses more than one zone.

The profile gains crush-zone-failure-domain (default datacenter),
crush-osd-failure-domain and crush-num-osd-failure-domains. The last
two fall back to crush-failure-domain and crush-num-failure-domains.
Like the other defaults, crush-zone-failure-domain=datacenter is stored
in every new profile. OSDECProfileGetReply uses the new key names.

CrushWrapper::get_rule_placement() reads the placement back from a rule
of the shapes that pools are created with, or from one that takes each
zone in turn: its root, device class, zone type and count, and OSD
failure domain and count. New CrushWrapper tests stretch_replica,
stretch_ec, stretch_ec_three_zones_device_class and rule_placement_*
cover the rules, their checks and the reading of their placement.

Assisted-by: Claude Code
Signed-off-by: Eric Zhang <emzhang@ibm.com>
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

For a stretch pool (peering_crush_bucket_count set), replicated or
erasure coded, min_size now applies to each zone.

Add OSDMap::get_stretch_zones(), which finds the zones below the take
steps of the pool's CRUSH rule that hold an OSD, and
num_acting_below_min_size(), which adds up how far each zone's part of
an acting set falls short of min_size; for any other pool it counts the
acting set's OSDs against min_size. In degraded stretch mode only the
mandatory zone counts. A pool with three zones and
peering_crush_bucket_count 2 is instead checked as a whole against
min_size times three. OSDMap::acting_set_writeable() adds
stretch_set_can_peer(), which is true for other pools.

PeeringState uses the shortfall in recoverable() and the recovery and
backfill priorities, and acting_set_writeable() in
PeeringState::acting_set_writeable(), the choice of async recovery
targets and check_new_interval(), so a stretch pool is judged by its
zones alone everywhere, and callers no longer branch on
is_stretch_pool(). An Incomplete PG resets when
peering_crush_bucket_count drops. The manager's ok-to-stop and
ok-to-upgrade checks use the shortfall too, and so does the monitor's
prime_pg_temp() for stretch pools; for other pools it keeps its own
count, which counts the holes of an erasure coded acting set. The
monitor no longer changes min_size when stretch mode enters or leaves
degraded mode.

choose_async_recovery_ec() now lowers its count of wanted shards only
when it removes one, for all pools. Zones of a CRUSH rule with a device
class are counted by their real bucket, through the new
CrushWrapper::get_non_shadow_id(), here and in stretch_set_can_peer()
and calc_replicated_acting_stretch().

The new unittest_stretch_ec_min_size (TestStretchECMinSize.cc) tests the
helpers, including rules with a device class, and stretch_set_can_peer()
with a device class rule.

Assisted-by: Claude Code
Signed-off-by: Eric Zhang <emzhang@ibm.com>
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

OSDMonitor::crush_rule_create_replica() builds a simple rule for one
zone and, with CrushWrapper::add_simple_stretch_rule(), a stretch rule
for more. ceph osd crush rule create-replicated and the rules generated
for pools over zones share it. An empty zone failure domain
defaults to the new osd_pool_default_zone_failure_domain option
(datacenter). The new osd_pool_default_num_zones option (1) is not read
yet. When pool creation asks for a rule, the handling of a rule that
already exists or is pending moves to
handle_crush_rule_creation_result().

Assisted-by: Claude Code
Signed-off-by: Eric Zhang <emzhang@ibm.com>
Signed-off-by: Callum James <callum.james@ibm.com>
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

ceph osd pool create takes --num_zones, --replica (copies per zone),
--min_size, --k, --m, --root, --zone_failure_domain,
--osd_failure_domain and --class. A replicated pool over num_zones zones
gets size num_zones * replica, an erasure coded pool num_zones * (k+m),
and min_size counts the copies or shards in one zone. num_zones must be
1 or 2, and a PG has at most 128 OSDs.

ceph osd pool create calls the rule and the autoscale mode crush_rule
and pg_autoscale_mode, as ceph osd pool set does, and still accepts rule
and autoscale_mode, positional or named, as ceph osd pool set accepts
them as property names. A parameter given under both names is refused.

The new PoolCreateParams builds a pool's parameters in four steps: the
osd_pool_default_* options, the erasure code profile (the named one or
the default), the command line, then checks of the result. Which
parameters may be given together is checked on the command line; the
values are checked against the cluster: CRUSH types, roots and device
classes, the plugin, pg_num, crimson and, for two zones, whether the
monitors and the CRUSH map allow stretch mode. An error caused by a
default names its option. Pool creation through librados checks the
defaults in the same way.

--k and --m create the profile <pool>-k<k>-m<m> from
osd_pool_default_erasure_code_profile and the CRUSH parameters given,
and the pool's rule is generated from it; an existing profile of that
name with another k or m is refused. The monitor passes the zone count,
root, failure domains and device class to the plugin's create_rule(),
and ceph osd crush rule create-erasure takes --num_zones. An erasure
coded pool over two zones must be FastEC, and
ECUtil::nonprimary_shards() marks its non-primary shards in every zone.

A profile whose stripe_unit is not a multiple of 4096 needs
--yes-i-really-mean-it, and only a pool in one zone can use one, as a
pool over zones is FastEC. Without --num_zones or --replica, a --size of
a replicated pool over the default zones that is a multiple of
osd_pool_default_num_zones gives size / num_zones replicas, because
tools such as Rook pass --size in stretch mode.
Deleting an erasure coded pool now also deletes its profile, unless
another pool uses it or it is "default". Creating a two-zone pool checks
that stretch mode could be enabled, but does not enable it.

New options osd_pool_default_root, osd_pool_default_osd_failure_domain
and osd_pool_default_class. osd_pool_default_size now counts the
replicas in each zone; it keeps its name for backward compatibility.
The TOO_FEW_OSDS health check, the mgr's check for enough OSDs and the
rook module take a new pool's size to be osd_pool_default_num_zones
times osd_pool_default_size.

ceph osd pool get shows replica, and num_zones with "all". ceph osd pool
set refuses a new size for a pool over several zones, resets min_size to
the default for the new size, as on main, and checks an erasure coded
pool's min_size against k to k+m.

Adds unittest_pool_create_params, nonprimary_shards tests in
unittest_ecutil, test_default_pool_size.py,
ceph_test_rados_api_ec_pool_create_pp and test_pool_defaults.cc, which
this commit does not build.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Callum James <callum.james@ibm.com>
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
Signed-off-by: Eric Zhang <emzhang@ibm.com>

generate_entity_map() records the source of each value it returns, but
kept the first section that set the option (global) rather than the
most specific one that gives the value (mon.a over mon over global).
ceph config get <who> printed that section next to a value it did not
set, and ceph osd pool default get, which reports where each pool
creation default comes from, needs the right one.

Test ConfigMap.result_sources.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

ceph osd pool default set sets the defaults that ceph osd pool create
uses. It takes the pool create parameters that describe every pool:
--pool_type, --num_zones, --crush_rule, --root, --zone_failure_domain,
--osd_failure_domain, --class, --replica, --min_size,
--erasure_code_profile or --k and --m, --pg_num, --pgp_num,
--pg_autoscale_mode, --bulk and --crimson. Each is written to its
osd_pool_default_* option in the global section of the configuration
database, --replica to osd_pool_default_size, which counts the copies in
each zone: --erasure_code_profile copies the named profile, --k and --m
change the default profile, and --crush_rule none returns to generated
rules. --rule and --autoscale_mode are still accepted, and
ceph osd pool default get shows the new names.

Nothing is written unless the checks that ceph osd pool create would
make with the resulting defaults pass, for the default pool type and for
each type whose parameters are given; --replica 1 also needs
mon_allow_pool_size_one and --yes-i-really-mean-it. The command fails if
a mon or mon.<id> section, a masked global value or the monitor's local
configuration would override a value it writes, and it is refused until
require_osd_release reaches vampire, the release with per-pool
num_zones.

ceph osd pool default get shows each default with its value, option and
source, as a table or with -f json. The POOL_DEFAULT_CONFIG_OVERRIDE health
warning reports a pool default with a global value that the
configuration database overrides for a monitor, or that a monitor's
local configuration sets to another value.

ceph osd pool create gives an erasure coded pool, or a replicated pool
over two zones, that has neither --crush_rule nor a CRUSH parameter the rule
osd_pool_default_crush_rule if it is of the pool's type and, for two
zones, divides the cluster into the pool's zones as the existing stretch
pools do. A two-zone replicated pool takes its replicas per zone and
zone failure domain from the defaults too.

ConfigMap::generate_entity_map() reports the most specific section that
sets a value, which the section column of ceph config get <who> shows.

Adds unittest_pool_defaults, tests of the new helpers to
unittest_pool_create_params, and ConfigMap.result_sources.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Eric Zhang <emzhang@ibm.com>
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

ceph osd pool set <pool> num_zones <n> moves a pool between one and two
zones. It takes the optional arguments --replica, --crush_rule,
--zone_failure_domain, --root, --osd_failure_domain, --class and
--no-class, which every other variable refuses. A generated rule has the
placement of the pool's current rule, which
CrushWrapper::get_rule_placement() reads, with the options given over
it; if that rule cannot be read, the command needs --crush_rule or all
of --root, --osd_failure_domain and --class or --no-class. The zone type
is --zone_failure_domain, else that of the pool's rule over zones, else
osd_pool_default_zone_failure_domain. It is refused until
require_osd_release reaches the release with per-pool num_zones.

Going to two zones, a replicated pool gets num_zones * replica copies
and --crush_rule or a generated stretch rule. An erasure coded pool must
be FastEC; it gets --crush_rule or the rule <pool>-stretch, generated
from that placement and its profile's k+m, and non-primary shards in
both zones. The pool gets the default per-zone min_size and the
peering_crush_bucket_* values for two zones. If stretch mode is enabled,
the pool's rule must divide the cluster as stretch mode does; otherwise
the pool enables stretch mode in the MonMap and the OSDMap. Going to one
zone, a replicated pool gets --replica or osd_pool_default_replica
copies and --crush_rule, or the default replicated rule if the placement
is the cluster default, else the rule <pool>-single-zone, an erasure
coded pool k+m shards and the rule <pool>-single-zone; when no stretched
pool is left, stretch mode is disabled. An erasure coded PG keeps only
the shards of its first zone, so num_zones 1 on an erasure coded pool
needs --yes-i-really-mean-it; its refusal names the section of the
documentation on when PGs become unavailable and suggests pool migration
instead. ceph osd pool set runs with Paxos plugged, so a MonMap change
commits with the pool change.

ceph osd pool set <pool> replica <n> sets a replicated pool's copies per
zone and its size to num_zones * n, and resets min_size to the default
for n, as ceph osd pool set <pool> size does. For a pool over two zones
it creates the rule <pool>-replica-<n> with the placement of the old
rule, its device class included, and refuses a rule whose placement
cannot be read.

ceph pg repeer and ceph osd pg-temp size the pg_temp they set by the
pool as the pending update leaves it, so that a size change earlier in
the same update cannot leave a pg_temp of the old size. When the padded
pg_temp a repeer would set equals the PG's acting set, the pg_temp of an
earlier repeer that the primary may already have asked to remove, the
repeer removes it instead, so that the acting set changes. Otherwise a
second repeer of an erasure coded PG could replace the primary's pending
removal and leave the PG waiting for an acting set change that never
comes; this also happens on main since a3cc500.

A rule that a pool stops using is removed when no other pool uses it and
it is not the default replicated rule. Deleting a pool now removes a
replicated pool's unused rule too, not only an erasure coded pool's.

Adds num_zones tests to ceph_test_rados_api_ec_pool_create_pp.

Assisted-by: Claude Code
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Callum James <callum.james@ibm.com>

Once require_osd_release reaches the release with per-pool num_zones,
ceph mon enable_stretch_mode and disable_stretch_mode go to OSDMonitor,
which changes every pool as ceph osd pool set num_zones does and sets
the pool creation defaults. enable_stretch_mode sets
osd_pool_default_num_zones 2, osd_pool_default_size 2 and
osd_pool_default_zone_failure_domain to the dividing bucket and
stretches every local pool; replicated pools, and stretched replicated
pools with another replica count, get 2 replicas per zone and
new_crush_rule, or without it a rule generated for each pool: <pool>, as
ceph osd pool set <pool> num_zones 2 generates it, or <pool>-replica-2
for a pool that already has two zones. It is refused if a legacy EC pool
exists or pool creation would fail with the new defaults.
disable_stretch_mode sets one zone and 3 replicas and unstretches every
pool over zones; it is refused while stretch mode recovers. Generated
rules are committed first, then the MonMap, configuration and OSDMap
change in one Paxos round. Both commands can be run again. Before the
upgrade commit they work as in earlier releases and write the same
values, so that OSDs of both releases need the same number of copies.

With no pools, enable_stretch_mode puts only the MonMap in stretch mode
and sets the pool creation defaults; the first pool over two zones then
sets the OSDMap's stretch flags. That pool's zone type must be the one
the tiebreaker was chosen for.

Creating a two-zone pool enables stretch mode once every check has
passed, so a failed create changes nothing. A pool created or stretched
while stretch mode is degraded or recovering joins that state. A
single-zone pool created in stretch mode stays local, and pool creation
no longer uses mon_stretch_pool_size or refuses erasure coded pools in
stretch mode. A replicated pool without --crush_rule gets a generated stretch
rule for two zones. After the upgrade commit, deleting the last
stretched pool disables stretch mode, with the MonMap and OSDMap changes
in one Paxos round.

A single-zone pool honours --root, --osd_failure_domain and --class and
their osd_pool_default_* defaults, when it is created and when it is
reduced to one zone. A replicated pool gets the default replicated rule
if that places it as asked, or if nothing asks for a placement other
than the stock one, as in earlier releases, and otherwise a generated
<pool>-single-zone rule. The default replicated rule is used as it is,
so a pool on a hand-written rule that reads as the stock placement can
end on it; the Advanced Options of stretch-mode.rst say so, and that
--root, --osd_failure_domain or --class keep the pool's placement.
--no-class forces no device class: the rule
builders take an optional class, which unset inherits an erasure code
profile's crush-device-class and empty drops.

The degraded, recovery and healthy transitions change every stretched
pool, including pools created or changed in the pending OSDMap, and the
monitors skip them once the MonMap leaves stretch mode. A pool whose
min_size counts all its copies, as before the upgrade commit, is halved
and restored as in earlier releases. At the upgrade commit,
convert_stretch_pools_at_commit() puts each pool of global stretch mode
whose copies divide evenly over two zones in two zones, with a per-zone
min_size.

Any other pool of global stretch mode, for example one of size 3, keeps
one zone and the rules of earlier releases.
STRETCH_MODE_POOL_NOT_CONVERTED names such pools before the commit,
while they can still be changed, and after it. After the commit,
disable_stretch_mode and ceph osd pool set <pool> crush_rule are refused
for them, as either would leave them peering over both zones; ceph osd
pool set <pool> num_zones 2 gives such a pool two zones, and num_zones 1
makes it an ordinary pool in one zone with 3 replicas, as a disable
does.

ceph osd pool stretch set refuses an erasure coded pool with EOPNOTSUPP
and points to ceph osd pool set num_zones. Otherwise it makes an
individual stretch pool as on main: the pool keeps one zone, with
replica equal to size and min_size the total given, from 1 to size;
--replica gives the copies in each zone that the rule spans. ceph osd
pool stretch unset refuses an erasure coded pool over several zones.
While an individual stretch pool exists, stretch mode is refused with a
pointer to ceph osd pool stretch unset, and ceph osd pool stretch set
and unset are refused while the MonMap or the OSDMap is in stretch mode.

ceph osd pool get takes num_zones, and ceph status no longer shows
"stretch_mode: ENABLED".

Adds unittest_osd_monitor and a MonMap encoding test.

The default erasure code profile is the template of erasure coded pools,
as the osd_pool_default_* options are of replicated pools: it is given
the placement keys it lacks from the options when it is created and on
ceph osd erasure-code-profile set default, and while the default type is
erasure ceph osd pool default set writes --root, --zone_failure_domain,
--osd_failure_domain, --class and --no-class into it instead of into the
options. Erasure coded pools take their placement, the zone type
included, from their profile.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
Signed-off-by: Eric Zhang <emzhang@ibm.com>
Signed-off-by: Callum James <callum.james@ibm.com>

Pools over zones need every daemon to read num_zones and count min_size
in each zone, so they start with the update that raises
require_osd_release to the release that ships them: the commit. That
update sets the OSDMap flag stretch_zones, converts the pools of global
stretch mode and marks down the up OSDs without OSD_STRETCH_ZONES. The
monitors check this flag, not require_osd_release, to tell whether the
upgrade is committed. The commit is refused while such OSDs are up
unless forced, and waits for an OSD boot pending in its update; after
it, boots without the bit are refused and no map is sent to such an OSD.

One check, why_osd_cannot_join(), says why an OSD cannot join: it is of
a release before require_osd_release (it lacks that release's SERVER_*
feature) or it lacks OSD_STRETCH_ZONES once the upgrade is committed.
preprocess_boot, prepare_boot (against the pending update too),
withhold_maps_from_downlevel_osd(), the mark-down in the update that
raises require_osd_release and ceph osd require-osd-release all use
it, so an OSD of an earlier release is kept out in the same way after
any release is committed. require-osd-release.rst says what happens to
such OSDs.

Before the commit, pools are created as earlier releases create them, in
global stretch mode too, so that their OSDs read them alike: a pool whose
num_zones comes only from the defaults is created in one zone, as
librados creates it. Only options that earlier releases cannot express,
such as a --num_zones above 1, are refused, with EPERM and a hint to run
ceph osd require-osd-release. Existing pools are not affected.

When the commit finds the OSDMap in stretch mode, global stretch mode of
the earlier release becomes pool creation defaults in the global section
of the configuration database: osd_pool_default_num_zones 2,
osd_pool_default_replica half of mon_stretch_pool_size, and
osd_pool_default_zone_failure_domain the type of the OSDMap's
stretch_mode_bucket.

ceph osd erasure-code-profile set replaces a profile that no pool uses,
committed or pending, with no flags, and a profile that a pool uses only
with --yes-i-really-mean-it, naming the pools when it refuses: OSDs read a
pool's profile whenever they load its PGs, so a new k, m or plugin
corrupts the pool's data. --force keeps its other use, a stripe_unit that
is not a multiple of 4096.

Adds tests to unittest_osd_monitor and unittest_pool_create_params.

At the commit the default profile gets the placement keys it lacks from
the options, and from then on every erasure coded pool gets a rule of
its own; before it, a pool that names no profile and gives no k, m or
CRUSH option shares the rule erasure-code, as in earlier releases.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Let the EC backend and peering test fixtures build an erasure coded
pool over several zones the way the monitor does.

For num_zones > 1, PGBackendTestFixture::setup_ec_pool() calls the new
OSDMapTestHelpers::enable_stretch_mode(), which installs a CRUSH map
with one datacenter per zone and a single-OSD host per OSD, adds the
rule built by add_simple_stretch_rule() and enables stretch mode in
the OSDMap, and then applies make_stretch_pool() to the pool.
create_ec_pool() records num_zones and the EC data and coding shard
counts, and takes the non-primary shards from
ECUtil::nonprimary_shards(). ECCrushTestFixture keeps a num_zones > 1
pool on that map instead of aborting, and installs its flat map and
"ec_rule" only for a pool with one zone.

OSDMapTestHelpers::set_degraded_stretch_mode(),
set_recovery_stretch_mode() and set_healthy_stretch_mode() make the
OSDMap changes of the monitor's stretch mode transitions, and
ECPeeringTestFixture::enter_*_stretch_mode() apply them and re-peer.

Once CRUSH places the shards, an OSD id is no longer a shard number,
so the log trim and committed-to hooks of ECPeeringTestFixture find
the primary's PeeringState through get_primary_test_pg().
MockECRecPred counts distinct relative shards (shard modulo k+m).

PGBackendTestFixture writes and EC reads return -EAGAIN while the
primary is peering and not yet active, as PrimaryLogPG would hold
the op. read_object() takes a fast_read argument,
simulate_multiple_osd_failures() moves from TestBackendBasics.cc into
the fixture, and scrub_object_call_count and run_recovery_call_count
let a test check that it really scrubbed or ran recovery.

TestECWithCRUSH.cc gains a TestECStretchPeering fixture for a two-zone
2+1 pool with stretch peering settings, without tests.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

In a pool stretched over several zones each zone holds a full copy of
the data: replica copies of a replicated pool, or a whole set of k+m
shards of an erasure coded pool. Split reads ignored zones: an erasure
coded split read always used the shards of the first zone, and a
replicated one any replica.

SplitOp::validate_flags(), now a static member, accepts LOCALIZE_READS
without BALANCE_READS when the pool has more than one zone; on a pool
with one zone such a read is still not split.
SplitOp::local_zone_for_acting_set() finds the zone nearest to the
client. It scores the first OSD present in each zone's part of the
acting set with CrushWrapper::get_common_ancestor_distance() against
the client's crush_location, and returns zone 0 when no zone can be
scored.

ECSplitOp::choose_read_shard() picks the absolute shard to read each
data shard from: with LOCALIZE_READS the copy in the nearest zone,
otherwise (BALANCE_READS) the copy in a zone chosen at random, per
data shard, among those whose OSD exists. If there is no such copy the
read is not split and goes to the primary. Multi-chunk split reads and
single-chunk direct reads both use it. Sub-reads are keyed by absolute
shard, which each SubRead records, and the reference sub-read is kept
as a pg_shard_t plus its acting index. A LOCALIZE_READS split read of
a stretched replicated pool uses only the replicas in the nearest
zone's part of the acting set. pg_pool_t::get_abs_shard() maps a
relative shard and a zone to the absolute shard.

Tests: unittest_split_ops_ut gains flag, zone selection and init_read()
tests against an Objecter with a real OSDMap, TestECWithCRUSH.cc tests
local_zone_for_acting_set() on a CRUSH map, the librados
LibRadosSplitOpZone and LibRadosSplitOpZoneStats suites run when the
cluster has two datacenter buckets, and ceph_test_rados gains
--crush-location.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

The EC backend generates its transactions per relative shard, so
written_shards in PG log entries and object_info_t::shard_versions
hold relative shards. In a pool over several zones the absolute shard
of a copy outside the first zone is not a relative shard, so every
lookup now converts it with pg_pool_t::get_relative_shard().

This covers rollback and the trimming of rollback objects
(PGBackend::rollback(), rollback_setattrs(), TrimmerPostRemove),
partial write tracking in PGBackend::partial_write(), divergent entry
handling in PGLog::merge_log(), rewind_divergent_log() and
proc_replica_log(), PeeringState::proc_master_log(), the missing set
updates for non-primary shards in pg_missing_t, the missing set
rebuild in read_log_and_missing(), backfill (update_range() and
scan_range_primary()) and the internal versions that the primary
reports for direct reads. partial_writes_last_complete stays keyed by
absolute shard.

PGLog::proc_replica_log() and rewind_divergent_log() take the pool,
and read_log_and_missing() takes an optional pool, which the OSD and
the log expansion in ceph-objectstore-tool pass; without one the shard
is used as it is. For a pool with one zone the relative and absolute
shards are the same.

Tests: TestPGLog.cc covers merge_log(), rewind_divergent_log(),
proc_replica_log(), _merge_object_divergent_entries() and the missing
set rebuild for a shard in the second zone.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

In an erasure coded pool over several zones each zone holds a whole
stripe of k+m shards, and the copies of a relative shard in different
zones must be identical. Deep scrub of an optimized EC pool whose
plugin supports CRC encode and decode kept the shard CRCs in maps of
k+m entries indexed by absolute shard, which a pool with more than one
zone overflows.

The CRCs are now collected per zone and keyed by relative shard
(collect_ec_zone_digests()). For a pool with more than one zone,
find_corrupt_zone_copies() first compares the copies of each relative
shard. It takes as reference the copy from a zone whose own stripe
decodes consistently, then the CRC that the most zones hold, then the
lowest zone. Every other copy with a different CRC is flagged as an EC
hash mismatch, so that shard is reported inconsistent ("data digest
differs from shard N"), counted as a deep error and not used as an
authoritative copy; its CRC is then replaced by the reference so that
it is not reported a second time. Each zone's stripe is then checked
on its own by the existing check that decodes every data shard from
the others, now split out into ec_stripe_mismatches() and
check_ec_stripe_digests(); its errors name the absolute shard. The
number of zones is the pool's, so a pool without EC optimizations,
whose backend has no stripe info, is checked as before.

setup_ec_digest_map() builds the map for the parity re-encode check by
relative shard, taking each relative shard from the first zone that
reported it. compare_obj_details() looks up a shard's expected version
in shard_versions by relative shard. A pool with one zone is checked
as before. Only an erasure coded pool is asked whether its plugin can
encode and decode CRCs, since ReplicatedBackend aborts on that question.

Tests: TestTScrubberBeECStretch and TestTScrubberBeECStretch3 in
test_scrubber_be.cc corrupt one zone copy of a data or parity shard in
two- and three-zone pools, and TestECUtil.cc gains tests of the zone
helpers and of per-zone shard maps rejecting absolute shards. The
test PG of test_scrubber_be.cc fails a test whose scrubber asks a
replicated pool about CRC encode and decode.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

A pool stretched over num_zones zones keeps a whole set of k+m shards
in each zone: absolute shard s of the acting set holds relative shard
s % (k+m) in zone s / (k+m). The EC backend now encodes, decodes and
plans reads and writes over the k+m relative shards, and uses the
absolute shard only to address OSDs, messages and objects. For a pool
with one zone the two are the same.

stripe_info_t gains get_rel_shard(), get_shard_zone(),
get_rel_shard_and_zone(), get_abs_shard() and zones_or(), which folds a
set of absolute shards onto relative shards; get_raw_shard() accepts an
absolute shard. Per-shard extent maps assert that they are not indexed
by a shard beyond k+m.

Reads prefer the primary's zone. get_all_avail_shards() maps the
acting, backfill and missing_loc shards to relative shards, taking
those in the primary's zone first and, when allowed, other zones only
for relative shards still missing. get_min_avail_to_read_shards(),
through the new select_shards_for_read(), first tries to decode from
the primary's zone alone and only then allows other zones;
ensure_primary_shard_for_omap() does the same. A shard that returned
an error is excluded by its absolute shard, so the retry can read the
other zone's copy. Read replies are recorded by relative shard.

Writes reach every zone. cache_ready() generates one transaction per
relative shard and gives each acting, recovery or backfill shard a
copy remapped to its absolute shard with Transaction::remap_shard().
Write planning sees the readable and writable relative shards.
Recovery decodes the relative shards missing in any zone once and
pushes each missing copy the data and version of its relative shard.

The recoverability predicate, direct reads and the DONTNEED hint for
parity shards also use the relative shard, and a shard is nonprimary if
its relative shard is. A pool that goes from two zones to one keeps
strays of the old zone, shards k+m and up, which are therefore still
read and recovered as copies of their relative shards.

Tests: TestECBackend.cc covers read shard selection and cache_ready()
over two and three zones, TestECUtil.cc the zone helpers,
test_ec_transaction.cc a two-zone write plan, and TestBackendBasics.cc
(MultiZoneWriteThenRead, MultiZoneFailover, TestECStretchBackend) and
TestECWithCRUSH.cc gain two-zone configurations.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

calc_ec_acting() fills each position from up, acting or a stray without
regard to zones, so a zone block's shards can end up in several zones.
Erasure coded pools with stretch peering rules
(peering_crush_bucket_count set) now get their acting set from the new
calc_ec_acting_stretch(). Replicated pools and pools without stretch
rules are unchanged.

An OSD can fill position i only if it holds shard i, is complete and
reaches the tail of the authoritative log. As in calc_ec_acting(),
up[i] comes first, then acting[i], then strays unless the call is
restricted to up and acting. With more than one zone, all positions of
a zone block are served from one CRUSH zone that no other block uses.
Every assignment of zones to blocks is tried: one whose shards can
recover the PG wins, then the one filling the most positions from up or
acting, then from up, then from zones holding enough of their block to
recover it, then the most positions, then the most from acting. No zone
gets more than size / peering_crush_bucket_target positions; with one
zone this is the only constraint.

With more than one zone, an up[i] that is not chosen is backfilled only
if its block would be served from its zone once the up OSDs hold their
shards, so a completed backfill target always joins the acting set.
Such a target may already hold its shard; as a backfill target it keeps
receiving writes. With one zone, an up[i] is backfilled when it cannot
serve its position. stretch_set_can_peer() skips the CRUSH_ITEM_NONE
holes that an EC acting set can have.

Add unittest_ec_acting_stretch (TestECActingStretch.cc) for two- and
three-zone pools, ZoneLoss_PeeredUntilDegradedStretchMode to
TestECWithCRUSH.cc and StretchSetCanPeer_NoneEntriesAndMandatoryMember
to TestStretchECMinSize.cc.

check_pg_upmaps() cancels an upmap that puts a zone block of an erasure
coded pool over zones in more than one zone, or two blocks in one zone,
as peering serves each block from one zone;
TestECStretchPeering.UpmapAcrossZoneBlocksKeepsPGActive tests it.

Assisted-by: Claude Code
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

In a pool over several zones, calc_ec_acting_stretch() keeps an up
shard as a backfill target while its zone block is still served from
another zone, even when the shard is complete, so that it gets every
write until the block moves to it. But when such a shard is a few
writes behind at the start of an interval, for example after its OSD
restarted, activate() restarts its backfill from scratch.

choose_acting() now records in usable_backfill_targets the backfill
targets of a multi-zone erasure coded pool that are complete and not
behind the tail of the authoritative log. activate() catches these up
from the log like any other peer, and update_calc_stats() takes their
missing objects from peer_missing instead of comparing object counts.
A target behind the authoritative log is still backfilled from scratch.
PrimaryLogPG::on_activate_complete() starts a backfill scan only when
needs_backfill() finds an incomplete target, rather than whenever there
are backfill targets. In other pools every backfill target is
incomplete once activate() has run, so they are unaffected.

TestECStretchPeering in TestECWithCRUSH.cc gets helpers that trim the
log, remap the PG with a pg-upmap and a pg_temp, and complete backfill.
New tests check that after a pg-upmap swaps the zone blocks under a
pg_temp the old acting set keeps serving while the up OSDs are
backfilled and the pg_temp is then dropped, that a returning zone keeps
its newest writes recoverable, and that a restarted complete up shard is
caught up from the log, has its missing object counted, or is
backfilled again when it is behind the authoritative log.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Add two- and three-zone erasure coded configurations to the parameters
of TestECFailoverWithPeering (unittest_ecfailover_with_peering), so its
existing tests also run on pools over several zones, and add tests for
what is specific to them.

New fixture helpers fail and restore a whole zone and take the pool
through degraded, recovery and healthy stretch mode as the monitor
would. The new tests cover the loss and return of each zone with full
and partial overwrites in between, per-zone min_size, adding a zone to
a single-zone pool and removing it again, a primary whose zone holds
fewer than k shards using the other zone, the same relative shard
missing in both zones, partial writes and their shard versions on
zone 1 shards with either zone's primary, attributes of an empty object,
delete and recreate, async recovery and divergent log rewind of zone 1
shards, rollback and roll-forward of partial overwrites on zone 1
shards, a change of zone order with every OSD up, recovery after going
to one zone from the strays of the old zone and from the pool's own
shards, and a three-zone pool
whose PG is not active while any one zone is down, since degraded
stretch mode only handles two zones.

OSD0DownAddNewOSDRecovery places its new OSD in the zone CRUSH map when
the pool has more than one zone, and ECCrushTestFixture drops unused
variables.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Add OSD perf counters for traffic between the zones of a pool with
stretch peering rules (peering_crush_bucket_count set). In an erasure
coded pool over zones a peer is in another zone when its shard is in
another zone block. In a replicated pool it is when its CRUSH ancestor
of type peering_crush_bucket_barrier differs from the primary's, which
the backend looks up once per peer and osdmap epoch.

For erasure coded and replicated pools the primary counts the
sub-writes it sends to another zone and their size
(cross_zone_write_ops, cross_zone_write_bytes) and the time until each
commits (cross_zone_write_latency and a latency histogram). Recovery
pushes to another zone are counted the same way
(cross_zone_recovery_push_ops, _bytes, _latency and a histogram). For
replicated pools the primary also counts recovery pulls from another
zone, the bytes they return and their latency (cross_zone_subread_ops,
cross_zone_subread_recv_bytes, cross_zone_subread_latency and a
histogram).

Nothing updates cross_zone_subread_sent_bytes yet, and reads from other
zones in erasure coded pools are not counted. ECListener gains
get_perf_logger(), which the classic and crimson PGs already provide
for peering; PrimaryLogPG forwards it to PG's, and the TestECBackend and
mock listener stubs implement it.

Assisted-by: Claude Code
Signed-off-by: Eric Zhang <emzhang@ibm.com>
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

scripts/zone_thrasher.py repeatedly takes down one randomly chosen zone
of a vstart cluster in stretch mode: it stops the zone's OSDs and
monitors, waits for degraded stretch mode, keeps the zone down for a
while, then restarts the monitors, waits for them to rejoin the quorum,
restarts the OSDs and waits until the cluster is out of degraded and
recovery stretch mode with every PG active+clean.

The zones are the datacenter buckets of the CRUSH tree that hold OSDs,
and each zone's monitors are found from the monitors' CRUSH locations;
the tiebreaker monitor is never stopped. Daemons are stopped with
SIGTERM using the PID files in out/ and restarted with ceph-osd -i or
ceph-mon -i. Options set the hold time, the timeouts, the delay between
iterations, the number of iterations and the random seed; --dry-run
only prints what would be done. On an error or Ctrl+C every zone
that is down is restarted before the script exits.

Assisted-by: Claude Code
Signed-off-by: Callum James <callum.james@ibm.com>
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

The dashboard creates a pool, then sends the size from its form with
`osd pool set <pool> size`, and sends it again when a pool is edited.
The Monitor refuses size for a pool over several zones: its copies are
set as replica, the copies in each zone. Once stretch mode makes new
pools two-zone pools, creating a pool with a size from the dashboard
failed.

For a pool over zones, a size equal to the pool's is not sent, and
another size is sent as replica, size / num_zones, or refused if it is
not a multiple of num_zones. Pools of one zone are unchanged.

PUT /api/osd/flags unsets every flag that a request leaves out, so the
flags API leaves out stretch_zones, which operators do not set or unset.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Bill Scales, ceph#72643 (comment):
the zone tests of split reads still let a read send a sub-read to a
primary in the other zone, for a version check that plain reads no
longer make. Once require_osd_release is past umbrella, a split read
takes the object's version from the shards it reads.

localize_reads_stay_in_zone and Zone1MultiChunkDataSubReadsStayInZone1
now expect no sub-read in the other zone, and their comments say so.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Emmanuel's review of ceph#72643
(r4435288238, r4435288252, r4435288291, r4435288316, r4435288322) and his
ceph#73016:

- the health checks include STRETCH_MODE_POOL_NOT_CONVERTED;
- ceph mon set_new_tiebreaker takes the Monitor's bare name;
- the 2 copies in each zone come from enable_stretch_mode or the upgrade
  commit, not from the default of osd_pool_default_size, which is 3, and
  the ls detail example is of a pool with replica 2;
- unstretching an erasure-coded pool in place re-peers its PGs, some of
  which can be unavailable for the whole transition, rather than taking
  the pool offline;
- recovery mode ends once no PG is degraded, inactive or unknown and
  mon_stretch_recovery_min_wait has passed, not when every PG is clean;
- MAX AVAIL adds up every take step of a rule, so a rule with one take
  step per zone does not report one zone's space;
- the page keeps the earlier page's stretch_mode1 and
  entering_stretch_mode anchors, and spells behavior as the other docs do.

Co-authored-by: eameh-LF <eameh@contractor.linuxfoundation.org>
Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Undo 69281aa, for Bill Scales,
ceph#72643 (comment).
Internal versions no longer go to the primary once require_osd_release
is past umbrella, but librados asks for the object's version on every
read by default, and a read that asks for it still sends a sub-read to
the primary. So localize_reads_stay_in_zone and
Zone1MultiChunkDataSubReadsStayInZone1 again allow one sub-read per read
in the other zone when the primary is there: with the tighter check
both failed in every teuthology run of
rados/singleton/all/rados-api-zones.yaml. Taking the version from the
shards read is https://tracker.ceph.com/issues/82981.

Assisted-by: Claude Code
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>