Skip to content

Commit fc290d3

Browse files
rkachachrkachach
authored
Merge pull request #67651 from ShwetaBhosale1/fix_issue_75189_nfs_rdma_support
mgr/cephadm: nfs rdma support Reviewed-by: Redouane Kachach <rkachach@ibm.com>
2 parents 2cb4cae + 66e7c83 commit fc290d3

22 files changed

Lines changed: 874 additions & 56 deletions

File tree

‎doc/cephadm/services/nfs.rst‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,128 @@ address is not present and ``monitoring_networks`` is specified, an IP address
7979
that matches one of the specified networks will be used. If neither condition
8080
is met, the default binding will happen on all available network interfaces.
8181

82+
NFS over RDMA
83+
-------------
84+
85+
NFS over RDMA is disabled by default. To enable it, set ``enable_rdma: true`` in
86+
the NFS service spec. You can optionally set ``rdma_port`` to use a custom RDMA
87+
port, if omitted, NFS Ganesha uses its default.
88+
89+
When RDMA is enabled:
90+
91+
* New exports in the cluster default to **Transports = TCP, RDMA**
92+
* For colocation, each entry in ``colocation_ports`` must include
93+
``rdma_port`` in addition to ``data_port`` and ``monitoring_port``.
94+
95+
Example with RDMA enabled:
96+
97+
.. code-block:: yaml
98+
99+
service_type: nfs
100+
service_id: mynfs
101+
placement:
102+
count: 1
103+
hosts: [host1]
104+
spec:
105+
port: 2049
106+
monitoring_port: 9587
107+
enable_rdma: true
108+
rdma_port: 20049 # optional
109+
110+
.. note:: If you use a bind address (e.g. ``virtual_ip``, ``ip_addrs``, or
111+
``networks``) with ``enable_rdma``, ensure the network interface for that
112+
address is RDMA-capable. On the host, run ``rdma link show`` and confirm the
113+
netdev for the interface with the bind IP is listed.
114+
115+
NFS Daemon Colocation
116+
----------------------
117+
118+
By default, cephadm avoids placing multiple NFS daemons on the same host. However,
119+
you can enable colocation to deploy multiple NFS daemons on the same host for
120+
increased capacity or redundancy.
121+
122+
.. note::
123+
When a host becomes unavailable, cephadm will automatically redeploy the
124+
affected NFS daemons on the remaining available hosts to maintain the desired
125+
``count``. This may result in multiple daemons running on the same host,
126+
even if colocation was not explicitly configured. The system ensures that
127+
the total number of running daemons matches the specified count across
128+
all available hosts.
129+
130+
Colocation with Custom Ports
131+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
132+
133+
For more control over port assignments, you can specify custom ports for colocated daemons
134+
using the ``colocation_ports`` parameter:
135+
136+
.. code-block:: yaml
137+
138+
service_type: nfs
139+
service_id: mynfs
140+
placement:
141+
count: 4
142+
hosts:
143+
- host1
144+
- host2
145+
spec:
146+
port: 2049
147+
monitoring_port: 9587
148+
colocation_ports:
149+
- data_port: 3049
150+
monitoring_port: 9588
151+
- data_port: 3050
152+
monitoring_port: 9589
153+
- data_port: 3051
154+
monitoring_port: 9590
155+
156+
In this configuration, 4 daemons total are deployed (2 per host), distributed across
157+
``host1`` and ``host2``:
158+
159+
* **host1, daemon 1**: ``port: 2049`` and ``monitoring_port: 9587``
160+
* **host1, daemon 2**: ``data_port: 3049`` and ``monitoring_port: 9588``
161+
* **host2, daemon 1**: ``port: 2049`` and ``monitoring_port: 9587``
162+
* **host2, daemon 2**: ``data_port: 3049`` and ``monitoring_port: 9588``
163+
164+
.. note::
165+
* The ``colocation_ports`` list defines ports for **additional** daemons only
166+
(2nd, 3rd, 4th, etc.). The first daemon always uses the base ``port`` and
167+
``monitoring_port`` from the spec.
168+
* The number of entries in ``colocation_ports`` should be ``count - 1``,
169+
to cover the node down scenario (or ``count_per_host - 1`` when using ``count_per_host``).
170+
* Each entry must specify both ``data_port`` and ``monitoring_port``. When
171+
``enable_rdma`` is true, each entry must also include ``rdma_port``.
172+
* If ``colocation_ports`` is not specified, ports will be automatically
173+
incremented for colocated daemons (e.g., 2049 → 2050 → 2051 for data ports,
174+
and 9587 → 9588 → 9589 for monitoring ports).
175+
176+
Per-Host Colocation
177+
~~~~~~~~~~~~~~~~~~~
178+
179+
You can also use ``count_per_host`` to specify exactly how many daemons should
180+
run on each host:
181+
182+
.. code-block:: yaml
183+
184+
service_type: nfs
185+
service_id: mynfs
186+
placement:
187+
count_per_host: 3
188+
hosts:
189+
- host1
190+
- host2
191+
- host3
192+
spec:
193+
port: 2049
194+
monitoring_port: 9587
195+
colocation_ports:
196+
- data_port: 3049
197+
monitoring_port: 9588
198+
- data_port: 4049
199+
monitoring_port: 9589
200+
201+
This will deploy exactly 3 NFS daemons on each of the 3 hosts (9 daemons total),
202+
with custom ports for the 2nd and 3rd daemons on each host.
203+
82204
TLS/SSL Example
83205
---------------
84206

‎doc/man/8/cephadm.rst‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Synopsis
1313
| [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR]
1414
| [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT]
1515
| [--retry RETRY] [--no-container-init]
16-
| {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,list-images,update-osd-service}
16+
| {version,pull,inspect-image,ls,list-networks,list-rdma,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,list-images,update-osd-service}
1717
| ...
1818
1919

@@ -25,6 +25,8 @@ Synopsis
2525
2626
| **cephadm** **list-networks**
2727
28+
| **cephadm** **list-rdma**
29+
2830
| **cephadm** **adopt** [-h] --name NAME --style STYLE [--cluster CLUSTER]
2931
| [--legacy-dir LEGACY_DIR] [--config-json CONFIG_JSON]
3032
| [--skip-firewalld] [--skip-pull]
@@ -347,6 +349,10 @@ list-networks
347349

348350
list IP networks
349351

352+
list-rdma
353+
---------
354+
355+
list RDMA devices and their netdev interfaces
350356

351357
ls
352358
--

‎doc/mgr/nfs.rst‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ Create NFS Ganesha Cluster
3131

3232
.. prompt:: bash #
3333

34-
ceph nfs cluster create <cluster_id> [<placement>] [--ingress] [--virtual_ip <value>] [--ingress-mode {default|keepalive-only|haproxy-standard|haproxy-protocol}] [--port <int>]
34+
ceph nfs cluster create <cluster_id> [<placement>] [--ingress] [--virtual_ip <value>] [--ingress-mode {default|keepalive-only|haproxy-standard|haproxy-protocol}] [--port <int>] [--enable-rdma] [--rdma_port <int>] [-i <spec_file>]
3535

3636
This creates a common recovery pool for all NFS Ganesha daemons, new user based on
3737
``cluster_id``, and a common NFS Ganesha config RADOS object.
@@ -290,7 +290,7 @@ Create CephFS Export
290290

291291
.. prompt:: bash #
292292

293-
ceph nfs export create cephfs --cluster-id <cluster_id> --pseudo-path <pseudo_path> --fsname <fsname> [--readonly] [--path=/path/in/cephfs] [--client_addr <value>...] [--squash <value>] [--sectype <value>...] [--cmount_path <value>] [--xprtsec <value>]
293+
ceph nfs export create cephfs --cluster-id <cluster_id> --pseudo-path <pseudo_path> --fsname <fsname> [--readonly] [--path=/path/in/cephfs] [--client_addr <value>...] [--squash <value>] [--sectype <value>...] [--cmount_path <value>] [--xprtsec <value>] [--transports <value>...]
294294

295295
This creates export RADOS objects containing the export block, where
296296

@@ -334,6 +334,12 @@ allowed to be any complete path hierarchy between ``/`` and the ``EXPORT {path}`
334334
.. note:: If this and the other ``EXPORT { FSAL {} }`` options are the same between multiple exports, those exports will share a single CephFS client.
335335
If not specified, the default is ``/``.
336336

337+
``<transports>`` is optional. List of NFS transport protocols. Valid values are
338+
``TCP``, ``UDP``, and ``RDMA``. Multiple values may be passed (e.g.
339+
``--transports TCP --transports RDMA`` or ``--transports TCP,RDMA``). If omitted,
340+
the export uses the default (e.g. TCP only, or TCP and RDMA when the cluster
341+
has RDMA enabled).
342+
337343
.. note:: Specifying values for sectype that require Kerberos will only function on servers
338344
that are configured to support Kerberos. Setting up NFS-Ganesha to support Kerberos
339345
can be found here `Kerberos setup for NFS Ganesha in Ceph <https://github.com/nfs-ganesha/nfs-ganesha/wiki/Kerberos-setup-for-NFS-Ganesha-in-Ceph>`_.
@@ -358,7 +364,7 @@ To export a *bucket*:
358364

359365
.. prompt:: bash #
360366

361-
ceph nfs export create rgw --cluster-id <cluster_id> --pseudo-path <pseudo_path> --bucket <bucket_name> [--user-id <user-id>] [--readonly] [--client_addr <value>...] [--squash <value>] [--sectype <value>...] [--xprtsec <value>]
367+
ceph nfs export create rgw --cluster-id <cluster_id> --pseudo-path <pseudo_path> --bucket <bucket_name> [--user-id <user-id>] [--readonly] [--client_addr <value>...] [--squash <value>] [--sectype <value>...] [--xprtsec <value>] [--transports <value>...]
362368

363369
For example, to export ``mybucket`` via NFS cluster ``mynfs`` at the
364370
pseudo-path ``/bucketdata`` to any host in the ``192.168.10.0/24`` network
@@ -402,6 +408,10 @@ multiple values may be separated by a comma (example: ``--sectype
402408
krb5p,krb5i``). The server will negotatiate a supported security type with the
403409
client preferring the supplied methods left-to-right.
404410

411+
``<transports>`` is optional. Valid values are ``TCP``, ``UDP``, and ``RDMA``.
412+
Multiple values may be passed. If omitted, defaults apply (e.g. TCP and RDMA
413+
when the cluster has RDMA enabled).
414+
405415
.. note:: Specifying values for sectype that require Kerberos will only
406416
function on servers that are configured to support Kerberos. Setting up
407417
NFS-Ganesha to support Kerberos is outside the scope of this document.
@@ -417,7 +427,7 @@ To export an RGW *user*:
417427

418428
.. prompt:: bash #
419429

420-
ceph nfs export create rgw --cluster-id <cluster_id> --pseudo-path <pseudo_path> --user-id <user-id> [--readonly] [--client_addr <value>...] [--squash <value>]
430+
ceph nfs export create rgw --cluster-id <cluster_id> --pseudo-path <pseudo_path> --user-id <user-id> [--readonly] [--client_addr <value>...] [--squash <value>] [--transports <value>...]
421431

422432
For example, to export *myuser* via NFS cluster *mynfs* at the pseudo-path */myuser* to any host in the ``192.168.10.0/24`` network
423433

‎src/cephadm/cephadm.py‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@
155155
executes_early,
156156
require_image
157157
)
158-
from cephadmlib.host_facts import HostFacts, list_networks
158+
from cephadmlib.host_facts import HostFacts, list_networks, list_rdma
159159
from cephadmlib.ssh import authorize_ssh_key, check_ssh_connectivity
160160
from cephadmlib.daemon_form import (
161161
DaemonForm,
@@ -3563,6 +3563,11 @@ def serialize_sets(obj: Any) -> Any:
35633563

35643564
print(json.dumps(r, indent=4, default=serialize_sets))
35653565

3566+
3567+
def command_list_rdma(ctx: CephadmContext) -> None:
3568+
r = list_rdma(ctx)
3569+
print(json.dumps(r, indent=4))
3570+
35663571
##################################
35673572

35683573

@@ -5011,6 +5016,10 @@ def _get_parser():
50115016
'list-networks', help='list IP networks')
50125017
parser_list_networks.set_defaults(func=command_list_networks)
50135018

5019+
parser_list_rdma = subparsers.add_parser(
5020+
'list-rdma', help='list RDMA devices and their netdev interfaces')
5021+
parser_list_rdma.set_defaults(func=command_list_rdma)
5022+
50145023
parser_adopt = subparsers.add_parser(
50155024
'adopt', help='adopt daemon deployed with a different tool')
50165025
parser_adopt.set_defaults(func=command_adopt)

‎src/cephadm/cephadmlib/daemons/nfs.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ def __init__(
6161
self.extra_args = dict_get(config_json, 'extra_args', [])
6262
self.files = dict_get(config_json, 'files', {})
6363
self.rgw = dict_get(config_json, 'rgw', {})
64+
self.enable_rdma = dict_get(config_json, 'enable_rdma', False)
6465

6566
# validate the supplied args
6667
self.validate()
@@ -238,6 +239,17 @@ def customize_container_args(
238239
self, ctx: CephadmContext, args: List[str]
239240
) -> None:
240241
args.append(ctx.container_engine.unlimited_pids_option)
242+
if self.enable_rdma:
243+
# Container args when NFS RDMA is enabled
244+
rdma_args: List[str] = [
245+
'-v',
246+
'/dev/infiniband:/dev/infiniband',
247+
'--cap-add=IPC_LOCK',
248+
'--ulimit',
249+
'memlock=-1:-1',
250+
'--privileged',
251+
]
252+
args.extend(rdma_args)
241253

242254
def default_entrypoint(self) -> str:
243255
return self.entrypoint

‎src/cephadm/cephadmlib/host_facts.py‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -860,6 +860,51 @@ def list_networks(ctx):
860860
return res
861861

862862

863+
def list_rdma(ctx: CephadmContext) -> List[Dict[str, str]]:
864+
"""List RDMA devices by parsing 'rdma link show' output.
865+
Returns a list of dicts with keys: link, state, physical_state, netdev.
866+
Returns empty list if rdma tool is not installed or command fails.
867+
"""
868+
execstr: Optional[str] = find_executable('rdma')
869+
if not execstr:
870+
logger.error("'rdma' command not found, no RDMA devices listed")
871+
return []
872+
try:
873+
out, _, _ = call_throws(
874+
ctx,
875+
[execstr, 'link', 'show'],
876+
verbosity=CallVerbosity.QUIET_UNLESS_ERROR,
877+
)
878+
except Exception as e:
879+
logger.error('rdma link show failed: %s', e)
880+
return []
881+
# Format: link <name> state <state> physical_state <phys> netdev <netdev>
882+
pattern = re.compile(
883+
r'link\s+(\S+)\s+state\s+(\S+)\s+physical_state\s+(\S+)\s+netdev\s+'
884+
r'(\S+)'
885+
)
886+
result: List[Dict[str, str]] = []
887+
for line in out.splitlines():
888+
line = line.strip()
889+
if not line:
890+
continue
891+
m = pattern.search(line)
892+
if m:
893+
result.append(
894+
{
895+
'link': m.group(1),
896+
'state': m.group(2),
897+
'physical_state': m.group(3),
898+
'netdev': m.group(4),
899+
}
900+
)
901+
else:
902+
logger.debug(
903+
"Skipped RDMA device '%s', as pattern did not match", line
904+
)
905+
return result
906+
907+
863908
def _list_ipv4_networks(
864909
ctx: CephadmContext,
865910
) -> Dict[str, Dict[str, Set[str]]]:

‎src/pybind/mgr/cephadm/inventory.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -950,6 +950,14 @@ def update_host_networks(
950950
self.networks[host] = nets
951951
self.last_network_update[host] = datetime_now()
952952

953+
def get_interface_for_ip(self, host: str, ip: str) -> Optional[str]:
954+
"""Return the network interface name that has the given IP on host, or None."""
955+
for _subnet, ifaces in self.networks.get(host, {}).items():
956+
for iface, ips in ifaces.items():
957+
if ip in ips:
958+
return iface
959+
return None
960+
953961
def update_daemon_config_deps(self, host: str, name: str, deps: List[str], stamp: datetime.datetime) -> None:
954962
self.daemon_config_deps[host][name] = {
955963
'deps': deps,

‎src/pybind/mgr/cephadm/schedule.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -347,6 +347,24 @@ def place(self):
347347

348348
def expand_candidates(ls: List[DaemonPlacement], num: int) -> List[DaemonPlacement]:
349349
r = []
350+
# Check if spec has custom colocation ports (converted to list format)
351+
if hasattr(self.spec, 'get_colocation_ports_list'):
352+
custom_ports_list = self.spec.get_colocation_ports_list()
353+
if custom_ports_list:
354+
# First daemon (i=0) always uses base ports from spec
355+
# Additional daemons (i=1,2,...) use colocation_ports if available
356+
for i in range(num):
357+
if i == 0:
358+
r.extend([dp.renumber_ports(0) for dp in ls])
359+
elif i - 1 < len(custom_ports_list):
360+
ports = custom_ports_list[i - 1]
361+
r.extend([DaemonPlacement(
362+
dp.daemon_type, dp.hostname, dp.network, dp.name,
363+
dp.ip, ports, dp.rank, dp.rank_generation
364+
) for dp in ls])
365+
else:
366+
r.extend([dp.renumber_ports(i) for dp in ls])
367+
return r
350368
for offset in range(num):
351369
r.extend([dp.renumber_ports(offset) for dp in ls])
352370
return r

‎src/pybind/mgr/cephadm/serve.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -450,6 +450,17 @@ def _refresh_host_networks(self, host: str) -> Optional[str]:
450450
self.mgr.cache.save_host(host)
451451
return None
452452

453+
async def get_rdma_devices(self, host: str) -> List[Dict[str, Any]]:
454+
"""Return list of RDMA devices on host from cephadm list-rdma, or [] on error."""
455+
try:
456+
out = await self._run_cephadm_json(
457+
host, 'mon', 'list-rdma', [], no_fsid=True,
458+
log_output=self.mgr.log_refresh_metadata)
459+
return out if isinstance(out, list) else []
460+
except OrchestratorError as e:
461+
self.log.error('Failed to get RDMA devices for host %s: %s', host, e)
462+
return []
463+
453464
def _refresh_host_osdspec_previews(self, host: str) -> Optional[str]:
454465
self.update_osdspec_previews(host)
455466
self.mgr.cache.save_host(host)

0 commit comments

Comments
 (0)