Skip to content

Subkey notification for hash fields#14958

Merged
ShooterIT merged 29 commits intoredis:unstablefrom
ShooterIT:subkey
Apr 17, 2026
Merged

Subkey notification for hash fields#14958
ShooterIT merged 29 commits intoredis:unstablefrom
ShooterIT:subkey

Conversation

@ShooterIT
Copy link
Copy Markdown
Member

@ShooterIT ShooterIT commented Mar 30, 2026

Motivation

Redis's existing keyspace notification system operates at the key level only — when a hash field is modified via HSET, HDEL, or HEXPIRE, the subscriber receives the key name and the event type, but not which fields were affected, therefore, these notifications has very little practical value.

This PR introduces a subkey notification system that extends keyspace events to include field-level (subkey) details for hash operations, through both Pub/Sub channels and the Module API.

New Pub/Sub Notification Channels

Four new channels are added:

Channel Format Payload
__subkeyspace@<db>__:<key> <event>|<len>:<subkey>[,...]
__subkeyevent@<db>__:<event> <key_len>:<key>|<len>:<subkey>[,...]
__subkeyspaceitem@<db>__:<key>\n<subkey> <event>
__subkeyspaceevent@<db>__:<event>|<key> <len>:<subkey>[,...]

Design rationale for 4 channels:

  • Subkeyspace: Subscribe to a specific key, receive all field changes in a single message — efficient for key-centric consumers.
  • Subkeyevent: Subscribe to a specific event type, receive key+fields — efficient for event-centric consumers.
  • Subkeyspaceitem: Subscribe to a specific key+field combination — the most selective, one message per field, no parsing needed.
  • Subkeyspaceevent: Subscribe to event+key combination, receiving only the affected fields — server-side filtering on both dimensions.

Subkeys are encoded in a length-prefixed format (<len>:<subkey>) to support binary-safe field names containing delimiters.

Safety guards:

  • Events containing | are skipped for __subkeyspace and __subkeyspaceevent channels (to avoid parsing ambiguity).
  • Keys containing \n are skipped for the __subkeyspaceitem channel (newline is the key/subkey separator).
  • Subkeys channels are only published when subkeys != NULL && count > 0.

Hash Command Integration

The following hash operations now emit subkey level notifications with the affected field names:

Command Event Subkeys
HSET / HMSET hset All fields being set
HSETNX hset The field (if set)
HDEL hdel All fields deleted
HGETDEL hdel / hexpired Deleted or lazily expired fields
HGETEX hexpire / hpersist / hdel / hexpired Affected fields per event
HINCRBY hincrby The field
HINCRBYFLOAT hincrbyfloat The field
HEXPIRE / HPEXPIRE / HEXPIREAT / HPEXPIREAT hexpire Updated fields
HPERSIST hpersist Persisted fields
HSETEX hset / hdel / hexpire / hexpired Affected fields per event
Field expiration (active/lazy) hexpired All expired fields (batched)

For field expiration, expired fields are collected into a dynamic array and sent as a single batched notification after the expiration loop, rather than one notification per field.

Module API

Three new APIs and one new callback type:

/* Function pointer type for keyspace event notifications with subkeys from modules. */
typedef void (*RedisModuleNotificationWithSubkeysFunc)(
    RedisModuleCtx *ctx, int type, const char *event,
    RedisModuleString *key, RedisModuleString **subkeys, int count);

/* Subscribe to keyspace notifications with subkey information.
 *
 * This is the extended version of RM_SubscribeToKeyspaceEvents. When subkeys
 * are available, the `subkeys` array and `count` are passed to the callback.
 * `subkeys` contains only the names of affected subkeys (values are not included),
 * and `count` is the number of elements. The array may contain duplicates when
 * the same subkey appears more than once in a command (e.g. HSET key f1 v1 f1 v2
 * produces subkeys=["f1","f1"], count=2). When no subkeys are present, `subkeys`
 * will be NULL and `count` will be 0. Whether events without subkeys are delivered
 * depends on the `flags` parameter (see below).
 *
 * `types` is a bit mask of event types the module is interested in
 * (using the same REDISMODULE_NOTIFY_* flags as RM_SubscribeToKeyspaceEvents).
 *
 * `flags` controls delivery filtering:
 *  - REDISMODULE_NOTIFY_FLAG_NONE: The callback is invoked for all matching
 *    events regardless of whether subkeys are present, so a separate
 *    RM_SubscribeToKeyspaceEvents registration can be omitted.
 *  - REDISMODULE_NOTIFY_FLAG_SUBKEYS_REQUIRED: The callback is only invoked
 *    when subkeys are not empty. Events without subkey information (e.g. SET,
 *    EXPIRE, DEL) are skipped.
 *
 * The callback signature is:
 *   void callback(RedisModuleCtx *ctx, int type, const char *event,
 *                 RedisModuleString *key, RedisModuleString **subkeys, int count);
 *
 * The subkeys array and its contents are only valid during the callback.
 * The underlying objects may be stack-allocated or temporary, so
 * RM_RetainString must NOT be used on them. To keep a subkey beyond
 * the callback (e.g. in a RM_AddPostNotificationJob callback), use
 * RM_HoldString (which handles static objects by copying) or
 * RM_CreateStringFromString to make a deep copy before returning.
 */
int RM_SubscribeToKeyspaceEventsWithSubkeys(RedisModuleCtx *ctx, int types, int flags, RedisModuleNotificationWithSubkeysFunc callback);

/* Unregister a module's callback from keyspace notifications with subkeys
 * for specific event types.
 *
 * This function removes a previously registered subscription identified by
 * the event mask, delivery flags, and the callback function.
 *
 * Parameters:
 *  - ctx: The RedisModuleCtx associated with the calling module.
 *  - types: The event mask representing the notification types to unsubscribe from.
 *  - flags: The delivery flags that were used during registration.
 *  - callback: The callback function pointer that was originally registered.
 *
 * Returns:
 *  - REDISMODULE_OK on successful removal of the subscription.
 *  - REDISMODULE_ERR if no matching subscription was found. */ 
int RM_UnsubscribeFromKeyspaceEventsWithSubkeys(
    RedisModuleCtx *ctx, int types, int flags,
    RedisModuleNotificationWithSubkeysFunc cb);

/* Like RM_NotifyKeyspaceEvent, but also triggers subkey-level notifications
 * when subkeys are provided. Both key-level (keyspace/keyevent) and
 * subkey-level (subkeyspace/subkeyevent/subkeyspaceitem/subkeyspaceevent)
 * channels are published to, depending on the server configuration.
 *
 * This is the extended version of RM_NotifyKeyspaceEvent and can actually
 * replace it. When called with subkeys=NULL and count=0, it behaves
 * identically to RM_NotifyKeyspaceEvent. */
int RM_NotifyKeyspaceEventWithSubkeys(
    RedisModuleCtx *ctx, int type, const char *event,
    RedisModuleString *key, RedisModuleString **subkeys, int count);

Configuration

Subkey notifications are controlled via the existing notify-keyspace-events configuration string with four new characters: notify-keyspace-events "STIV"

S -> Subkeyspace events, published with __subkeyspace@<db>__:<key> prefix.
T -> Subkeyevent events, published with __subkeyevent@<db>__:<event> prefix.
I -> Subkeyspaceitem events, published per subkey with __subkeyspaceitem@<db>__:<key>\n<subkey> prefix.
V -> Subkeyspaceevent events, published with __subkeyspaceevent@<db>__:<event>|<key> prefix.

These flags are independent from the existing key-level flags (K, E, etc.). Enabling subkey notifications does not implicitly enable or depend on keyspace/keyevent notifications, and vice versa.

Known Limitations

  • Duplicate fields in subkey notifications: Subkey notification payloads may contain duplicate field names when the same field is affected more than once within a single command. Since duplicate fields are not the common case and deduplication would introduce significant overhead on every notification, we chose not to deduplicate at this time.
  • Subkey is sds encoding object: We assume the subkey is sds encoding object, and access it by subkey->ptr, and there is an assert, redis will crash if not.

Note

Medium Risk
Touches core keyspace notification and module API surfaces and changes notification behavior/perf paths across many hash commands and expiry code, so regressions could affect Pub/Sub and modules. Guardrails and extensive new tests reduce but don’t eliminate risk.

Overview
Adds subkey-level keyspace notifications (hash field names) on top of existing key-level KSN, gated by four new notify-keyspace-events classes (S,T,I,V) and documented in redis.conf.

Extends the notifications pipeline with notifyKeyspaceEventWithSubkeys() / isSubkeyNotifyEnabled() and publishes new Pub/Sub channel variants (__subkeyspace@.., __subkeyevent@.., __subkeyspaceitem@.., __subkeyspaceevent@..) with length-prefixed subkey payloads and ambiguity guards; also short-circuits work when there are no Pub/Sub subscribers.

Updates the module KSN dispatcher to accept optional subkeys, adds subscribe/unsubscribe + notify APIs for modules (RM_*WithSubkeys), caches subscriber type masks for faster checks, and wires new notification flags into server.h/redismodule.h and config parsing.

Integrates subkey emission across hash mutations and hash-field expiration paths (batching expired fields where possible) and adjusts tests to cover module callbacks, replication, and the new Pub/Sub channel formats; minor vector API tweaks support stack-backed field collection.

Reviewed by Cursor Bugbot for commit f6613bc. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread src/t_hash.c Outdated
@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Mar 30, 2026

🤖 Augment PR Summary

Summary: This PR adds field-level (subkey) notifications for hash operations, extending Redis keyspace notifications to include the affected hash fields.

Changes:

  • Introduces four new Pub/Sub channel families for subkey notifications: __subkeyspace, __subkeyevent, __subkeyspaceitem, and __subkeyspaceevent.
  • Extends notification flag parsing/serialization to support new notify-keyspace-events specifiers S/T/I/V.
  • Adds a new internal API notifyKeyspaceEventWithSubkeys() and helper utilities for encoding subkey payloads in a length-prefixed format.
  • Extends the module keyspace notification dispatcher to optionally deliver subkeys, plus new module APIs to subscribe/unsubscribe and to emit notifications with subkeys.
  • Integrates subkey-aware notifications across hash commands (HSET/HDEL/HINCRBY/HEXPIRE/HPERSIST/HGETEX/HGETDEL/HSETEX, and lazy/active field expiration paths).
  • Adds stack-vs-heap local-array macros to reduce allocation overhead for common “few fields” cases.
  • Updates unit tests for both the module API behavior and the new Pub/Sub channel formats, including replication coverage.

Technical Notes: Subkeys are published in a binary-safe <len>:<subkey> encoding, and some channels include additional separators (|, \n) with guards to avoid ambiguous formats.

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 3 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

Comment thread src/notify.c
Comment thread src/t_hash.c Outdated
Comment thread src/config.c Outdated
Comment thread src/t_hash.c Outdated
Comment thread src/t_hash.c Outdated
Comment thread src/t_hash.c
Comment thread src/notify.c
@ShooterIT
Copy link
Copy Markdown
Member Author

Hi @oranagra would you like to take a look, or just the top comment?

Hi @oshadmi would you like to review the module APIs

@oranagra
Copy link
Copy Markdown
Member

oranagra commented Apr 6, 2026

description LGTM

Comment thread src/t_hash.c Outdated
@ShooterIT ShooterIT added the release-notes indication that this issue needs to be mentioned in the release notes label Apr 8, 2026
@github-project-automation github-project-automation Bot moved this to Todo in Redis 8.8 Apr 8, 2026
@ShooterIT
Copy link
Copy Markdown
Member Author

Hi @alonre24 Maybe you also care about module API, please have a look when you are available

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 9f3a362. Configure here.

Comment thread src/t_hash.c
@ShooterIT
Copy link
Copy Markdown
Member Author

Hi @fcostaoliveira i refactored some code based on new vector data structure, could you please rerun tests again?

@fcostaoliveira
Copy link
Copy Markdown
Collaborator

Hi @fcostaoliveira i refactored some code based on new vector data structure, could you please rerun tests again?

on it :)

Comment thread src/t_hash.c Outdated
Comment thread src/t_hash.c Outdated
@fcostaoliveira
Copy link
Copy Markdown
Collaborator

Benchmark results — commit 6b6f0f80 (vector refactor)

Runner: x86-aws-m7i.metal-24xl + arm-aws-m8g.metal-24xl (bare metal)
Baseline: redis/redis:unstable (3 datapoints)
Comparison: ShooterIT/redis:subkey @ 6b6f0f80 (1 datapoint)
Tests: 16 hash data type benchmarks (load, hgetall, hexpire, hpexpire, hexpireat, hpexpireat)

x86 (Intel Sapphire Rapids) — 0 regressions, 0 improvements

Test Baseline PR Change
load-50f-1000B 26,389 ±0.5% 26,131 -1.0%
load-50f-100B 63,684 ±0.4% 63,759 +0.1%
load-50f-10B-expiration 20,909 ±1.2% 21,241 +1.6%
load-50f-1000B-expiration 18,507 ±0.2% 18,361 -0.8%
load-50f-10B-long-exp 20,893 ±1.1% 21,215 +1.5%
load-50f-10B-short-exp 21,536 ±0.9% 21,784 +1.1%
hgetall-50f-100B 165,531 ±2.1% 164,507 -0.6%
hexpire-5f 140,457 ±0.5% 140,972 +0.4%
hexpire-50f 30,347
hpexpire-5f 140,457 ±0.5% 139,806 -0.5%
hpexpire-50f 29,705 ±0.3% 30,174 +1.6%
hexpireat-5f 140,035 ±0.7% 141,300 +0.9%
hexpireat-50f 30,288
hpexpireat-5f 144,188
hpexpireat-50f 31,038 ±0.7% 30,544 -1.6%

All within noise. No performance impact on x86.

ARM (Neoverse-V2) — 2 small regressions (1dp, needs confirmation)

Test Baseline PR Change
hexpireat-50f 33,252 ±0.2% 32,113 -3.4%
hpexpireat-50f 33,536 ±0.2% 32,390 -3.4%

All other ARM tests: No Change.

The two -3.4% regressions on ARM are single-datapoint — both on the 50-field expireat variants. Baseline has very low variance (0.2%), so this warrants a re-trigger to confirm.

Summary

The vector refactor has no measurable performance impact on hash data type commands on x86. ARM shows two borderline regressions (-3.4%) on the hexpireat/hpexpireat 50-field tests — recommending a re-trigger for 3dp confirmation.

@ShooterIT ShooterIT requested a review from moticless April 16, 2026 03:26
@ShooterIT
Copy link
Copy Markdown
Member Author

thanks @filipecosta90 , the performance degradation caused by this PR is relatively small. cc @moticless

@moticless
Copy link
Copy Markdown
Collaborator

@tezc , can you review please related changes of cluster ASM? Thanks.

Comment thread src/module.c
@tezc
Copy link
Copy Markdown
Collaborator

tezc commented Apr 16, 2026

@moticless @ShooterIT I don't see any change that may affect ASM. Please point me if there are such parts and want me to take a look.

@moticless
Copy link
Copy Markdown
Collaborator

@moticless @ShooterIT I don't see any change that may affect ASM. Please point me if there are such parts and want me to take a look.

@tezc , True. my mistake. just minor refactor of moduleHasSubscribersForKeyspaceEvent(). No side effects.

@ShooterIT ShooterIT requested a review from moticless April 16, 2026 14:15
@ShooterIT
Copy link
Copy Markdown
Member Author

Daily CI: https://github.com/ShooterIT/redis/actions/runs/24544487006 Most of them are successful, and the failed cases have nothing to do with this PR. So merging

@ShooterIT ShooterIT merged commit 4757561 into redis:unstable Apr 17, 2026
17 of 18 checks passed
@ShooterIT ShooterIT deleted the subkey branch April 17, 2026 05:39
@github-project-automation github-project-automation Bot moved this from Todo to Done in Redis 8.8 Apr 17, 2026
@sundb sundb added the state:needs-doc-pr requires a PR to redis-doc repository label Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

action:run-benchmark Triggers the benchmark suite for this Pull Request release-notes indication that this issue needs to be mentioned in the release notes state:needs-doc-pr requires a PR to redis-doc repository

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

9 participants