Go feature server build wrong redis key that be made from INT32 type entity

@franciscojavierarceo

cc. @franciscojavierarceo
These days, I try to contributing Go feature server in this repository.(e.g recent PR(#5352) that I made)
At present, I use Go feature server(alpha) and apply it on my production environment. My feast environment is below.

registry: aws s3 storage(in my fix: Reopen for integration test about add s3 storage-based registry store in Go feature server #5352 , I locally succeed in integrating with s3 storage)
offline store: spark + s3
online store: aws elasticache valkey(like redis)

But when using Go feature server with redis, I encountered the bug in building redis hash key.

At first, I define my entities, feature views and register them. At this time, one of my entities has int32 type. This int32 type is supported according to feast docs.
After that, I finish materializing features from offline store to online store and run Go feature server. Finally, client application request to go feature server.

But, I check that a specific feature is materialized well but from go feature server, the feature value is null. So instead of go feature server, I tried to fetch the feature with feast python sdk. When using feast python sdk, the feature value is not null. So I confused at this circumstance.

So, I deeply dive about this bug. I found that the features from other entity that has string type is not null from go feature server. So I paid attention to the entity that has int32 type.

I have two feast environments(dev, prod). At present, the prod environment is in the problem. So in dev environment, I intentionally cast the type of entity from int32 to string and re-register, materialize it.
After that, when I fetch the feature from go feature server in dev, I surprised that the feature value is not null!

So, I guess that the cause of this issue is maybe in this source code in go feature server. the probelm is in making hashkey for fetching(HMGet) to redis. Because, in the dev environment, hashkey by made from go feature server source code can be found in the redis. I check it by redis cli or redis insight GUI tool. But in the prod environment, the hashkey from go feature server was not in redis.

abnormal case(prod)

I build and run go feature server by debug mode and evaluate the redisKeys variable. I copy the data from redisKeys variable, and try to find the key in redis but there is no hashkey in redis.

So, I want to fix this bug.. Can you advise to me? I mention @shuchu based on the commit history about integrating redis as online store in go feature server.

Expected Behavior

In the same feature, the fetched feature from feast-sdk and go feature server is the same.

Current Behavior

In the same feature, the fetched feature from feast-sdk is normal but feature from go feature server is null.

Steps to reproduce

Specifications

Version: locally development but the problematic source code is the same as in the master branch.
Platform: MacOS 64bit arm64 OS platform
Subsystem:

Possible Solution

What is the potential cause of this bug?

@jyejare

CC @jyejare @ntkathole

can you provide some minimal example feature view and mock data?

@franciscojavierarceo

@franciscojavierarceo cc. @jyejare @ntkathole
For you, I ready script for register and mock data. The mock data can be download from below link.

feast_issue_#5357_entity_int32.csv

And I wrote reigster.py script based on this mock data.

# register.py
from feast.types import Int64, Float64
from feast import (
    FileSource,
    Entity,
    ValueType,
    FeatureView,
    Field
)

user = Entity(name="user", join_keys=["entity_id"], value_type=ValueType.INT64)

source = FileSource(
    name="dummy_data_source",
    path="/your/path/to/mock-data/feast_issue_#5357_entity_int32.csv",
    timestamp_field="timestamp"
)

fv = FeatureView(
    name="user_fv",
    entities=[user],
    source=source,
    schema=[
        Field(name="price", dtype=Int64),
        Field(name="discount", dtype=Float64)
    ]
)

Finally, I gave you feature_store.yaml. I use s3 storage but you can use local stroage as registry store. But you ensure that using redis as online store for testing my issue.

project: my_project
provider: aws
registry:
  path: s3://my-bucket/registry.db
  cache_ttl_seconds: 600  # default
entity_key_serialization_version: 2
online_store:
  type: redis
  redis_type: redis_cluster
  connection_string: localhost:6379

If any issues occur in the testing environment, let me know! In my production, I run Go feature server. So I try to fix it ASAP.. 😢 . If you have no time to fix it, I am willing to try to do it.

@iamcodingcat

@iamcodingcat Could you rebuild the feast environment with the serialization version 3 and retry?

@jyejare

@jyejare I will try to your guide. But while testing, I asked for your rebuild meaning. You mean that I must specify serialization version 3 including register and materialize features?

@jyejare

@jyejare At first, I register and materialize feature using serialization version 2 but when Go feature server uses serialization version 3, the hashkey doesn't still exist in redis.

I will com back after trying all serialization version 3 from register, materialize and Go feature server.

@jyejare

@jyejare After I test as version 3 from register, materilalize and go feature server, the error is still.
I store all data that use version 2 and 3. So in the redis GUI tools, the data by version is below attachment

the hashkey format between version 2 and 3 seems to be different. So specifying to version 3 is affected but it is not likely to resolve this problem. the hashkey version 3 is not in redis. Can you test it on this issue?

@iamcodingcat

@iamcodingcat thank you for finding this issue. I'm check on it now. by any chance, can you tell me which Redis version you are using locally for debugging? or you use the AWS elasticache valkey?

At the same time, maybe you want to read this piece of code from the Expedia group: https://github.com/EXPEbdodla/feast/blob/b72c2daac80ac22d1d8160f155bb55a1bdbf16f7/go/internal/feast/onlinestore/redisonlinestore.go#L135

They run the Go Feature server in production.

@shuchu

@iamcodingcat thank you for finding this issue. I'm check on it now. by any chance, can you tell me which Redis version you are using locally for debugging? or you use the AWS elasticache valkey?

@shuchu thanks for reply! I say my test environment. I only locally run go feature server and use registry as aws s3 remote storrage, online store as aws elasticache valkey. And my valkey version is 8.0.1

At the same time, maybe you want to read this piece of code from the Expedia group: https://github.com/EXPEbdodla/feast/blob/b72c2daac80ac22d1d8160f155bb55a1bdbf16f7/go/internal/feast/onlinestore/redisonlinestore.go#L135

They run the Go Feature server in production.

@shuchu When I compare Expedia group's source code(your link) with our feast master branch source code, it looks all the same. Only the difference is just making the logics to each function. Do you know the difference between them in terms of processing logic? Even more, I am appropriately replacing OnlineRead method in master branch with your metioned Expedia commit, in the test, the issue still exists.

@shuchu

@shuchu @jyejare cc. @franciscojavierarceo
Hi, I investigate it more and I am likely to find the root cause of this issue. But I ask you to check and review my found cause again.

First of all, Although register and materialize using entity type as INT32, go feature server is unconditionally regard as INT64. I check this by debug mode in below attachment.

Private opinion) This another issue causes on potential feature inconsistency between feast python sdk and go feature server like this issue. I saw the keys that will request to redis are different each other.(the difference is more including just only \x00(null) data when INT64) Let’s set this issue aside for now.

So to avoid type confusion, I cast from INT32 to INT64 from my source data(I use SparkSource and parquet files in s3) and I simulate the following test

The environment I tested under is as follows:
- When registering and materializing, the entity ID type was set to INT64.
- For registration, materialization, and the Go feature server, the entity_key_serialization_version was fixed to 2.

Currently, when retrieving features using the Feast Python SDK with that entity ID, the lookup works correctly. Even using the Go feature server, the lookup does work well. To investigate more, I ran both the Python SDK and the Go feature server in debug mode, and compared the two Redis keys(e.g python sdk[1] and Go feature server[2]) each used for lookup.

entity join key name: sku_id
entity key value: 60922975

At first, I check the hex binary and string data in below variables in the feast python sdk(normal case)

variable name: redis_key_bin
- data: \x02\x00\x00\x00sku_id\x04\x00\x00\x00\x08\x00\x00\x00_\x9c\xa1\x03\x00\x00\x00\x00recommendations (For easy distinguishment, I split a few pieces)
  - x02\x00\x00\x00
  - sku_id
  - \x04\x00\x00\x00\x08\x00\x00\x00
  - _\x9c\xa1\
  - \x03\x00\x00\x00\x00
  - recommendations
variable name: hset_keys
- data: [b'\x8d\x06\xaa\xbe', '_ts:sku_sparse_fv']
  - \x8d\x06\xaa\xbe
  - _ts:sku_sparse_fv

After that, I compare above binary or string data with the data from Go feature server. Below, I build go feature server in debug mode. And I got the binary data and convert hex binary data to string based on ASCII code. The value in parenthesis is converted from ASCII code table.

variable name: redisKey
- data: 02000000736b755f696404000000080000005f9ca103000000007265636f6d6d656e646174696f6e73(I split a few pieces, too)
  - 02 00 00 00 => is same with the value in redis_key_bin variable
  - 73(s) 6b(k) 75(u) 5f(_) 69(i) 64(d) => this is sku_id. So the same, too
  - 04 00 00 00 08 00 00 00 => same, too
  - 5f(_) 9c a1 => same, too
  - 03 00 00 00 00 => same, too
  - 72(r) 65(e) 63(c) 6f(o) 6d(m) 6d(m) 65(e) 6e(n) 64(d) 61(a) 74(t) 69(i) 6f(o) 6e(n) 73(s) => this is recommendations. So the same, too
variable name: hsetKeys
- data: [8d06aabe 5f74733a736b755f7370617273655f6676]
  - 8d 06 aa be => is same with \x8d\x06\xaa\xbe of hset_keys variable in feast python sdk
  - 5f(_) 74(t) 73(s) 3a(:) 73(s) 6b(k) 75(u) 5f(_) 73(s) 70(p) 61(a) 72(r) 73(s) 65(e) 5f(_) 66(f) 76(v) => this is _ts:sku_sparse_fv. So the same, too

Finally, Whether feast python sdk or Go feature server, the keys to redis seems to be same. So, due to above simulation, I know the reason whay Go feature server response is not null. I ask you to doubely check it again, please!!

More, I test it below another environment, Go feature server lookup well.

Source data: INT32
register: INT64

In this test case, materialization is done well, and fetching feature from go feature server is done, too.

@iamcodingcat

@iamcodingcat Perfect Debug. I think you are on the right path here.

@jyejare

@jyejare thanks for review. Then I’ll go ahead and close this issue.However, I do think that at some point the Go feature server should be improved to handle INT32 entity types properly as well. If possible in the future, I’ll try to work on improving that part too!

@iamcodingcat

glad to see this is solved. Thank you, @iamcodingcat

Navigation Menu

Repository navigation

Go feature server build wrong redis key that be made from `INT32` type entity #5357

Expected Behavior

Current Behavior

Steps to reproduce

Specifications

Possible Solution

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Go feature server build wrong redis key that be made from INT32 type entity #5357

Description

Expected Behavior

Current Behavior

Steps to reproduce

Specifications

Possible Solution

Activity

franciscojavierarceo commented on May 16, 2025

franciscojavierarceo commented on May 16, 2025

iamcodingcat commented on May 16, 2025

jyejare commented on May 16, 2025

iamcodingcat commented on May 16, 2025

iamcodingcat commented on May 16, 2025

iamcodingcat commented on May 16, 2025

shuchu commented on May 16, 2025

shuchu commented on May 16, 2025

iamcodingcat commented on May 17, 2025

iamcodingcat commented on May 17, 2025

jyejare commented on May 17, 2025

iamcodingcat commented on May 18, 2025

shuchu commented on May 18, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Go feature server build wrong redis key that be made from `INT32` type entity #5357