Skip to content

Commit 442ca5a

Browse files
mrzzyterryyylim
mrzzy
and
terryyylim
authored
Add Feature Tables API to Core & Python SDK (#1019)
* Reorganise existing protos in CoreService by type. Signed-off-by: Terence <terencelimxp@gmail.com> * Add new FeatureTables API to Core Protobuf definitions Signed-off-by: Terence <terencelimxp@gmail.com> * Fix name collision in proto java outer classname with message name Signed-off-by: Terence <terencelimxp@gmail.com> * Add missing max age field to Feature Table proto. Signed-off-by: Terence <terencelimxp@gmail.com> * Add Flyway DB migration to add Feature Table API. Signed-off-by: Terence <terencelimxp@gmail.com> * Rename options field to options_json and change type to text. * Options to be stored as Protobuf JSON. * Change from varchar to text to remove char limit Signed-off-by: Terence <terencelimxp@gmail.com> * FeatureTable: Rename entity_names to entities Signed-off-by: Terence <terencelimxp@gmail.com> * Revert Reorganise existing protos in CoreService by type as it make it hard for reviewers to review changes Signed-off-by: Terence <terencelimxp@gmail.com> * Add FeatureSource entity for native representation of FeatureSource protobuf Signed-off-by: Terence <terencelimxp@gmail.com> * Add missing nullable annotation on FeatureSource entity. Signed-off-by: Terence <terencelimxp@gmail.com> * Update ListFeatureTablesRequest's Filter to follow naming convention. Signed-off-by: Terence <terencelimxp@gmail.com> * Add missing serialization code for FeatureSource's field mapping. Signed-off-by: Terence <terencelimxp@gmail.com> * Split Feature proto from FeatureTable proto. Signed-off-by: Terence <terencelimxp@gmail.com> * Update FeatureTable entity_names field to entities Signed-off-by: Terence <terencelimxp@gmail.com> * Revert putting project in feature table spec Signed-off-by: Terence <terencelimxp@gmail.com> * Update ListFeatureTable Proto to return full FeatureTable objects and limit to listing from one Project. Signed-off-by: Terence <terencelimxp@gmail.com> * Fix typo in CoreService proto Signed-off-by: Terence <terencelimxp@gmail.com> * Add FeatureV2 core model to store FeatureSpecV2 Signed-off-by: Terence <terencelimxp@gmail.com> * Add FeatureTable core model to store FeatureTable protos Signed-off-by: Terence <terencelimxp@gmail.com> * Fix naming grammar in CoreService proto Signed-off-by: Terence <terencelimxp@gmail.com> * Standardise naming of specifying projects in CoreService proto Signed-off-by: Terence <terencelimxp@gmail.com> * Rename FeatureSource proto to FeatureSourceSpec for compatiblity. Signed-off-by: Terence <terencelimxp@gmail.com> * Update FeatureSource model to store type specific options as seperate columns instead of JSON. Signed-off-by: Terence <terencelimxp@gmail.com> * Add FeatureTableTest unit test to test FeatureTable core model Signed-off-by: Terence <terencelimxp@gmail.com> * Add FeatureTableValidator to validate FeatureTableSpec protobufs Signed-off-by: Terence <terencelimxp@gmail.com> * Add listFeatureTables(), applyFeatureTable() & getFeatureTable() to Core's SpecService Signed-off-by: Terence <terencelimxp@gmail.com> * Add FeatureTableRepository to save & retrieve FeatureTables in database. Signed-off-by: Terence <terencelimxp@gmail.com> * Fix hibernate errors on Feast Core boot. Signed-off-by: Terence <terencelimxp@gmail.com> * Implement listFeatureTables() , applyFeatureTable(), and getFeatureTable() in CoreServiceImpl Signed-off-by: Terence <terencelimxp@gmail.com> * Add applyFeatureSet integration tests SpecServiceIT Signed-off-by: Terence <terencelimxp@gmail.com> * Various fixes for creating FeatureTabes with applyFeatureTable Signed-off-by: Terence <terencelimxp@gmail.com> * Fixed bug with updating FeatureTable Signed-off-by: Terence <terencelimxp@gmail.com> * Update ListFeatureTables Signed-off-by: Terence <terencelimxp@gmail.com> * Add Python SDK Signed-off-by: Terence <terencelimxp@gmail.com> * Update GetFeatureTable Signed-off-by: Terence <terencelimxp@gmail.com> * Remove unused proto imports and generate go protos Signed-off-by: Terence <terencelimxp@gmail.com> * Fix ListFeatureTables IT Signed-off-by: Terence <terencelimxp@gmail.com> * Update comment to generalize FeatureSource's field mapping to all fields instead of just for feature. Signed-off-by: Terence <terencelimxp@gmail.com> * Fix feature table validator condition Signed-off-by: Terence <terencelimxp@gmail.com> * Fix feature table unit tests Signed-off-by: Terence <terencelimxp@gmail.com> * Update feature source proto Signed-off-by: Terence <terencelimxp@gmail.com> * Address PR comments Signed-off-by: Terence <terencelimxp@gmail.com> * Replace test with IT Signed-off-by: Terence <terencelimxp@gmail.com> * Update IT config Signed-off-by: Terence <terencelimxp@gmail.com> * Fix removal of entity check Signed-off-by: Terence <terencelimxp@gmail.com> * Fix test sort issue Signed-off-by: Terence <terencelimxp@gmail.com> * Store source options as json Signed-off-by: Terence <terencelimxp@gmail.com> * Update go protos Signed-off-by: Terence <terencelimxp@gmail.com> * Remove go FeatureSource proto Signed-off-by: Terence <terencelimxp@gmail.com> * Increase IT max pool size Signed-off-by: Terence <terencelimxp@gmail.com> * Reduce pool size instead Signed-off-by: Terence <terencelimxp@gmail.com> * Replace mutablemapping with dict Signed-off-by: Terence <terencelimxp@gmail.com> * Standardize use of timestamp_column instead of ts_column Signed-off-by: Terence <terencelimxp@gmail.com> Co-authored-by: Terence Lim <terencelimxp@gmail.com>
1 parent 87ee594 commit 442ca5a

35 files changed

+5913
-476
lines changed

‎common-test/src/main/java/feast/common/it/DataGenerator.java‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,16 @@
1717
package feast.common.it;
1818

1919
import com.google.common.collect.ImmutableList;
20+
import com.google.protobuf.Duration;
21+
import feast.proto.core.DataSourceProto.DataSource;
22+
import feast.proto.core.DataSourceProto.DataSource.BigQueryOptions;
23+
import feast.proto.core.DataSourceProto.DataSource.FileOptions;
24+
import feast.proto.core.DataSourceProto.DataSource.KafkaOptions;
2025
import feast.proto.core.EntityProto;
26+
import feast.proto.core.FeatureProto;
27+
import feast.proto.core.FeatureProto.FeatureSpecV2;
2128
import feast.proto.core.FeatureSetProto;
29+
import feast.proto.core.FeatureTableProto.FeatureTableSpec;
2230
import feast.proto.core.SourceProto;
2331
import feast.proto.core.StoreProto;
2432
import feast.proto.types.ValueProto;
@@ -130,6 +138,15 @@ public static EntityProto.EntitySpecV2 createEntitySpecV2(
130138
.build();
131139
}
132140

141+
public static FeatureProto.FeatureSpecV2 createFeatureSpecV2(
142+
String name, ValueProto.ValueType.Enum valueType, Map<String, String> labels) {
143+
return FeatureProto.FeatureSpecV2.newBuilder()
144+
.setName(name)
145+
.setValueType(valueType)
146+
.putAllLabels(labels)
147+
.build();
148+
}
149+
133150
public static FeatureSetProto.FeatureSet createFeatureSet(
134151
SourceProto.Source source,
135152
String projectName,
@@ -193,4 +210,65 @@ public static FeatureSetProto.FeatureSet createFeatureSet(
193210
return createFeatureSet(
194211
source, projectName, name, Collections.emptyMap(), Collections.emptyMap());
195212
}
213+
214+
// Create a Feature Table spec without DataSources configured.
215+
public static FeatureTableSpec createFeatureTableSpec(
216+
String name,
217+
List<String> entities,
218+
Map<String, ValueProto.ValueType.Enum> features,
219+
int maxAgeSecs,
220+
Map<String, String> labels) {
221+
222+
return FeatureTableSpec.newBuilder()
223+
.setName(name)
224+
.addAllEntities(entities)
225+
.addAllFeatures(
226+
features.entrySet().stream()
227+
.map(
228+
entry ->
229+
FeatureSpecV2.newBuilder()
230+
.setName(entry.getKey())
231+
.setValueType(entry.getValue())
232+
.putAllLabels(labels)
233+
.build())
234+
.collect(Collectors.toList()))
235+
.setMaxAge(Duration.newBuilder().setSeconds(3600).build())
236+
.putAllLabels(labels)
237+
.build();
238+
}
239+
240+
public static DataSource createFileDataSourceSpec(
241+
String fileURL, String fileFormat, String timestampColumn, String datePartitionColumn) {
242+
return DataSource.newBuilder()
243+
.setType(DataSource.SourceType.BATCH_FILE)
244+
.setFileOptions(
245+
FileOptions.newBuilder().setFileFormat(fileFormat).setFileUrl(fileURL).build())
246+
.setTimestampColumn(timestampColumn)
247+
.setDatePartitionColumn(datePartitionColumn)
248+
.build();
249+
}
250+
251+
public static DataSource createBigQueryDataSourceSpec(
252+
String bigQueryTableRef, String timestampColumn, String datePartitionColumn) {
253+
return DataSource.newBuilder()
254+
.setType(DataSource.SourceType.BATCH_BIGQUERY)
255+
.setBigqueryOptions(BigQueryOptions.newBuilder().setTableRef(bigQueryTableRef).build())
256+
.setTimestampColumn(timestampColumn)
257+
.setDatePartitionColumn(datePartitionColumn)
258+
.build();
259+
}
260+
261+
public static DataSource createKafkaDataSourceSpec(
262+
String servers, String topic, String classPath, String timestampColumn) {
263+
return DataSource.newBuilder()
264+
.setType(DataSource.SourceType.STREAM_KAFKA)
265+
.setKafkaOptions(
266+
KafkaOptions.newBuilder()
267+
.setTopic(topic)
268+
.setBootstrapServers(servers)
269+
.setClassPath(classPath)
270+
.build())
271+
.setTimestampColumn(timestampColumn)
272+
.build();
273+
}
196274
}

‎common-test/src/main/java/feast/common/it/SimpleCoreClient.java‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717
package feast.common.it;
1818

1919
import feast.proto.core.*;
20+
import feast.proto.core.CoreServiceProto.ApplyFeatureTableRequest;
21+
import feast.proto.core.FeatureTableProto.FeatureTableSpec;
2022
import java.util.Arrays;
2123
import java.util.Collections;
2224
import java.util.List;
@@ -75,6 +77,13 @@ public List<EntityProto.Entity> simpleListEntities(
7577
.getEntitiesList();
7678
}
7779

80+
public List<FeatureTableProto.FeatureTable> simpleListFeatureTables(
81+
CoreServiceProto.ListFeatureTablesRequest.Filter filter) {
82+
return stub.listFeatureTables(
83+
CoreServiceProto.ListFeatureTablesRequest.newBuilder().setFilter(filter).build())
84+
.getTablesList();
85+
}
86+
7887
public List<FeatureSetProto.FeatureSet> simpleListFeatureSets(
7988
String projectName, String featureSetName, Map<String, String> labels) {
8089
return stub.listFeatureSets(
@@ -131,6 +140,15 @@ public EntityProto.Entity simpleGetEntity(String projectName, String name) {
131140
.getEntity();
132141
}
133142

143+
public FeatureTableProto.FeatureTable simpleGetFeatureTable(String projectName, String name) {
144+
return stub.getFeatureTable(
145+
CoreServiceProto.GetFeatureTableRequest.newBuilder()
146+
.setName(name)
147+
.setProject(projectName)
148+
.build())
149+
.getTable();
150+
}
151+
134152
public void updateFeatureSetStatus(
135153
String projectName, String name, FeatureSetProto.FeatureSetStatus status) {
136154
stub.updateFeatureSetStatus(
@@ -190,4 +208,14 @@ public FeatureSetProto.FeatureSet getFeatureSet(String projectName, String featu
190208
.build())
191209
.getFeatureSet();
192210
}
211+
212+
public FeatureTableProto.FeatureTable applyFeatureTable(
213+
String projectName, FeatureTableSpec spec) {
214+
return stub.applyFeatureTable(
215+
ApplyFeatureTableRequest.newBuilder()
216+
.setProject(projectName)
217+
.setTableSpec(spec)
218+
.build())
219+
.getTable();
220+
}
193221
}

‎common-test/src/main/java/feast/common/util/TestUtil.java‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@
2121

2222
import feast.common.logging.AuditLogger;
2323
import feast.common.logging.config.LoggingProperties;
24+
import feast.proto.core.FeatureProto.FeatureSpecV2;
25+
import feast.proto.core.FeatureTableProto.FeatureTableSpec;
26+
import java.util.Comparator;
27+
import java.util.stream.Collectors;
2428
import org.springframework.boot.info.BuildProperties;
2529

2630
public class TestUtil {
@@ -37,4 +41,34 @@ public static void setupAuditLogger() {
3741

3842
new AuditLogger(loggingProperties, buildProperties);
3943
}
44+
45+
/**
46+
* Compare if two Feature Table specs are equal. Disregards order of features/entities in spec.
47+
*/
48+
public static boolean compareFeatureTableSpec(FeatureTableSpec spec, FeatureTableSpec otherSpec) {
49+
spec =
50+
spec.toBuilder()
51+
.clearFeatures()
52+
.addAllFeatures(
53+
spec.getFeaturesList().stream()
54+
.sorted(Comparator.comparing(FeatureSpecV2::getName))
55+
.collect(Collectors.toSet()))
56+
.clearEntities()
57+
.addAllEntities(spec.getEntitiesList().stream().sorted().collect(Collectors.toSet()))
58+
.build();
59+
60+
otherSpec =
61+
otherSpec
62+
.toBuilder()
63+
.clearFeatures()
64+
.addAllFeatures(
65+
spec.getFeaturesList().stream()
66+
.sorted(Comparator.comparing(FeatureSpecV2::getName))
67+
.collect(Collectors.toSet()))
68+
.clearEntities()
69+
.addAllEntities(spec.getEntitiesList().stream().sorted().collect(Collectors.toSet()))
70+
.build();
71+
72+
return spec.equals(otherSpec);
73+
}
4074
}

‎core/src/main/java/feast/core/dao/FeatureTableRepository.java‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
* Copyright 2018-2020 The Feast Authors
4+
*
5+
* Licensed under the Apache License, Version 2.0 (the "License");
6+
* you may not use this file except in compliance with the License.
7+
* You may obtain a copy of the License at
8+
*
9+
* https://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
package feast.core.dao;
18+
19+
import feast.core.model.FeatureTable;
20+
import java.util.List;
21+
import java.util.Optional;
22+
import org.springframework.data.jpa.repository.JpaRepository;
23+
24+
/** JPA repository for querying FeatureTables stored. */
25+
public interface FeatureTableRepository extends JpaRepository<FeatureTable, Long> {
26+
// Find single FeatureTable by project and name
27+
Optional<FeatureTable> findFeatureTableByNameAndProject_Name(String name, String projectName);
28+
29+
// Find FeatureTables by project
30+
List<FeatureTable> findAllByProject_Name(String projectName);
31+
}

‎core/src/main/java/feast/core/grpc/CoreServiceImpl.java‎

Lines changed: 102 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
import io.grpc.StatusRuntimeException;
3535
import io.grpc.stub.StreamObserver;
3636
import java.util.List;
37+
import java.util.NoSuchElementException;
3738
import java.util.stream.Collectors;
3839
import lombok.extern.slf4j.Slf4j;
3940
import net.devh.boot.grpc.server.service.GrpcService;
@@ -285,8 +286,8 @@ public void applyFeatureSet(
285286
String projectId = null;
286287

287288
try {
288-
FeatureSet featureSet = specService.imputeProjectName(request.getFeatureSet());
289-
projectId = featureSet.getSpec().getProject();
289+
FeatureSet featureSet = request.getFeatureSet();
290+
projectId = SpecService.resolveProjectName(featureSet.getSpec().getProject());
290291
authorizationService.authorizeRequest(SecurityContextHolder.getContext(), projectId);
291292
ApplyFeatureSetResponse response = specService.applyFeatureSet(featureSet);
292293
responseObserver.onNext(response);
@@ -391,4 +392,103 @@ public void listProjects(
391392
Status.INTERNAL.withDescription(e.getMessage()).withCause(e).asRuntimeException());
392393
}
393394
}
395+
396+
@Override
397+
public void applyFeatureTable(
398+
ApplyFeatureTableRequest request,
399+
StreamObserver<ApplyFeatureTableResponse> responseObserver) {
400+
String projectName = SpecService.resolveProjectName(request.getProject());
401+
String tableName = request.getTableSpec().getName();
402+
403+
try {
404+
// Check if user has authorization to apply feature table
405+
authorizationService.authorizeRequest(SecurityContextHolder.getContext(), projectName);
406+
407+
ApplyFeatureTableResponse response = specService.applyFeatureTable(request);
408+
responseObserver.onNext(response);
409+
responseObserver.onCompleted();
410+
} catch (AccessDeniedException e) {
411+
log.info(
412+
String.format(
413+
"ApplyFeatureTable: Not authorized to access project to apply: %s", projectName));
414+
responseObserver.onError(
415+
Status.PERMISSION_DENIED
416+
.withDescription(e.getMessage())
417+
.withCause(e)
418+
.asRuntimeException());
419+
} catch (org.hibernate.exception.ConstraintViolationException e) {
420+
log.error(
421+
String.format(
422+
"ApplyFeatureTable: Unable to apply Feature Table due to a conflict: "
423+
+ "Ensure that name is unique within Project: (name: %s, project: %s)",
424+
projectName, tableName));
425+
responseObserver.onError(
426+
Status.ALREADY_EXISTS.withDescription(e.getMessage()).withCause(e).asRuntimeException());
427+
} catch (IllegalArgumentException e) {
428+
log.error(
429+
String.format(
430+
"ApplyFeatureTable: Invalid apply Feature Table Request: (name: %s, project: %s)",
431+
projectName, tableName));
432+
responseObserver.onError(
433+
Status.INVALID_ARGUMENT
434+
.withDescription(e.getMessage())
435+
.withCause(e)
436+
.asRuntimeException());
437+
} catch (UnsupportedOperationException e) {
438+
log.error(
439+
String.format(
440+
"ApplyFeatureTable: Unsupported apply Feature Table Request: (name: %s, project: %s)",
441+
projectName, tableName));
442+
responseObserver.onError(
443+
Status.UNIMPLEMENTED.withDescription(e.getMessage()).withCause(e).asRuntimeException());
444+
} catch (Exception e) {
445+
log.error("ApplyFeatureTable Exception has occurred:", e);
446+
responseObserver.onError(
447+
Status.INTERNAL.withDescription(e.getMessage()).withCause(e).asRuntimeException());
448+
}
449+
}
450+
451+
@Override
452+
public void listFeatureTables(
453+
ListFeatureTablesRequest request,
454+
StreamObserver<ListFeatureTablesResponse> responseObserver) {
455+
try {
456+
ListFeatureTablesResponse response = specService.listFeatureTables(request.getFilter());
457+
responseObserver.onNext(response);
458+
responseObserver.onCompleted();
459+
} catch (IllegalArgumentException e) {
460+
log.error(String.format("ListFeatureTable: Invalid list Feature Table Request"));
461+
responseObserver.onError(
462+
Status.INVALID_ARGUMENT
463+
.withDescription(e.getMessage())
464+
.withCause(e)
465+
.asRuntimeException());
466+
} catch (Exception e) {
467+
log.error("ListFeatureTable: Exception has occurred: ", e);
468+
responseObserver.onError(
469+
Status.INTERNAL.withDescription(e.getMessage()).withCause(e).asRuntimeException());
470+
}
471+
}
472+
473+
@Override
474+
public void getFeatureTable(
475+
GetFeatureTableRequest request, StreamObserver<GetFeatureTableResponse> responseObserver) {
476+
try {
477+
GetFeatureTableResponse response = specService.getFeatureTable(request);
478+
479+
responseObserver.onNext(response);
480+
responseObserver.onCompleted();
481+
} catch (NoSuchElementException e) {
482+
log.error(
483+
String.format(
484+
"GetFeatureTable: No such Feature Table: (project: %s, name: %s)",
485+
request.getProject(), request.getName()));
486+
responseObserver.onError(
487+
Status.NOT_FOUND.withDescription(e.getMessage()).withCause(e).asRuntimeException());
488+
} catch (Exception e) {
489+
log.error("GetFeatureTable: Exception has occurred: ", e);
490+
responseObserver.onError(
491+
Status.INTERNAL.withDescription(e.getMessage()).withCause(e).asRuntimeException());
492+
}
493+
}
394494
}

0 commit comments

Comments
 (0)