Skip to content

Commit 16bb358

Browse files
pyalexpyalex
authored
Run JobCoontroller as separate application (feast-dev#951)
* moving jc * unify baseIT * move ingestion job related code to contrib * infra fix simple client * fix core rest it * revert bq changes * fix path * fix job coordinator it * fix path in docker * jc web port * bean validation from apache.bval * e2e auth * default jc_url * jc_ip in dataflow e2e * docker compose & health impl * core-host property * clean * more clean up * format * rename jc -> jobcontroller * format * jobcontroller docker compose config * pr comments * pr comments * label regex * fix version label
1 parent 7d8126b commit 16bb358

138 files changed

Lines changed: 3428 additions & 1616 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/complete.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,11 @@ jobs:
77
runs-on: [self-hosted]
88
strategy:
99
matrix:
10-
component: [core, serving, jupyter]
10+
component: [core, serving, jobcontroller, jupyter]
1111
env:
1212
GITHUB_PR_SHA: ${{ github.event.pull_request.head.sha }}
1313
REGISTRY: gcr.io/kf-feast
14-
MAVEN_CACHE: gs://feast-templocation-kf-feast/.m2.2019-10-24.tar
14+
MAVEN_CACHE: gs://feast-templocation-kf-feast/.m2.2020-08-19.tar
1515
steps:
1616
- uses: actions/checkout@v2
1717
- uses: GoogleCloudPlatform/github-actions/setup-gcloud@master

‎.github/workflows/master_only.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,9 @@ jobs:
1111
runs-on: [self-hosted]
1212
strategy:
1313
matrix:
14-
component: [core, serving, jupyter, ci]
14+
component: [core, serving, jobcontroller, jupyter, ci]
1515
env:
16-
MAVEN_CACHE: gs://feast-templocation-kf-feast/.m2.2019-10-24.tar
16+
MAVEN_CACHE: gs://feast-templocation-kf-feast/.m2.2020-08-19.tar
1717
steps:
1818
- uses: actions/checkout@v2
1919
- uses: GoogleCloudPlatform/github-actions/setup-gcloud@master

‎Makefile‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,12 +117,16 @@ build-push-docker:
117117
@$(MAKE) push-core-docker registry=$(REGISTRY) version=$(VERSION)
118118
@$(MAKE) push-serving-docker registry=$(REGISTRY) version=$(VERSION)
119119
@$(MAKE) push-ci-docker registry=$(REGISTRY) version=$(VERSION)
120+
@$(MAKE) push-jobcontroller-docker registry=$(REGISTRY) version=$(VERSION)
120121

121-
build-docker: build-core-docker build-serving-docker build-ci-docker
122+
build-docker: build-core-docker build-serving-docker build-ci-docker build-jobcontroller-docker
122123

123124
push-core-docker:
124125
docker push $(REGISTRY)/feast-core:$(VERSION)
125126

127+
push-jobcontroller-docker:
128+
docker push $(REGISTRY)/feast-jobcontroller:$(VERSION)
129+
126130
push-serving-docker:
127131
docker push $(REGISTRY)/feast-serving:$(VERSION)
128132

@@ -135,6 +139,9 @@ push-jupyter-docker:
135139
build-core-docker:
136140
docker build -t $(REGISTRY)/feast-core:$(VERSION) -f infra/docker/core/Dockerfile .
137141

142+
build-jobcontroller-docker:
143+
docker build -t $(REGISTRY)/feast-jobcontroller:$(VERSION) -f infra/docker/jobcontroller/Dockerfile .
144+
138145
build-serving-docker:
139146
docker build -t $(REGISTRY)/feast-serving:$(VERSION) -f infra/docker/serving/Dockerfile .
140147

‎common-test/pom.xml‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Copyright 2018-2020 The Feast Authors
4+
~
5+
~ Licensed under the Apache License, Version 2.0 (the "License");
6+
~ you may not use this file except in compliance with the License.
7+
~ You may obtain a copy of the License at
8+
~
9+
~ https://www.apache.org/licenses/LICENSE-2.0
10+
~
11+
~ Unless required by applicable law or agreed to in writing, software
12+
~ distributed under the License is distributed on an "AS IS" BASIS,
13+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
~ See the License for the specific language governing permissions and
15+
~ limitations under the License.
16+
~
17+
-->
18+
<project xmlns="http://maven.apache.org/POM/4.0.0"
19+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
20+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
21+
<modelVersion>4.0.0</modelVersion>
22+
23+
<parent>
24+
<artifactId>feast-parent</artifactId>
25+
<groupId>dev.feast</groupId>
26+
<version>${revision}</version>
27+
</parent>
28+
29+
<name>Feast Common Test</name>
30+
<description>Feast common module with test utilities</description>
31+
<artifactId>feast-common-test</artifactId>
32+
33+
<build>
34+
<plugins>
35+
<plugin>
36+
<groupId>org.apache.maven.plugins</groupId>
37+
<artifactId>maven-surefire-plugin</artifactId>
38+
<version>3.0.0-M4</version>
39+
<configuration>
40+
<argLine>-Xms2048m -Xmx2048m -Djdk.net.URLClassPath.disableClassPathURLCheck=true</argLine>
41+
</configuration>
42+
</plugin>
43+
</plugins>
44+
</build>
45+
46+
<dependencies>
47+
<dependency>
48+
<groupId>dev.feast</groupId>
49+
<artifactId>datatypes-java</artifactId>
50+
<version>${project.version}</version>
51+
<scope>compile</scope>
52+
</dependency>
53+
<dependency>
54+
<groupId>dev.feast</groupId>
55+
<artifactId>feast-common</artifactId>
56+
<version>${project.version}</version>
57+
<scope>compile</scope>
58+
</dependency>
59+
<dependency>
60+
<groupId>com.google.protobuf</groupId>
61+
<artifactId>protobuf-java-util</artifactId>
62+
</dependency>
63+
<!--compileOnly 'org.projectlombok:lombok:1.18.12'-->
64+
<dependency>
65+
<groupId>org.projectlombok</groupId>
66+
<artifactId>lombok</artifactId>
67+
<version>${lombok.version}</version>
68+
</dependency>
69+
<dependency>
70+
<groupId>javax.validation</groupId>
71+
<artifactId>validation-api</artifactId>
72+
</dependency>
73+
<dependency>
74+
<groupId>com.google.auto.value</groupId>
75+
<artifactId>auto-value-annotations</artifactId>
76+
</dependency>
77+
<dependency>
78+
<groupId>com.google.code.gson</groupId>
79+
<artifactId>gson</artifactId>
80+
</dependency>
81+
82+
<!-- Logging -->
83+
<dependency>
84+
<groupId>org.slf4j</groupId>
85+
<artifactId>slf4j-api</artifactId>
86+
</dependency>
87+
88+
<dependency>
89+
<groupId>org.hamcrest</groupId>
90+
<artifactId>hamcrest-library</artifactId>
91+
</dependency>
92+
<dependency>
93+
<groupId>org.springframework.boot</groupId>
94+
<artifactId>spring-boot-test</artifactId>
95+
<version>${spring.boot.version}</version>
96+
</dependency>
97+
<dependency>
98+
<groupId>org.springframework.boot</groupId>
99+
<artifactId>spring-boot-test-autoconfigure</artifactId>
100+
</dependency>
101+
<dependency>
102+
<groupId>org.springframework</groupId>
103+
<artifactId>spring-test</artifactId>
104+
<version>${spring.version}</version>
105+
</dependency>
106+
<dependency>
107+
<groupId>org.testcontainers</groupId>
108+
<artifactId>junit-jupiter</artifactId>
109+
<version>1.14.3</version>
110+
</dependency>
111+
<dependency>
112+
<groupId>org.testcontainers</groupId>
113+
<artifactId>postgresql</artifactId>
114+
<version>1.14.3</version>
115+
</dependency>
116+
<dependency>
117+
<groupId>org.testcontainers</groupId>
118+
<artifactId>kafka</artifactId>
119+
<version>1.14.3</version>
120+
</dependency>
121+
<dependency>
122+
<groupId>org.junit.jupiter</groupId>
123+
<artifactId>junit-jupiter-api</artifactId>
124+
<version>5.6.2</version>
125+
</dependency>
126+
<dependency>
127+
<groupId>org.springframework.kafka</groupId>
128+
<artifactId>spring-kafka</artifactId>
129+
</dependency>
130+
<dependency>
131+
<groupId>io.prometheus</groupId>
132+
<artifactId>simpleclient</artifactId>
133+
<version>0.8.0</version>
134+
</dependency>
135+
<dependency>
136+
<groupId>org.apache.commons</groupId>
137+
<artifactId>commons-lang3</artifactId>
138+
<version>3.4</version>
139+
</dependency>
140+
<dependency>
141+
<groupId>org.awaitility</groupId>
142+
<artifactId>awaitility</artifactId>
143+
<version>3.0.0</version>
144+
</dependency>
145+
<dependency>
146+
<groupId>org.awaitility</groupId>
147+
<artifactId>awaitility-proxy</artifactId>
148+
<version>3.0.0</version>
149+
</dependency>
150+
<dependency>
151+
<groupId>org.mockito</groupId>
152+
<artifactId>mockito-core</artifactId>
153+
<version>${mockito.version}</version>
154+
<scope>compile</scope>
155+
</dependency>
156+
<dependency>
157+
<groupId>io.rest-assured</groupId>
158+
<artifactId>rest-assured</artifactId>
159+
<version>4.2.0</version>
160+
</dependency>
161+
<dependency>
162+
<groupId>io.rest-assured</groupId>
163+
<artifactId>json-path</artifactId>
164+
<version>4.2.0</version>
165+
</dependency>
166+
<dependency>
167+
<groupId>io.rest-assured</groupId>
168+
<artifactId>xml-path</artifactId>
169+
<version>4.2.0</version>
170+
</dependency>
171+
</dependencies>
172+
</project>

core/src/test/java/feast/core/it/BaseIT.java renamed to common-test/src/main/java/feast/common/it/BaseIT.java

Lines changed: 28 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -14,37 +14,24 @@
1414
* See the License for the specific language governing permissions and
1515
* limitations under the License.
1616
*/
17-
package feast.core.it;
17+
package feast.common.it;
1818

19-
import feast.core.config.FeastProperties;
20-
import feast.core.util.KafkaSerialization;
21-
import feast.proto.core.IngestionJobProto;
2219
import io.prometheus.client.CollectorRegistry;
23-
import java.sql.Connection;
24-
import java.sql.SQLException;
25-
import java.sql.Statement;
20+
import java.sql.*;
21+
import java.util.ArrayList;
2622
import java.util.HashMap;
2723
import java.util.List;
2824
import java.util.Map;
29-
import java.util.stream.Collectors;
30-
import javax.persistence.EntityManager;
31-
import javax.persistence.PersistenceContext;
32-
import javax.persistence.Table;
3325
import org.apache.kafka.clients.consumer.ConsumerConfig;
34-
import org.apache.kafka.clients.producer.ProducerConfig;
3526
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
3627
import org.apache.kafka.common.serialization.StringDeserializer;
37-
import org.apache.kafka.common.serialization.StringSerializer;
38-
import org.hibernate.engine.spi.SessionImplementor;
3928
import org.junit.jupiter.api.*;
4029
import org.springframework.boot.test.context.SpringBootTest;
4130
import org.springframework.context.annotation.Bean;
4231
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
4332
import org.springframework.kafka.config.KafkaListenerContainerFactory;
4433
import org.springframework.kafka.core.ConsumerFactory;
4534
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
46-
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
47-
import org.springframework.kafka.core.KafkaTemplate;
4835
import org.springframework.kafka.listener.ConcurrentMessageListenerContainer;
4936
import org.springframework.test.annotation.DirtiesContext;
5037
import org.springframework.test.context.ActiveProfiles;
@@ -94,7 +81,7 @@ static void properties(DynamicPropertyRegistry registry) {
9481
public class SequentialFlow {
9582
@AfterAll
9683
public void tearDown() throws Exception {
97-
cleanTables(entityManager);
84+
cleanTables();
9885
}
9986
}
10087

@@ -118,60 +105,45 @@ public ConsumerFactory<String, byte[]> testConsumerFactory() {
118105
Map<String, Object> props = new HashMap<>();
119106

120107
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
121-
props.put(ConsumerConfig.GROUP_ID_CONFIG, "test");
108+
props.put(ConsumerConfig.GROUP_ID_CONFIG, this.getClass().getName());
122109

123110
return new DefaultKafkaConsumerFactory<>(
124111
props, new StringDeserializer(), new ByteArrayDeserializer());
125112
}
126-
127-
@Bean
128-
public KafkaTemplate<String, IngestionJobProto.FeatureSetSpecAck> specAckKafkaTemplate(
129-
FeastProperties feastProperties) {
130-
FeastProperties.StreamProperties streamProperties = feastProperties.getStream();
131-
Map<String, Object> props = new HashMap<>();
132-
133-
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
134-
135-
KafkaTemplate<String, IngestionJobProto.FeatureSetSpecAck> t =
136-
new KafkaTemplate<>(
137-
new DefaultKafkaProducerFactory<>(
138-
props, new StringSerializer(), new KafkaSerialization.ProtoSerializer<>()));
139-
t.setDefaultTopic(streamProperties.getSpecsOptions().getSpecsAckTopic());
140-
return t;
141-
}
142113
}
143114

144115
/**
145116
* Truncates all tables in Database (between tests or flows). Retries on deadlock
146117
*
147-
* @param em EntityManager
148118
* @throws SQLException
149119
*/
150-
public static void cleanTables(EntityManager em) throws SQLException {
151-
List<String> tableNames =
152-
em.getMetamodel().getEntities().stream()
153-
.map(e -> e.getJavaType().getAnnotation(Table.class).name())
154-
.collect(Collectors.toList());
155-
156-
// this trick needed to get EntityManager with Transaction
157-
// and we don't want to wrap whole class into @Transactional
158-
em = em.getEntityManagerFactory().createEntityManager();
159-
// Transaction needed only once to do unwrap
160-
SessionImplementor session = em.unwrap(SessionImplementor.class);
161-
162-
// and here we're actually don't want any transactions
163-
// but instead we pulling raw connection
164-
// to be able to retry query if needed
165-
// since retrying rollbacked transaction is not that easy
166-
Connection connection = session.connection();
120+
public static void cleanTables() throws SQLException {
121+
Connection connection =
122+
DriverManager.getConnection(
123+
postgreSQLContainer.getJdbcUrl(),
124+
postgreSQLContainer.getUsername(),
125+
postgreSQLContainer.getPassword());
126+
127+
List<String> tableNames = new ArrayList<>();
128+
Statement statement = connection.createStatement();
129+
ResultSet rs =
130+
statement.executeQuery(
131+
"SELECT table_name FROM information_schema.tables WHERE table_schema='public'");
132+
while (rs.next()) {
133+
tableNames.add(rs.getString(1));
134+
}
135+
136+
if (tableNames.isEmpty()) {
137+
return;
138+
}
167139

168140
// retries are needed since truncate require exclusive lock
169141
// and that often leads to Deadlock
170142
// since SpringApp is still running in another thread
171-
var num_retries = 5;
172-
for (var i = 1; i <= num_retries; i++) {
143+
int num_retries = 5;
144+
for (int i = 1; i <= num_retries; i++) {
173145
try {
174-
Statement statement = connection.createStatement();
146+
statement = connection.createStatement();
175147
statement.execute(String.format("truncate %s cascade", String.join(", ", tableNames)));
176148
} catch (SQLException e) {
177149
if (i == num_retries) {
@@ -184,8 +156,6 @@ public static void cleanTables(EntityManager em) throws SQLException {
184156
}
185157
}
186158

187-
@PersistenceContext EntityManager entityManager;
188-
189159
/** Used to determine SequentialFlows */
190160
public Boolean isSequentialTest(TestInfo testInfo) {
191161
try {
@@ -201,7 +171,7 @@ public void tearDown(TestInfo testInfo) throws Exception {
201171
CollectorRegistry.defaultRegistry.clear();
202172

203173
if (!isSequentialTest(testInfo)) {
204-
cleanTables(entityManager);
174+
cleanTables();
205175
}
206176
}
207177
}

core/src/test/java/feast/core/it/DataGenerator.java renamed to common-test/src/main/java/feast/common/it/DataGenerator.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
* See the License for the specific language governing permissions and
1515
* limitations under the License.
1616
*/
17-
package feast.core.it;
17+
package feast.common.it;
1818

1919
import com.google.common.collect.ImmutableList;
2020
import feast.proto.core.FeatureSetProto;

0 commit comments

Comments
 (0)