Skip to content

Azure example terraform #1274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jklegar merged 6 commits into from
Jan 27, 2021
Merged

Azure example terraform #1274

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0eeaa65
example terraform for azure
jklegar Dec 29, 2020
b21302b
azure tf adjustments
jklegar Jan 14, 2021
3717711
more adjustments
jklegar Jan 19, 2021
f7a2ce9
changes to example notebook
jklegar Jan 19, 2021
f438086
add default aks namespace
jklegar Jan 20, 2021
5e93fbb
remove extra print statement
jklegar Jan 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
more adjustments 
Signed-off-by: Jacob Klegar <jacob@tecton.ai>
  • Loading branch information
@jklegar
jklegar committed Jan 27, 2021
commit 37177114606a38e5841193726d580a349704e4d9
4 changes: 2 additions & 2 deletions infra/terraform/azure/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ This serves as a guide on how to deploy Feast on Azure. At the end of this guide
2. Feast services running on AKS
3. Azure Cache (Redis) as online store
4. Spark operator on AKS
5. Kafka running on AKS.
5. Kafka running on HDInsight.
Copy link
Copy Markdown
Member

@woop woop Jan 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, is there a reason we are using HDInsights over Event Hubs for Kafka?

Copy link
Copy Markdown
Collaborator Author

@jklegar jklegar Jan 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HDInsight actually runs Kafka (it's essentially Kafka managed by Azure), whereas Event Hubs is Kafka-client-compatible but doesn't support certain Kafka features. Seemed faster and more likely to work using HDInsight


# Steps

1. Create a tfvars file, e.g. `my.tfvars`. A sample configuration is as below:

```
name_prefix = "feast-0-9"
name_prefix = "feast09"
resource_group = "Feast" # pre-exisiting resource group
aks_namespace = "default"
```
Expand Down
3 changes: 1 addition & 2 deletions infra/terraform/azure/helm.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,15 +50,14 @@ locals {
envOverrides = {
feast_redis_host = azurerm_redis_cache.main.hostname,
feast_redis_port = azurerm_redis_cache.main.ssl_port,
feast_redis_ssl = true
feast_spark_launcher = "k8s"
feast_spark_staging_location = "wasbs://${azurerm_storage_container.staging.name}@${azurerm_storage_account.main.name}.blob.core.windows.net/artifacts/"
feast_historical_feature_output_location : "wasbs://${azurerm_storage_container.staging.name}@${azurerm_storage_account.main.name}.blob.core.windows.net/out/"
feast_historical_feature_output_format : "parquet"
demo_kafka_brokers : azurerm_hdinsight_kafka_cluster.main.https_endpoint
demo_data_location : "wasbs://${azurerm_storage_container.staging.name}@${azurerm_storage_account.main.name}.blob.core.windows.net/test-data/"
feast_azure_blob_account_name = azurerm_storage_account.main.name
feast_azure_blob_account_access_key = azurerm_storage_account.main.primary_access_key
feast_spark_ingestion_jar = "./feast/spark/ingestion/target/feast-ingestion-spark-develop.jar"
}
}
}
Expand Down
15 changes: 15 additions & 0 deletions infra/terraform/azure/kafka.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,24 @@ resource "azurerm_hdinsight_kafka_cluster" "main" {
vm_size = var.kafka_head_vm_size
username = "${var.name_prefix}-kafka-user"
password = random_password.feast-kafka-role-password.result
subnet_id = azurerm_subnet.kafka.id
virtual_network_id = azurerm_virtual_network.main.id
}
worker_node {
vm_size = var.kafka_worker_vm_size
username = "${var.name_prefix}-kafka-user"
password = random_password.feast-kafka-role-password.result
number_of_disks_per_node = var.kafka_worker_disks_per_node
target_instance_count = var.kafka_worker_target_instance_count
subnet_id = azurerm_subnet.kafka.id
virtual_network_id = azurerm_virtual_network.main.id
}
zookeeper_node {
vm_size = var.kafka_zookeeper_vm_size
username = "${var.name_prefix}-kafka-user"
password = random_password.feast-kafka-role-password.result
subnet_id = azurerm_subnet.kafka.id
virtual_network_id = azurerm_virtual_network.main.id
}
}
}
Expand All @@ -58,3 +64,12 @@ resource "random_password" "feast-kafka-gateway-password" {
min_special = 1
min_numeric = 1
}

resource "kubernetes_secret" "feast-kafka-gateway-secret" {
metadata {
name = "feast-kafka-gateway"
}
data = {
kafka-gateway-password = random_password.feast-kafka-gateway-password.result
}
}
29 changes: 12 additions & 17 deletions infra/terraform/azure/provider.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,26 @@ provider "azurerm" {
features {}
}

data "azurerm_kubernetes_cluster" "main" {
name = "${var.name_prefix}-aks"
resource_group_name = data.azurerm_resource_group.main.name
}

provider "helm" {
version = "~> 1.3.2"
kubernetes {
host = "${data.azurerm_kubernetes_cluster.main.kube_config.0.host}"
username = "${data.azurerm_kubernetes_cluster.main.kube_config.0.username}"
password = "${data.azurerm_kubernetes_cluster.main.kube_config.0.password}"
client_certificate = "${base64decode(data.azurerm_kubernetes_cluster.main.kube_config.0.client_certificate)}"
client_key = "${base64decode(data.azurerm_kubernetes_cluster.main.kube_config.0.client_key)}"
cluster_ca_certificate = "${base64decode(data.azurerm_kubernetes_cluster.main.kube_config.0.cluster_ca_certificate)}"
host = azurerm_kubernetes_cluster.main.kube_config.0.host
username = azurerm_kubernetes_cluster.main.kube_config.0.username
password = azurerm_kubernetes_cluster.main.kube_config.0.password
client_certificate = base64decode(azurerm_kubernetes_cluster.main.kube_config.0.client_certificate)
client_key = base64decode(azurerm_kubernetes_cluster.main.kube_config.0.client_key)
cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.main.kube_config.0.cluster_ca_certificate)
load_config_file = false
}
}

provider "kubernetes" {
version = "~> 1.13.3"
host = "${data.azurerm_kubernetes_cluster.main.kube_config.0.host}"
username = "${data.azurerm_kubernetes_cluster.main.kube_config.0.username}"
password = "${data.azurerm_kubernetes_cluster.main.kube_config.0.password}"
client_certificate = "${base64decode(data.azurerm_kubernetes_cluster.main.kube_config.0.client_certificate)}"
client_key = "${base64decode(data.azurerm_kubernetes_cluster.main.kube_config.0.client_key)}"
cluster_ca_certificate = "${base64decode(data.azurerm_kubernetes_cluster.main.kube_config.0.cluster_ca_certificate)}"
host = azurerm_kubernetes_cluster.main.kube_config.0.host
username = azurerm_kubernetes_cluster.main.kube_config.0.username
password = azurerm_kubernetes_cluster.main.kube_config.0.password
client_certificate = base64decode(azurerm_kubernetes_cluster.main.kube_config.0.client_certificate)
client_key = base64decode(azurerm_kubernetes_cluster.main.kube_config.0.client_key)
cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.main.kube_config.0.cluster_ca_certificate)
load_config_file = false
}
8 changes: 6 additions & 2 deletions infra/terraform/azure/redis.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ resource "azurerm_redis_cache" "main" {
location = data.azurerm_resource_group.main.location
resource_group_name = data.azurerm_resource_group.main.name
capacity = var.redis_capacity
family = "C"
sku_name = "Standard"
family = "P"
sku_name = "Premium"
redis_configuration {
enable_authentication = false
}
subnet_id = azurerm_subnet.redis.id
}
8 changes: 0 additions & 8 deletions infra/terraform/azure/versions.tf
View file
Open in desktop

This file was deleted.

18 changes: 16 additions & 2 deletions infra/terraform/azure/vnet.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,22 @@ resource "azurerm_virtual_network" "main" {
}

resource "azurerm_subnet" "main" {
name = "${var.name_prefix}-subnet"
name = "${var.name_prefix}-aks-subnet"
resource_group_name = data.azurerm_resource_group.main.name
virtual_network_name = azurerm_virtual_network.main.name
address_prefixes = ["10.1.0.0/16"]
address_prefixes = ["10.1.0.0/24"]
}

resource "azurerm_subnet" "redis" {
name = "${var.name_prefix}-redis-subnet"
resource_group_name = data.azurerm_resource_group.main.name
virtual_network_name = azurerm_virtual_network.main.name
address_prefixes = ["10.1.128.0/24"]
}

resource "azurerm_subnet" "kafka" {
name = "${var.name_prefix}-kafka-subnet"
resource_group_name = data.azurerm_resource_group.main.name
virtual_network_name = azurerm_virtual_network.main.name
address_prefixes = ["10.1.64.0/24"]
}