Terraform modules to deploy a cross-account centralized logging solution using Amazon OpenSearch Service with Cognito authentication and CloudWatch Subscription Filters.
- AWS Account: An active AWS account with the necessary permissions to create and manage the required resources.
- Terraform: Terraform (version 1.0.0 or higher) installed on your local machine.
- AWS CLI: AWS CLI (version 2.x) installed and configured with the appropriate credentials and region.
- Basic understanding of OpenSearch, Cognito, CloudWatch Subscription Filters, and Terraform.
- Create an AWS OpenSearch domain with Cognito authentication.
- Centralize logs from multiple AWS accounts using CloudWatch subscription filters and a Lambda function.
- A Lambda function that serves as a log forwarder that receives CloudWatch logs, processes them, and forwards the log data to an OpenSearch cluster.
Deploy the OpenSearch domain and Cognito authentication: This step sets up the OpenSearch domain with Cognito authentication and creates two Cognito user groups - one for admin users and another for limited users. Update the master_user_emails and limited_user_emails variables to manage user group membership. On initial deployment, only the master_users_role_arn backend role will be created on OpenSearch Dashboards.
module "central_logs" {
source = "../../"
# OpenSearch
ebs_volume_size = var.ebs_volume_size
ebs_volume_type = var.ebs_volume_type
ebs_iops = var.ebs_iops
instance_count = var.instance_count
instance_type = var.instance_type
engine_version = var.engine_version
automated_snapshot_start_hour = var.automated_snapshot_start_hour
dedicated_master_count = var.dedicated_master_count
dedicated_master_enabled = var.dedicated_master_enabled
dedicated_master_type = var.dedicated_master_type
domain_endpoint_options_enforce_https = var.domain_endpoint_options_enforce_https
domain_endpoint_options_tls_security_policy = var.domain_endpoint_options_tls_security_policy
encrypt_at_rest_enabled = var.encrypt_at_rest_enabled
zone_awareness_enabled = var.zone_awareness_enabled
availability_zone_count = var.availability_zone_count
warm_count = var.warm_count
warm_enabled = var.warm_enabled
warm_type = var.warm_type
node_to_node_encryption_enabled = var.node_to_node_encryption_enabled
# Cognito
master_user_emails = var.master_user_emails
limited_user_emails = var.limited_user_emails
master_user_group_name = var.master_user_group_name
limited_user_group_name = var.limited_user_group_name
advanced_security_options_enabled = var.advanced_security_options_enabled
mfa_configuration = var.mfa_configuration
context = module.this.context
}Create CloudWatch Subscription Filters to aggregate logs from multiple AWS accounts and use a unified Lambda function to send these logs to OpenSearch. The Lambda function receives a payload from CloudWatch Log Groups and use the information to ensure proper indexing on OpenSearch.
module "cloudwatch_subscription_filters" {
source = "../../modules/cloudwatch-subscription-filters"
central_logs_enable = var.central_logs_enable
central_logs_account_id = var.central_logs_account_id
central_logs_domain_name = var.central_logs_domain_name
central_logs_opensearch_arn = var.central_logs_opensearch_arn
central_logs_opensearch_endpoint = var.central_logs_opensearch_endpoint
destination_lambda_arn = var.destination_lambda_arn
filter_pattern = var.filter_pattern
log_group_name = var.log_group_name
subscription_filter_name = var.subscription_filter_name
context = module.this.context
}To configure limited user access: After initial deployment, manually add the
limited_users_role_arnto the desired role mapping in the OpenSearch Dashboards security settings using an admin user account.
| Name | Version |
|---|---|
| terraform | >= 1.0 |
| aws | ~> 4.60.0 |
| Name | Source | Version |
|---|---|---|
| terraform_null_label | cloudposse/terraform-null-label | 0.25.0 |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| enable_central_logs | A boolean flag to enable/disable central logs | bool |
true |
yes |
| limited_user_emails | Email addresses of the admin users of central logs Opensearch dashboard | list(string) |
[] |
yes |
| limited_user_group_name | User group name of limited users | string |
limited-users |
yes |
| master_user_emails | Email addresses of the admin users of central logs Opensearch dashboard | list(string) |
[] |
yes |
| master_user_group_name | User group name of master users | string |
master-users |
yes |
| instance_count | Number of data nodes in the cluster | number |
2 |
no |
| instance_type | The type of the instance | string |
t3.small.search |
no |
| engine_version | Version of OpenSearch engine to deploy | string |
OpenSearch_2.5 |
no |
| ebs_volume_size | EBS volumes for data storage in GB | number |
10 |
yes |
| ebs_volume_type | Storage type of EBS volumes | string |
gp2 |
no |
| ebs_iops | The baseline input/output (I/O) performance of EBS volumes attached to data nodes. Applicable only for the Provisioned IOPS EBS volume type | number |
0 |
no |
| encrypt_at_rest_enabled | Whether to enable encryption at rest | bool |
true |
no |
| advanced_security_options_enabled | AWS Elasticsearch Kibana enhanced security plugin enabling (forces new resource) | bool |
true |
no |
| domain_endpoint_options_enforce_https | Whether or not to require HTTPS | bool |
true |
no |
| domain_endpoint_options_tls_security_policy | The name of the TLS security policy that needs to be applied to the HTTPS endpoint | string |
Policy-Min-TLS-1-2-2019-07 |
no |
| dedicated_master_enabled | Indicates whether dedicated master nodes are enabled for the cluster | bool |
false |
no |
| dedicated_master_count | Number of dedicated master nodes in the cluster | number |
0 |
no |
| dedicated_master_type | Instance type of the dedicated master nodes in the cluster | string |
"t3.small.search" |
no |
| zone_awareness_enabled | Enable zone awareness for Elasticsearch cluster | bool |
true |
no |
| warm_enabled | Whether AWS UltraWarm is enabled | bool |
false |
no |
| warm_count | Number of UltraWarm nodes | number |
2 |
no |
| warm_type | Type of UltraWarm nodes | string |
"ultrawarm1.medium.search" |
no |
| availability_zone_count | Number of Availability Zones for the domain to use | number |
2 |
no |
| node_to_node_encryption_enabled | Whether to enable node-to-node encryption | bool |
true |
no |
| automated_snapshot_start_hour | Hour at which automated snapshots are taken, in UTC | number |
0 |
no |
| mfa_configuration | A boolean flag to enable or disable MFA configuration on Cognito. Note that enabling MFA requires requesting AWS support to exit the sandbox and move to Amazon SNS production environment, as well as increasing the spending limit. | bool |
false |
no |
| Name | Description |
|---|---|
| opensearch_dashboards_endpoint | The endpoint URL for the OpenSearch Dashboards. |
| opensearch_engine_version | The version of OpenSearch engine. |
| limited_users_role_arn | The Amazon Resource Name (ARN) of the IAM Role for limited users. |
As of April 2023, the AWS CLI does not natively support directly adding ISM Policies, roles, role mappings, or creating dashboards on a managed OpenSearch domain. Consequently, the Terraform AWS Provider is also unable to support these features.
However, you can interact with the OpenSearch configuration using the RESTful API provided by OpenSearch, which allows you to send HTTP requests to manage your domain. In our case where Cognito Authentication is used, it is necessary to use an AWS SDK (such as Boto3 for Python) to obtain an access token from Cognito before making requests to the OpenSearch API. This specific topic falls outside the scope of this solution for now. Please feel free to contribute.
MIT License. See LICENSE for full details.