Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added mirroring support #327

Merged
merged 34 commits into from
Nov 29, 2024
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
fc93ca8
feat: added mirroring support
Ak-sky Oct 29, 2024
87a2643
feat: added mirroring support
Ak-sky Oct 29, 2024
a2a6373
feat: added mirroring support
Ak-sky Oct 29, 2024
c658f17
added mirroring test
Ak-sky Oct 29, 2024
62a3e2d
Merge branch 'main' into mirror
Ak-sky Nov 5, 2024
ed32059
Merge remote-tracking branch 'origin/main' into mirror
Ak-sky Nov 13, 2024
087f175
Merge branch 'main' into mirror
Ak-sky Nov 18, 2024
e966f58
added mirroring feat
Ak-sky Nov 19, 2024
31dbb39
added mirroring feat
Ak-sky Nov 19, 2024
05b231e
added mirroring feat
Ak-sky Nov 19, 2024
f8c66a9
minor fixes
Ak-sky Nov 19, 2024
f7a54d7
minor fixes
Ak-sky Nov 19, 2024
853574f
minor fixes
Ak-sky Nov 19, 2024
3f1117b
removed mirroring ex
Ak-sky Nov 20, 2024
cdbd699
resolved conflicts
Ak-sky Nov 20, 2024
e3b4b70
added tests
Ak-sky Nov 20, 2024
8843366
minor fixes
Ak-sky Nov 21, 2024
a51810c
fix auth policy var exposure
Ak-sky Nov 21, 2024
38becf9
minor fixes
Ak-sky Nov 21, 2024
1016854
updated mirroring variable
Ak-sky Nov 21, 2024
88dc2e1
updated mirroring variable description
Ak-sky Nov 21, 2024
899fcba
updated mirroring variable description
Ak-sky Nov 21, 2024
122ac00
minor fixes
Ak-sky Nov 22, 2024
2b02a8a
minor fixes
Ak-sky Nov 22, 2024
4639bc8
auth policy fixes
Ak-sky Nov 22, 2024
45c7db6
auth policy fixes
Ak-sky Nov 22, 2024
849c925
Merge remote-tracking branch 'origin/main' into mirror
Ak-sky Nov 24, 2024
63e6d15
updated test
Ak-sky Nov 24, 2024
8513199
fscloud in schematics
Ak-sky Nov 26, 2024
8849900
updated mirroring variable in fscloud
Ak-sky Nov 26, 2024
1f2dae1
addressed review comments
Ak-sky Nov 29, 2024
45b566b
addressed review comments
Ak-sky Nov 29, 2024
8a578f0
addressed review comments
Ak-sky Nov 29, 2024
3ceb859
addressed review comments
Ak-sky Nov 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ The Event Streams service supports payload data encryption that uses a root key
* [Basic example](./examples/basic)
* [Complete example with topics and schema creation.](./examples/complete)
* [Financial Services Cloud profile example](./examples/fscloud)
* [Mirroring example](./examples/mirroring)
* [Contributing](#contributing)
<!-- END OVERVIEW HOOK -->

Expand Down Expand Up @@ -106,26 +107,32 @@ You need the following permissions to run this module.
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
| <a name="requirement_ibm"></a> [ibm](#requirement\_ibm) | >= 1.70.0, <2.0.0 |
| <a name="requirement_ibm"></a> [ibm](#requirement\_ibm) | >= 1.71.0, <2.0.0 |
| <a name="requirement_time"></a> [time](#requirement\_time) | >= 0.9.1 |

### Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_cbr_rule"></a> [cbr\_rule](#module\_cbr\_rule) | terraform-ibm-modules/cbr/ibm//modules/cbr-rule-module | 1.28.1 |
| <a name="module_es_guid_crn_parser"></a> [es\_guid\_crn\_parser](#module\_es\_guid\_crn\_parser) | terraform-ibm-modules/common-utilities/ibm//modules/crn-parser | 1.1.0 |

### Resources

| Name | Type |
|------|------|
| [ibm_event_streams_mirroring_config.es_mirroring_config](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/event_streams_mirroring_config) | resource |
| [ibm_event_streams_schema.es_schema](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/event_streams_schema) | resource |
| [ibm_event_streams_topic.es_topic](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/event_streams_topic) | resource |
| [ibm_iam_authorization_policy.en_service_policy](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/iam_authorization_policy) | resource |
| [ibm_iam_authorization_policy.kms_policy](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/iam_authorization_policy) | resource |
| [ibm_iam_authorization_policy.mirroring_policy](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/iam_authorization_policy) | resource |
| [ibm_resource_instance.es_instance](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/resource_instance) | resource |
| [ibm_resource_key.service_credentials](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/resource_key) | resource |
| [ibm_resource_tag.es_access_tag](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/resource_tag) | resource |
| [time_sleep.wait_for_authorization_policy](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource |
| [time_sleep.wait_for_en_service_policy](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource |
| [time_sleep.wait_for_kms_authorization_policy](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource |
| [time_sleep.wait_for_mirroring_authorization_policy](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource |

### Inputs

Expand All @@ -140,6 +147,9 @@ You need the following permissions to run this module.
| <a name="input_kms_encryption_enabled"></a> [kms\_encryption\_enabled](#input\_kms\_encryption\_enabled) | Set this to true to control the encryption keys used to encrypt the data that you store in IBM Cloud® Databases. If set to false, the data is encrypted by using randomly generated keys. For more info on Key Protect integration, see https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-key-protect. For more info on HPCS integration, see https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs | `bool` | `false` | no |
| <a name="input_kms_key_crn"></a> [kms\_key\_crn](#input\_kms\_key\_crn) | The root key CRN of the key management service (Key Protect or Hyper Protect Crypto Services) to use to encrypt the payload data. [Learn more](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-managing_encryption) about integrating Hyper Protect Crypto Services with Event Streams. | `string` | `null` | no |
| <a name="input_metrics"></a> [metrics](#input\_metrics) | Enhanced metrics to activate, as list of strings. Only allowed for enterprise plans. Allowed values: 'topic', 'partition', 'consumers'. | `list(string)` | `[]` | no |
| <a name="input_mirroring"></a> [mirroring](#input\_mirroring) | Mirroring configuration | <pre>object({<br/> source_crn = string<br/> source_alias = string<br/> target_alias = string<br/> })</pre> | `null` | no |
| <a name="input_mirroring_enabled"></a> [mirroring\_enabled](#input\_mirroring\_enabled) | Set this to true to enable mirroring. Mirroring enables messages in one Event Streams service instance to be continuously copied to a second instance to increase resiliency. See https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-mirroring. | `bool` | `false` | no |
| <a name="input_mirroring_topic_patterns"></a> [mirroring\_topic\_patterns](#input\_mirroring\_topic\_patterns) | The list of the topics to set in instance. Required only if var.mirroring\_enabled is set to true. | `list(string)` | `null` | no |
| <a name="input_plan"></a> [plan](#input\_plan) | The plan for the Event Streams instance. Possible values: `lite`, `standard`, `enterprise-3nodes-2tb`. | `string` | `"standard"` | no |
| <a name="input_region"></a> [region](#input\_region) | The region where the Event Streams are created. | `string` | `"us-south"` | no |
| <a name="input_resource_group_id"></a> [resource\_group\_id](#input\_resource\_group\_id) | The resource group ID where the Event Streams instance is created. | `string` | n/a | yes |
Expand All @@ -163,6 +173,8 @@ You need the following permissions to run this module.
| <a name="output_kafka_broker_version"></a> [kafka\_broker\_version](#output\_kafka\_broker\_version) | The Kafka version |
| <a name="output_kafka_brokers_sasl"></a> [kafka\_brokers\_sasl](#output\_kafka\_brokers\_sasl) | (Array of Strings) Kafka brokers use for interacting with Kafka native API |
| <a name="output_kafka_http_url"></a> [kafka\_http\_url](#output\_kafka\_http\_url) | The API endpoint to interact with Event Streams REST API |
| <a name="output_mirroring_config_id"></a> [mirroring\_config\_id](#output\_mirroring\_config\_id) | The ID of the mirroring config in CRN format |
| <a name="output_mirroring_topic_patterns"></a> [mirroring\_topic\_patterns](#output\_mirroring\_topic\_patterns) | Mirroring topic patterns |
| <a name="output_service_credentials_json"></a> [service\_credentials\_json](#output\_service\_credentials\_json) | The service credentials JSON map. |
| <a name="output_service_credentials_object"></a> [service\_credentials\_object](#output\_service\_credentials\_object) | The service credentials object. |
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
Expand Down
2 changes: 1 addition & 1 deletion examples/basic/version.tf
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ terraform {
required_providers {
ibm = {
source = "IBM-Cloud/ibm"
version = "= 1.70.0"
version = "= 1.71.0"
}
}
}
4 changes: 2 additions & 2 deletions examples/complete/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ variable "region" {
variable "prefix" {
type = string
description = "The prefix to apply to all resources created by this example."
default = "event-streams-com"
default = "geretain-sky"
Ak-sky marked this conversation as resolved.
Show resolved Hide resolved
}

variable "resource_group" {
type = string
description = "An existing resource group name to use for this example. If not specified, a new resource group is created."
default = null
default = "geretain-aakash-rg"
Ak-sky marked this conversation as resolved.
Show resolved Hide resolved
}

variable "resource_tags" {
Expand Down
2 changes: 1 addition & 1 deletion examples/complete/version.tf
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
# Use latest version of provider in non-basic examples to verify latest version works with module
ibm = {
source = "IBM-Cloud/ibm"
version = ">= 1.70.0"
version = ">= 1.71.0"
}
}
}
10 changes: 10 additions & 0 deletions examples/fscloud/outputs.tf
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,13 @@ output "service_credentials_object" {
value = module.event_streams.service_credentials_object
sensitive = true
}

output "mirroring_config_id" {
description = "The ID of the mirroring config in CRN format"
value = module.event_streams.mirroring_config_id
}

output "mirroring_topic_patterns" {
description = "Mirroring topic patterns"
value = module.event_streams.mirroring_topic_patterns
}
2 changes: 1 addition & 1 deletion examples/fscloud/version.tf
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
# Use latest version of provider in non-basic examples to verify latest version works with module
ibm = {
source = "IBM-Cloud/ibm"
version = ">= 1.70.1"
version = ">= 1.71.0"
}
}
}
9 changes: 9 additions & 0 deletions examples/mirroring/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Mirroring example

An end-to-end example that creates an IBM Event Streams for IBM Cloud instance and a mirroring instance.

This example uses the IBM Cloud Terraform provider to create the following infrastructure.

- A new resource group, if one is not passed in.
- A instance of Event Streams in the provided resource group and region.
- A instance of Event Streams in the provided resource group and region with mirroring enabled.
42 changes: 42 additions & 0 deletions examples/mirroring/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
##############################################################################
# Resource Group
##############################################################################

module "resource_group" {
source = "terraform-ibm-modules/resource-group/ibm"
version = "1.1.6"
# if an existing resource group is not set (null) create a new one using prefix
resource_group_name = var.resource_group == null ? "${var.prefix}-resource-group" : null
existing_resource_group_name = var.resource_group
}

##############################################################################
# Events-streams-source-instance
##############################################################################
module "source_event_streams" {
source = "../../"
resource_group_id = module.resource_group.resource_group_id
es_name = "${var.prefix}-es"
tags = var.resource_tags
plan = "enterprise-3nodes-2tb"
}

##############################################################################
# Events-streams-mirroring-instance
##############################################################################

module "event_streams_mirror" {
depends_on = [module.source_event_streams]
source = "../../"
resource_group_id = module.resource_group.resource_group_id
es_name = "${var.prefix}-mirror"
tags = var.resource_tags
plan = "enterprise-3nodes-2tb"
mirroring_enabled = true
mirroring_topic_patterns = ["topic-1", "topic-2"]
mirroring = {
source_crn = module.source_event_streams.crn
source_alias = "source-alias"
target_alias = "target-alias"
}
}
48 changes: 48 additions & 0 deletions examples/mirroring/outputs.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
##############################################################################
# Outputs
##############################################################################

output "resource_group_name" {
description = "Resource group name"
value = module.resource_group.resource_group_name
}

output "resource_group_id" {
description = "Resource group ID"
value = module.resource_group.resource_group_id
}

output "crn" {
description = "Event Streams mirror instance crn"
value = module.event_streams_mirror.crn
}

output "guid" {
description = "Event Streams instance guid"
value = module.event_streams_mirror.guid
}

output "kafka_brokers_sasl" {
description = "(Array of Strings) Kafka brokers use for interacting with Kafka native API"
value = module.event_streams_mirror.kafka_brokers_sasl
}

output "kafka_http_url" {
description = "The API endpoint to interact with Event Streams REST API"
value = module.event_streams_mirror.kafka_http_url
}

output "kafka_broker_version" {
description = "The Kafka version"
value = module.event_streams_mirror.kafka_broker_version
}

output "mirroring_config_id" {
description = "The ID of the mirroring config in CRN format"
value = module.event_streams_mirror.mirroring_config_id
}

output "mirroring_topic_patterns" {
description = "Mirroring topic patterns"
value = module.event_streams_mirror.mirroring_topic_patterns
}
4 changes: 4 additions & 0 deletions examples/mirroring/provider.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
provider "ibm" {
ibmcloud_api_key = var.ibmcloud_api_key
region = var.region
}
29 changes: 29 additions & 0 deletions examples/mirroring/variables.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
variable "ibmcloud_api_key" {
type = string
description = "The IBM Cloud API key."
sensitive = true
}

variable "region" {
type = string
description = "The region where the Event Streams mirroring instance is created."
default = "us-south"
}

variable "prefix" {
type = string
description = "The prefix to apply to all resources created by this example."
default = "event-streams"
}

variable "resource_group" {
type = string
description = "An existing resource group name to use for this example. If not specified, a new resource group is created."
default = null
}

variable "resource_tags" {
type = list(string)
description = "The list of tags associated with the Event Steams instance."
default = []
}
9 changes: 9 additions & 0 deletions examples/mirroring/version.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
terraform {
required_version = ">= 1.3.0"
required_providers {
ibm = {
source = "IBM-Cloud/ibm"
version = ">= 1.71.0"
}
}
}
Loading