Add fields to `azurerm_*_cost_management_export`: `partition_data` and `export_data_options.data_version` #26219

joshk0 · 2024-06-04T15:34:07Z

Community Note

Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
Please do not leave "+1" or "me too" comments, they generate extra noise for PR followers and do not help prioritize for review

Description

Add fields to azurerm_*_cost_management_export: partition_data and export_data_options.version.

export_data_options.version maps to export.definition.dataset.configuration.dataVersion - No default. Versions can proliferate and even though we can collect a large list of possible versions it's better to leave it free form.

partition_data maps to export.partition_data. Defaults to false.

PR Checklist

I have followed the guidelines in our Contributing Documentation.
I have checked to ensure there aren't other open Pull Requests for the same update/change.
I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
I have written new tests for my resource or datasource changes & updated any relevent documentation.
I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
(For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

azurerm_*_cost_management_export - support for the partition_data and export_data_options.data_version properties [Support for partition_data in azurerm_subscription_cost_management_export #23747]

This is a (please select all that apply):

Bug Fix
New Feature (ie adding a service, resource, or data source)
Enhancement
Breaking Change

Related Issue(s)

Fixes #23747

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

joshk0 · 2024-06-04T15:34:38Z

Please let me know if I need to run the live Azure acceptance tests myself in order to proceed.

joshk0 · 2024-06-12T14:28:32Z

Hello, is it possible to get a review for this?

katbyte

thanks for the pr @joshk0 - for booleans we add _enabled to the end of them to mark them as bools. once that change is made i think this is good to go

website/docs/r/billing_account_cost_management_export.html.markdown

joshk0 · 2024-06-22T17:27:33Z

@katbyte, done. Please have a look. Thank you!

joshk0 · 2024-07-03T14:58:42Z

I have rebased the branch to include the new go-azure-sdk. maybe it will work now?

Could someone please approve the workflows?

stephybun

Thanks for this PR @joshk0. There are some minor things that need fixing up to bring this more in-line with other resources in the provider.

Most resources have a basic and a complete test configuration, where the basic only includes properties that are Required and the complete configuration includes properties that are Optional. I see that we're missing a complete test config here, do you think you could look into adding that so we can validate the partition_data_enabled property?

internal/services/costmanagement/export_resource_base.go

website/docs/r/resource_group_cost_management_export.html.markdown

website/docs/r/subscription_cost_management_export.html.markdown

internal/services/costmanagement/export_billing_account_resource_test.go

internal/services/costmanagement/export_resource_group_resource_test.go

joshk0 · 2024-07-08T15:38:54Z

Hey there,

Most resources have a basic and a complete test configuration, where the basic only includes properties that are Required and the complete configuration includes properties that are Optional. I see that we're missing a complete test config here, do you think you could look into adding that so we can validate the partition_data_enabled property?

In other words, does this mean that the tests that already exist should in fact be left totally unmodified (currently we add data_version and partition_data_enabled values in the update flow, after merging your suggestions), and we add a new _complete test which does set those fields?

Or is it good to test the optional fields functioning properly in the update flow?

joshk0 · 2024-07-08T15:39:47Z

Sorry, both update and requiresImport test case mutate the new optional fields.

stephybun

Thanks for making those changes @joshk0. Unfortunately now none of the test configs have partition_data_enabled so we're unable to test this change. As mentioned in my previous comment, we cannot add this property to the basic configuration since it's optional, and it cannot be added to the update configuration since it would trigger a ForceNew. Can you please add a complete test configuration that include this property?

joshk0 · 2024-07-31T18:36:00Z

Thanks for making those changes @joshk0. Unfortunately now none of the test configs have partition_data_enabled so we're unable to test this change. As mentioned in my previous comment, we cannot add this property to the basic configuration since it's optional, and it cannot be added to the update configuration since it would trigger a ForceNew. Can you please add a complete test configuration that include this property?

Sure I was just asking a question and doing things that did not require an answer to that question (which you didn't directly answer, but did indirectly through hints in your answer.) I will add the test case now.

stephybun

Apologies @joshk0! I must have overlooked the question you left about the test when I did my re-review.

Thank you for adding the complete test configuration. I've kicked off the tests but we have some test failures:

------- Stdout: -------
=== RUN   TestAccResourceGroupCostManagementExport_complete
=== PAUSE TestAccResourceGroupCostManagementExport_complete
=== CONT  TestAccResourceGroupCostManagementExport_complete
    testcase.go:121: Step 1/2 error: Error running apply: exit status 1
        Error: creating Scoped Export (Scope: "/subscriptions/*******/resourceGroups/acctestRG-cm-240801080753627897"
        Export Name: "accrg240801080753627897"): unexpected status 400 (400 Bad Request) with error: BadRequest: Request properties validation failed: Invalid deliveryInfo destination rootFolderPath; value cannot begin with a forward slash
          with azurerm_resource_group_cost_management_export.test,
          on terraform_plugin_test.tf line 47, in resource "azurerm_resource_group_cost_management_export" "test":
          47: resource "azurerm_resource_group_cost_management_export" "test" {
        creating Scoped Export (Scope:
        "/subscriptions/*******/resourceGroups/acctestRG-cm-240801080753627897"
        Export Name: "accrg240801080753627897"): unexpected status 400 (400 Bad
        Request) with error: BadRequest: Request properties validation failed:
        Invalid deliveryInfo destination rootFolderPath; value cannot begin with a
        forward slash
--- FAIL: TestAccResourceGroupCostManagementExport_complete (132.73s)
FAIL

------- Stdout: -------
=== RUN   TestAccSubscriptionCostManagementExport_update
=== PAUSE TestAccSubscriptionCostManagementExport_update
=== CONT  TestAccSubscriptionCostManagementExport_update
    testcase.go:121: Step 1/6 error: Error running apply: exit status 1
        Error: creating Scoped Export (Scope: "/subscriptions/*******"
        Export Name: "accs240801080753607043"): unexpected status 400 (400 Bad Request) with error: BadRequest: Request properties validation failed: Export data version:  is not supported.
          with azurerm_subscription_cost_management_export.test,
          on terraform_plugin_test.tf line 52, in resource "azurerm_subscription_cost_management_export" "test":
          52: resource "azurerm_subscription_cost_management_export" "test" {
        creating Scoped Export (Scope:
        "/subscriptions/*******"
        Export Name: "accs240801080753607043"): unexpected status 400 (400 Bad
        Request) with error: BadRequest: Request properties validation failed: Export
        data version:  is not supported.
--- FAIL: TestAccSubscriptionCostManagementExport_update (157.92s)
FAIL

joshk0 · 2024-08-05T14:12:02Z

Hi @stephybun , just pushed some fixes that I think will work? If you could kick off the flows when you have a chance.

stephybun

@joshk0 unfortunately we're still getting the same failures:

=== CONT  TestAccResourceGroupCostManagementExport_basic
    testcase.go:121: Step 1/2 error: Error running apply: exit status 1
        Error: creating Scoped Export (Scope: "/subscriptions/*******/resourceGroups/acctestRG-cm-240807120359725390"
        Export Name: "accrg240807120359725390"): unexpected status 400 (400 Bad Request) with error: BadRequest: Request properties validation failed: Export data version:  is not supported.

…bled` and `export_data_options.version` `export_data_options.version` maps to `export.definition.dataset.configuration.dataVersion` No default. Versions can proliferate and even though we can collect a large list of possible versions it's better to leave it free form. `partition_data_enabled` maps to `export.partition_data`. Defaults to false.

joshk0 · 2024-08-07T13:42:02Z

@stephybun one more time please

joshk0 · 2024-08-07T15:50:34Z

I had to vendor so one more time again sorry @stephybun

stephybun

Still receiving the same error:

        Error: creating Scoped Export (Scope: "/subscriptions/*******"
        Export Name: "accs240808092720759225"): unexpected status 400 (400 Bad Request) with error: BadRequest: Request properties validation failed: Export data version:  is not supported.

I currently don't have the capacity to dig deeper into this, best guess based on the error message being returned by the API

Property isn't properly supported by the API, I had a brief look in the portal and I don't see an option for data version
Value being used for data version isn't valid/supported

joshk0 · 2024-08-08T14:48:48Z

Don't worry about trying to figure it out, I will just keep banging on it here just need the PRs to run

joshk0 · 2024-08-08T15:27:19Z

@stephybun
How do I run these tests myself? I reviewed the log of the unit tests and it seems like different unit tests are failing. I think I need to see the full output of the test run to better understand what I need to do.

joshk0 · 2024-08-08T15:27:38Z

BTW, this is the failure from the unit-test phase:

2024-08-08T09:17:26.6137210Z panic: test timed out after 30s
2024-08-08T09:17:26.6137509Z running tests:
2024-08-08T09:17:26.6137778Z    TestAccPluginSDKAndDecoder (30s)
2024-08-08T09:17:26.6138183Z    TestAccPluginSDKAndDecoderOptionalComputed (30s)
2024-08-08T09:17:26.6138723Z    TestAccPluginSDKAndDecoderOptionalComputedOverride (30s)
2024-08-08T09:17:26.6139208Z    TestAccPluginSDKAndDecoderSets (30s)
2024-08-08T09:17:26.6139560Z    TestAccPluginSDKAndEncoder (30s)
2024-08-08T09:17:26.6145361Z    TestAccPluginSDKReturnsComputedFields (30s)

joshk0 · 2024-08-08T15:30:24Z

Property isn't properly supported by the API, I had a brief look in the portal and I don't see an option for data version

Yes, this is something you can only do via the console today.

Value being used for data version isn't valid/supported

Perhaps it has to be mandatory. The reality is that in the portal you get to choose among a variety of different versions based on the export you are making.

flanakin · 2024-08-16T23:11:45Z

internal/services/costmanagement/export_billing_account_resource_test.go

@@ -39,6 +39,25 @@ func TestAccBillingAccountCostManagementExport_basic(t *testing.T) {
 	})
 }

+func TestAccBillingAccountCostManagementExport_complete(t *testing.T) {
+	if os.Getenv("ARM_BILLING_ACCOUNT") == "" {


nit: Subscriptions are supported as well

flanakin · 2024-08-16T23:18:20Z

internal/services/costmanagement/export_resource_base.go

+
+					"data_version": {
+						Type:         pluginsdk.TypeString,
+						Optional:     true,


I don't think this should be optional. I'm pretty sure it's required unless the API is defaulting.

flanakin · 2024-08-16T23:21:07Z

internal/services/costmanagement/client/client.go

@@ -6,10 +6,10 @@ package client
 import (
 	"fmt"

-	"github.com/hashicorp/go-azure-sdk/resource-manager/costmanagement/2021-10-01/exports"


I would probably keep the old version as well. I believe there are some gaps between what's supported in each version.

@jojohpm would know more.

If we have a provision to keep 2 versions, then we could keep both.
2021-10-01 = Exports v1. Legacy exports.
2023-07-01-preview = Exports v2. The new one with file partitioning, compression, file format, newer datasets etc... This is going to be the default experience going forward.

jojohpm

Added the comment.

jojohpm · 2024-08-16T23:43:09Z

internal/services/costmanagement/client/client.go

@@ -6,10 +6,10 @@ package client
 import (
 	"fmt"

-	"github.com/hashicorp/go-azure-sdk/resource-manager/costmanagement/2021-10-01/exports"


If we have a provision to keep 2 versions, then we could keep both.
2021-10-01 = Exports v1. Legacy exports.
2023-07-01-preview = Exports v2. The new one with file partitioning, compression, file format, newer datasets etc... This is going to be the default experience going forward.

wyattfry · 2025-01-02T16:51:24Z

Hello @joshk0 , could you fix the merge conflicts?

joshk0 changed the title ~~Exports add fieldsdd fields to azurerm_*_cost_management_export: partition_data and export_data_options.data_version~~ Add fields to azurerm_*_cost_management_export: partition_data and export_data_options.data_version Jun 4, 2024

github-actions bot added documentation dependencies service/cost-management service/systemcentervirtualmachinemanager size/XL labels Jun 4, 2024

joshk0 force-pushed the exports-add-fields branch from f945c9a to 6673a2e Compare June 4, 2024 17:12

github-actions bot added size/M and removed size/XL labels Jun 4, 2024

github-actions bot added the waiting-response label Jun 21, 2024

katbyte requested changes Jun 21, 2024

View reviewed changes

website/docs/r/billing_account_cost_management_export.html.markdown Outdated Show resolved Hide resolved

github-actions bot removed the waiting-response label Jun 22, 2024

github-actions bot added the waiting-response label Jun 24, 2024

joshk0 force-pushed the exports-add-fields branch from 2ec11a7 to e60895b Compare July 3, 2024 14:58

github-actions bot removed the waiting-response label Jul 3, 2024

stephybun requested changes Jul 8, 2024

View reviewed changes

github-actions bot added waiting-response and removed waiting-response labels Jul 8, 2024

joshk0 force-pushed the exports-add-fields branch from 3d1a83d to 86c9289 Compare July 8, 2024 15:41

github-actions bot added the waiting-response label Jul 31, 2024

stephybun requested changes Jul 31, 2024

View reviewed changes

github-actions bot removed the waiting-response label Jul 31, 2024

joshk0 force-pushed the exports-add-fields branch from 86c9289 to 578cc5f Compare July 31, 2024 21:03

github-actions bot added the waiting-response label Aug 1, 2024

stephybun requested changes Aug 1, 2024

View reviewed changes

github-actions bot removed the waiting-response label Aug 5, 2024

github-actions bot added the waiting-response label Aug 7, 2024

stephybun reviewed Aug 7, 2024

View reviewed changes

github-actions bot removed the waiting-response label Aug 7, 2024

joshk0 added 3 commits August 7, 2024 09:41

Fix tests

e63053a

One more test fix

286c206

joshk0 force-pushed the exports-add-fields branch from 6ad9b7d to 286c206 Compare August 7, 2024 13:41

github-actions bot added the waiting-response label Aug 7, 2024

vendor

a0d990b

github-actions bot removed the waiting-response label Aug 7, 2024

github-actions bot added the waiting-response label Aug 8, 2024

stephybun requested changes Aug 8, 2024

View reviewed changes

github-actions bot removed the waiting-response label Aug 8, 2024

flanakin reviewed Aug 16, 2024

View reviewed changes

jojohpm reviewed Aug 16, 2024

View reviewed changes

rcskosir added the enhancement label Oct 14, 2024

katbyte requested a review from a team as a code owner November 14, 2024 00:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fields to `azurerm_*_cost_management_export`: `partition_data` and `export_data_options.data_version` #26219

Add fields to `azurerm_*_cost_management_export`: `partition_data` and `export_data_options.data_version` #26219

joshk0 commented Jun 4, 2024

joshk0 commented Jun 4, 2024

joshk0 commented Jun 12, 2024

katbyte left a comment

joshk0 commented Jun 22, 2024

joshk0 commented Jul 3, 2024 •

edited

Loading

stephybun left a comment

joshk0 commented Jul 8, 2024

joshk0 commented Jul 8, 2024

stephybun left a comment

joshk0 commented Jul 31, 2024

stephybun left a comment

joshk0 commented Aug 5, 2024

stephybun left a comment

joshk0 commented Aug 7, 2024

joshk0 commented Aug 7, 2024

stephybun left a comment

joshk0 commented Aug 8, 2024

joshk0 commented Aug 8, 2024

joshk0 commented Aug 8, 2024

joshk0 commented Aug 8, 2024

flanakin Aug 16, 2024

flanakin Aug 16, 2024

flanakin Aug 16, 2024 •

edited

Loading

jojohpm Aug 16, 2024

jojohpm left a comment

jojohpm Aug 16, 2024

wyattfry commented Jan 2, 2025

Add fields to azurerm_*_cost_management_export: partition_data and export_data_options.data_version #26219

Are you sure you want to change the base?

Add fields to azurerm_*_cost_management_export: partition_data and export_data_options.data_version #26219

Conversation

joshk0 commented Jun 4, 2024

Community Note

Description

PR Checklist

Changes to existing Resource / Data Source

Testing

Change Log

Related Issue(s)

joshk0 commented Jun 4, 2024

joshk0 commented Jun 12, 2024

katbyte left a comment

Choose a reason for hiding this comment

joshk0 commented Jun 22, 2024

joshk0 commented Jul 3, 2024 • edited Loading

stephybun left a comment

Choose a reason for hiding this comment

joshk0 commented Jul 8, 2024

joshk0 commented Jul 8, 2024

stephybun left a comment

Choose a reason for hiding this comment

joshk0 commented Jul 31, 2024

stephybun left a comment

Choose a reason for hiding this comment

joshk0 commented Aug 5, 2024

stephybun left a comment

Choose a reason for hiding this comment

joshk0 commented Aug 7, 2024

joshk0 commented Aug 7, 2024

stephybun left a comment

Choose a reason for hiding this comment

joshk0 commented Aug 8, 2024

joshk0 commented Aug 8, 2024

joshk0 commented Aug 8, 2024

joshk0 commented Aug 8, 2024

flanakin Aug 16, 2024

Choose a reason for hiding this comment

flanakin Aug 16, 2024

Choose a reason for hiding this comment

flanakin Aug 16, 2024 • edited Loading

Choose a reason for hiding this comment

jojohpm Aug 16, 2024

Choose a reason for hiding this comment

jojohpm left a comment

Choose a reason for hiding this comment

jojohpm Aug 16, 2024

Choose a reason for hiding this comment

wyattfry commented Jan 2, 2025

Add fields to `azurerm_*_cost_management_export`: `partition_data` and `export_data_options.data_version` #26219

Add fields to `azurerm_*_cost_management_export`: `partition_data` and `export_data_options.data_version` #26219

joshk0 commented Jul 3, 2024 •

edited

Loading

flanakin Aug 16, 2024 •

edited

Loading