Downloaded State files are in the wrong format to use 'terraform state push ...'

[jderieg]

Bug description 🐞

We have a user who downloaded a state file from Terrakube through the gui, modified it locally, and then tried to use a 'terraform state push' to update it back to the workspace in Terrakube. This, however, did not work because apparently the format that Terrakube uses to publish the json for the state file does not match what Terraform is expecting. Specifically, the state file in the gui is missing header items such as 'version' and 'serial'.
I looked in the S3 bucket where the files are kept, and it looks like Terrakube stores the latest version of the state file in the correct format as 'terraform.tfstate', but then the historical state files in a subdirectory called 'state'. The historical state files do not seem to be in the correct format, but that is what's shown and available for download when looking at the state in the GUI.

Steps to reproduce

Use the 'download' button to download a state file in Terrakube. Then from a cli, do a 'terraform state pull' for the same state file. Compare the json. They are different.

Expected behavior

The downloaded state file should be able to be used to do a 'terraform state push'. It should be in the format that Terraform recognizes as a valid state file.

Example repository

No response

Anything else?

No response

[alfespa17]

Hello @jderieg

That won't work because what you see in the UI is the json that you get when you run "terraform show -json", the structure of that json is different than the state file.

You could create a simple terraform file locally, run terraform apply and run terraform show -json, once you have that file you can open the state file locally with notepad and you will see that there are some differences

[alfespa17]

As an alternative you could download the state file using the following endpoint using the API

{{TERRAKUBE-API}}/remote/tfe/v2/workspaces/{workspaceId}/current-state-version

Terrakube partially implement the TFC API, we have some methods that are used when you use the remote state backend or the cloud block

You will find this field "hosted-state-download-url"

The methods that we have implemented can be found in this file

https://github.com/AzBuilder/terrakube/blob/main/api/src/main/java/org/terrakube/api/plugin/state/RemoteTfeController.java

[jcanizalez]

I believe it would be beneficial to store the history state and not just the JSON output. If we decide to implement a feature to revert to a previous state in the future, relying solely on the JSON output would make this impossible. Therefore, we might consider using this issue as an opportunity to implement that feature. I don’t recall if the Terraform Cloud UI displays the JSON state or the Terraform show output. However, if we have both files, we could also implement the option to download the state in the UI.

[jderieg]

That's actually ultimately the thing that this user was trying to do: revert to a previous state. So that feature would be awesome!
Yes, app.terraform.io displays the actual json state in the UI.

[alfespa17]

This could be added but I will have to add support to our terraform/tofu library firts.

I created this issue

[jderieg]

Looking more at the 'terraform show -json' command, I'm not really sure what that's good for. They say the 'terraform show' command is to output in human-readable format, and the -json switch is to make it 'machine-readable', but if it's not in a format that terraform/tofu can use to manipulate, I'm not sure what good it is to even show in the UI. It seems like it would be better to only have the output generated by 'terraform pull' in the first place (the way Terraform Cloud shows it).

For reference: https://developer.hashicorp.com/terraform/cli/commands/show

[alfespa17]

A little update now our terraform/tofu library support to use "terraform/tofu state pull", now I can modify terrakube code 👍 to save the value and show it in the UI

[alfespa17]

Hello @jderieg

I got this working in my workspace, now you can see one additional tab in the workspace history where you can check the raw state file, this can be tested in 2.20.0-beta.9

Keep in mind that the raw state will be shown after a new apply/destroy, we don't have a way to generate the state file for previous workspace states.

By the way @jcanizalez it has some small bug you can switch from "diagram to code" or "diagram to raw" but if you switch from "code to raw" the editor wont update the information, I have no idea how to fix that maybe you can check when you have some free time.

[jcanizalez]

Sure, I will check it out later.

By the way, I'm not sure if it's better to rename the Code tab to JSON Output or something similar.