Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IMPORT PGDUMP cannot import from Azure Blob Storage in Government #47163

Closed
nlowe opened this issue Apr 8, 2020 · 0 comments · Fixed by #80511
Closed

IMPORT PGDUMP cannot import from Azure Blob Storage in Government #47163

nlowe opened this issue Apr 8, 2020 · 0 comments · Fixed by #80511
Labels
A-disaster-recovery C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-disaster-recovery

Comments

@nlowe
Copy link

nlowe commented Apr 8, 2020

Describe the problem

It would appear that the Account Base URL for Azure Storage is hard-coded to the Azure Public Cloud end-point:

u, err := url.Parse(fmt.Sprintf("https://%s.blob.core.windows.net", conf.AccountName))

(more recently:

u, err := url.Parse(fmt.Sprintf("https://%s.blob.core.windows.net", conf.AccountName))
)).

This prevents consumers with blob storage in Azure Government Cloud from restoring backups from their blob storage accounts.

To Reproduce

NOTE: All Azure Services in the following steps were created in the Azure US Government Cloud

  1. Setup a Secure CockroachDB Cluster in Azure AKS in following https://github.com/cockroachdb/cockroach/tree/master/cloud/kubernetes

  2. Create an Azure Storage Blob Account

  3. Create an Azure Storage Blob Container in the account created in step 2

  4. Upload backup.sql taken from cockroach dump to the storage container

  5. Create the database via cockroach sql:

     CREATE DATABASE test;
  6. Attempt to restore the database:

    IMPORT PGDUMP 'azure://mycontainer/test.sql?AZURE_ACCOUNT_NAME=myaccount&AZURE_ACCOUNT_KEY=***' WITH max_row_size='100MB';
    
    Error: pq: failed to create azure reader: -> github.com/cockroachdb/cockroach/vendor/github.com/Azure/azure-pipeline-go/pipeline.newDefaultHTTPClientFactory.func1.1, /go/src/github.com/cockroachdb/cockroach/vendor/github.com/Azure/azure-pipeline-go/pipeline/core.go:234
    HTTP request failed
    
    Get https://myaccount.blob.core.windows.net/mycontainer/test.sql?timeout=61: dial tcp: lookup myaccount.blob.core.windows.net on 192.168.0.10:53: no such host
    
    Failed running "sql"

Expected behavior

A user is either able to specify the full path to their storage account or force CockroachDB to look for storage accounts in different azure environments like US Government Cloud.

Additional data / screenshots

I also tried specifying the full account URL:

IMPORT PGDUMP 'azure://myaccount.blob.core.usgovcloudapi.net/mycontainer/test.sql?AZURE_ACCOUNT_NAME=myaccount&AZURE_ACCOUNT_KEY=***' WITH max_row_size='100MB';

but that clearly wasn't interpreted correctly:

Get https://myaccount.blob.core.windows.net/mycontainer.blob.core.usgovcloudapi.net/test.sql?timeout=61: dial tcp: lookup myaccount.blob.core.windows.net on 192.168.0.10:53: no such host

After finding this Stack Overflow Post about using the Azure GO SDKs with Azure Government, I even tried setting AZURE_ENVIRONMENT to AZUREUSGOVERNMENTCLOUD on all of my nodes including the client I was using to execute the IMPORT PGDUMP. This had no effect.

Environment:

  • CockroachDB version: v2.1.0
  • Server OS: Official Docker Containers in Kubernetes 1.16
  • Client app: cockroach sql

Additional context

Another method like HTTP or Node Local files will have to be used for backup/restore.

Jira issue: CRDB-5033

@mwang1026 mwang1026 added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label May 27, 2021
nlowe-sx added a commit to nlowe-sx/cockroach that referenced this issue Apr 25, 2022
The Azure Storage cloud provider learned a new parameter, AZURE_ENVIRONMENT,
which specifies which azure environment the storage account in question
belongs to. This allows cockroach to backup and restore data to Azure
Storage Accounts outside the main Azure Public Cloud. For backwards
compatibility, this defaults to "AzurePublicCloud" if AZURE_ENVIRONMENT
is not specified.

Fixes cockroachdb#47163
nlowe-sx added a commit to nlowe-sx/cockroach that referenced this issue Apr 26, 2022
The Azure Storage cloud provider learned a new parameter, AZURE_ENVIRONMENT,
which specifies which azure environment the storage account in question
belongs to. This allows cockroach to backup and restore data to Azure
Storage Accounts outside the main Azure Public Cloud. For backwards
compatibility, this defaults to "AzurePublicCloud" if AZURE_ENVIRONMENT
is not specified.

Fixes cockroachdb#47163

Release note (general): When using Azure Cloud Storage for data operations,
cockroach now calculates the Storage Account URL from the provided
AZURE_ENVIRONMENT query parameter. This defaults to AzurePublicCloud if not
specified to maintain backwards compatibility.
@craig craig bot closed this as completed in 0777ef7 Apr 29, 2022
blathers-crl bot pushed a commit that referenced this issue Apr 29, 2022
The Azure Storage cloud provider learned a new parameter, AZURE_ENVIRONMENT,
which specifies which azure environment the storage account in question
belongs to. This allows cockroach to backup and restore data to Azure
Storage Accounts outside the main Azure Public Cloud. For backwards
compatibility, this defaults to "AzurePublicCloud" if AZURE_ENVIRONMENT
is not specified.

Fixes #47163

Release note (general): When using Azure Cloud Storage for data operations,
cockroach now calculates the Storage Account URL from the provided
AZURE_ENVIRONMENT query parameter. This defaults to AzurePublicCloud if not
specified to maintain backwards compatibility.
blathers-crl bot pushed a commit that referenced this issue Apr 29, 2022
The Azure Storage cloud provider learned a new parameter, AZURE_ENVIRONMENT,
which specifies which azure environment the storage account in question
belongs to. This allows cockroach to backup and restore data to Azure
Storage Accounts outside the main Azure Public Cloud. For backwards
compatibility, this defaults to "AzurePublicCloud" if AZURE_ENVIRONMENT
is not specified.

Fixes #47163

Release note (general): When using Azure Cloud Storage for data operations,
cockroach now calculates the Storage Account URL from the provided
AZURE_ENVIRONMENT query parameter. This defaults to AzurePublicCloud if not
specified to maintain backwards compatibility.
adityamaru pushed a commit that referenced this issue May 2, 2022
The Azure Storage cloud provider learned a new parameter, AZURE_ENVIRONMENT,
which specifies which azure environment the storage account in question
belongs to. This allows cockroach to backup and restore data to Azure
Storage Accounts outside the main Azure Public Cloud. For backwards
compatibility, this defaults to "AzurePublicCloud" if AZURE_ENVIRONMENT
is not specified.

Fixes #47163

Release note (sql change): When using Azure Cloud Storage for data operations,
cockroach now calculates the Storage Account URL from the provided
AZURE_ENVIRONMENT query parameter. This defaults to AzurePublicCloud if not
specified to maintain backwards compatibility.
This parameter should not be used when the cluster is in a mixed version
or upgrading state, as nodes that have not been upgraded will continue to
send requests to the AzurePublicCloud even in the presence of this parameter.
adityamaru pushed a commit that referenced this issue May 4, 2022
The Azure Storage cloud provider learned a new parameter, AZURE_ENVIRONMENT,
which specifies which azure environment the storage account in question
belongs to. This allows cockroach to backup and restore data to Azure
Storage Accounts outside the main Azure Public Cloud. For backwards
compatibility, this defaults to "AzurePublicCloud" if AZURE_ENVIRONMENT
is not specified.

Fixes #47163

Release note (sql change): When using Azure Cloud Storage for data operations,
cockroach now calculates the Storage Account URL from the provided
AZURE_ENVIRONMENT query parameter. This defaults to AzurePublicCloud if not
specified to maintain backwards compatibility.
This parameter should not be used when the cluster is in a mixed version
or upgrading state, as nodes that have not been upgraded will continue to
send requests to the AzurePublicCloud even in the presence of this parameter.
adityamaru pushed a commit that referenced this issue May 4, 2022
The Azure Storage cloud provider learned a new parameter, AZURE_ENVIRONMENT,
which specifies which azure environment the storage account in question
belongs to. This allows cockroach to backup and restore data to Azure
Storage Accounts outside the main Azure Public Cloud. For backwards
compatibility, this defaults to "AzurePublicCloud" if AZURE_ENVIRONMENT
is not specified.

Fixes #47163

Release note (general): When using Azure Cloud Storage for data operations,
cockroach now calculates the Storage Account URL from the provided
AZURE_ENVIRONMENT query parameter. This defaults to AzurePublicCloud if not
specified to maintain backwards compatibility.
This parameter should not be used when the cluster is in a mixed version
or upgrading state, as nodes that have not been upgraded will continue
to send requests to the AzurePublicCloud even in the presence of this
parameter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-disaster-recovery C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-disaster-recovery
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants