FINAL: Git storage option feature #1279
Conversation
Co-authored-by: Arin Ghazarian <aringhazarian@github.com>
Co-authored-by: Arin Ghazarian <aringhazarian@github.com>
PART1: Add `use-github-storage` option
PART 2:`use-github-storage` option validation
begonaguereca
changed the title
Unit Test Results825 tests 825 ✅ 22s ⏱️ Results for commit 1478850. ♻️ This comment has been updated with latest results. |
Co-authored-by: Arin Ghazarian <aringhazarian@github.com>
Co-authored-by: Arin Ghazarian <aringhazarian@github.com>
Co-authored-by: Arin Ghazarian <aringhazarian@github.com>
Co-authored-by: Arin Ghazarian <aringhazarian@github.com>
…ests Github Owned Storage integration tests
Co-authored-by: Arin Ghazarian <aringhazarian@github.com>
Co-authored-by: Arin Ghazarian <aringhazarian@github.com>
Multipart uploads updated logic
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| catch (Exception ex) | ||
| { | ||
| throw new OctoshiftCliException("Failed during multipart upload.", ex); | ||
| } |
Check notice
Code scanning / CodeQL
Generic catch clause Note
|
|
||
| try | ||
| { | ||
| var (responseContent, headers) = await _client.PostWithFullResponseAsync(uploadUrl, body); |
Check warning
Code scanning / CodeQL
Useless assignment to local variable Warning
| catch (Exception ex) | ||
| { | ||
| throw new OctoshiftCliException("Failed to start upload.", ex); | ||
| } |
Check notice
Code scanning / CodeQL
Generic catch clause Note
| try | ||
| { | ||
| // Make the PATCH request and retrieve headers | ||
| var (responseContent, headers) = await _client.PatchWithFullResponseAsync(nextUrl, content); |
Check warning
Code scanning / CodeQL
Useless assignment to local variable Warning
| catch (Exception ex) | ||
| { | ||
| throw new OctoshiftCliException("Failed to upload part.", ex); | ||
| } |
Check notice
Code scanning / CodeQL
Generic catch clause Note
| catch (Exception ex) | ||
| { | ||
| throw new OctoshiftCliException("Failed to complete upload.", ex); | ||
| } |
Check notice
Code scanning / CodeQL
Generic catch clause Note
| // Using a MemoryStream as a valid stream implementation | ||
| using var archiveContent = new MemoryStream(new byte[] { 1, 2, 3 }); | ||
| var expectedUri = "gei://archive/123456"; | ||
| var jsonResponse = $"{{ \"uri\": \"{expectedUri}\" }}"; |
Check warning
Code scanning / CodeQL
Useless assignment to local variable Warning test
|
There was a problem hiding this comment.
I just looked at the integration tests and the reason they are failing is because the generate-script commands for gei and bbs2gh don't support the --use-github-storage option yet! What I recommend is to revert the INT test changes for now and add them back (in a separate PR) once we add the option to the generate-script commands.
| @@ -117,7 +120,7 @@ await retryPolicy.Retry(async () => | |||
| _tokens.Add("AWS_ACCESS_KEY_ID", Environment.GetEnvironmentVariable("AWS_ACCESS_KEY_ID")); | |||
| _tokens.Add("AWS_SECRET_ACCESS_KEY", Environment.GetEnvironmentVariable("AWS_SECRET_ACCESS_KEY")); | |||
| var awsBucketName = Environment.GetEnvironmentVariable("AWS_BUCKET_NAME"); | |||
| archiveUploadOptions = $" --aws-bucket-name {awsBucketName} --aws-region {AWS_REGION}"; | |||
| archiveUploadOptions = $" --aws-bucket-name {awsBucketName} --aws-region {AWS_REGION} --use-github-storage {useGithubStorage}"; | |||
There was a problem hiding this comment.
There are a couple of things here:
--use-github-storageis not yet a supported option for thegenerate-scriptcommands so this ultimately won't work until we add the option.- Even if the GitHub storage option was supported by
generate-scriptthis still wouldn't work because both AWS and GitHub storage options are provided at the same time which should be a validation error.
| [InlineData("http://e2e-bbs-7-21-9-win-2019.eastus.cloudapp.azure.com:7990", false, true, false)] | ||
| [InlineData("http://e2e-bbs-8-5-0-linux-2204.eastus.cloudapp.azure.com:7990", true, false, false)] | ||
| [InlineData("http://e2e-bbs-8-5-0-linux-2204.eastus.cloudapp.azure.com:7990", false, false, true)] | ||
| public async Task Basic(string bbsServer, bool useSshForArchiveDownload, bool useAzureForArchiveUpload, bool useGithubStorage) |
There was a problem hiding this comment.
At this point I recommend adding an enum for the upload options as a class member (nested enum) right after all private readonly member declarations:
public enum ArchiveUploadOption { AzureStorage, AwsS3, GithubStorage };Then change the method signature and the inline data like the following:
[InlineData("http://e2e-bbs-8-5-0-linux-2204.eastus.cloudapp.azure.com:7990", true, ArchiveUploadOption.AzureStorage)]
[InlineData("http://e2e-bbs-7-21-9-win-2019.eastus.cloudapp.azure.com:7990", false, ArchiveUploadOption.AzureStorage)]
[InlineData("http://e2e-bbs-8-5-0-linux-2204.eastus.cloudapp.azure.com:7990", true, ArchiveUploadOption.AwsS3)]
[InlineData("http://e2e-bbs-8-5-0-linux-2204.eastus.cloudapp.azure.com:7990", true, ArchiveUploadOption.GithubStorage)]
public async Task Basic(string bbsServer, bool useSshForArchiveDownload, bool useAzureForArchiveUpload, bool useGithubStorage)| @@ -83,7 +87,7 @@ await retryPolicy.Retry(async () => | |||
| }); | |||
|
|
|||
| await _targetHelper.RunGeiCliMigration( | |||
| $"generate-script --github-source-org {githubSourceOrg} --github-target-org {githubTargetOrg} --ghes-api-url {GHES_API_URL} --download-migration-logs", _tokens); | |||
| $"generate-script --github-source-org {githubSourceOrg} --github-target-org {githubTargetOrg} --ghes-api-url {GHES_API_URL} --use-github-storage {useGithubStorage} --download-migration-logs", _tokens); | |||
There was a problem hiding this comment.
In case of false for useGithubStorge it shouldn't add the option at all.
Summary:
This PR introduces the ability for the GEI CLI to upload archives to GitHub-owned storage using multipart uploads. To enable this, a new CLI option, --use-github-storage, has been added. This option allows users to explicitly specify that their archives should be uploaded to GitHub's managed storage instead of their own.
Key Features:
Multipart Upload Support: Leveraging multipart upload for reliable and efficient large file uploads. This ensures the robustness of the uploads, minimizing the chances of failure during transmission. We're investigating libraries to handle multipart uploads, but for now, this feature is not rolled by hand unless necessary.
New Option: --use-github-storage:
This option is required to upload archives to GitHub-owned storage and will be hidden until GitHub-owned storage reaches general availability (GA).
By default, archives are still uploaded to user-specified storage locations unless the --use-github-storage flag is explicitly set.
Motivation:
As discussed during our EDR, we want to allow users to utilize GitHub-owned storage as an option without making it the default. This approach lets us roll out GitHub-managed storage in stages and gather feedback while providing an explicit way to opt-in to this feature.
Check in BBS MigrateRepoCommandArgs removed, relating to changes in this PR: #1057
Take it out for a 🚗 :
*** Remember to export env variables***
Migrating from GitHub Enterprise Server:
Successful migration_id:
RM_kgHaACRlZjNiMmExMy02ZTRkLTQ0NWYtOTc4Yy05YTk1MTQ5NzE2OTkMigrating from GitHub Enterprise Server- Multi part upload:
Successful migration_id: RM_kgDaACQ0MzZmN2RjMS0wNjk4LTRlYjctYmMzNy0wMTAxNGRhOTQ3YTU
Migrating from BBS:
Source repo: https://test-bbs-o.githubapp.com/projects/IM/repos/codeowners-test/browse
You'll need to add yourself to the pinhole firewall at https://portal.azure.com/?wa=wsignin1.0#@githubazure.onmicrosoft.com/resource/subscriptions/7c45ec63-636b-445b-8185-54accbc0190d/resourceGroups/octoshift-test-bbs/providers/Microsoft.Compute/virtualMachines/bbs-test/networkSettings
Here's the SSH key (and other creds) for our bbs-test-o instance: https://start.1password.com/open/i?a=LKXPAKKGYNBF7IOHIU6VPDFEU4&v=akrfryl3erhcqdcvgiqqtvh254&i=23dceluq4vdvpptr7qen3wcbiu&h=github.1password.com
Successful migration_id: RM_kgHaACRhODI4NWRmZS0yNWM3LTQ1YjEtYTJhOC1jYmZjNTkxZDE0Mjk
Closes: https://github.ghe.com/github/octoshift/issues/8969
ThirdPartyNotices.txt(if applicable)