Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

support multiple corpus accounts #334

Merged
28 commits merged into from
Jan 6, 2021
Merged

support multiple corpus accounts #334

28 commits merged into from
Jan 6, 2021

Conversation

bmc-msft
Copy link
Contributor

@bmc-msft bmc-msft commented Nov 20, 2020 •
edited
Loading

Add support for sharding across multiple storage accounts for blob containers used for corpus management.

Things to note:

  1. Additional storage accounts must be in the same resource group, support the "blob" endpoint, and have the tag storage_type with the value corpus. A utility is provided (src/utils/add-corpus-storage-accounts), which adds storage accounts.
  2. If any secondary storage accounts exist, they are used by default for containers.
  3. Storage account names are cached in memory the Azure Function instance forever. Upon adding new storage accounts, the app needs to be restarted to pick up the new accounts.

@bmc-msft bmc-msft marked this pull request as draft November 20, 2020 18:34
demoray and others added 18 commits November 20, 2020 13:46
@demoray
cache storage account lookups until restart
1386cb2
@demoray
ignore type here too
3b9da40
@bmc-msft
Merge branch 'main' into multiple-storage-accounts
ef605ec
@bmc-msft
Merge branch 'main' into multiple-storage-accounts
05dcae7
@demoray
Merge remote-tracking branch 'upstream/main' into multiple-storage-ac…
01417c7 
…counts
@demoray
use the Container primitive type, rather than str
a841327
@demoray
Merge branch 'multiple-storage-accounts' of github.com:bmc-msft/onefu…
0ca7305 
…zz into multiple-storage-accounts
@demoray
don't cache get_account_by_container
c2ff77d
@demoray
Add tool to create & register additional storage accounts
47bf3ac
@demoray
iterate over storage accounts directly
d0ee929
@demoray
if secondary accounts exist, always use those
8ec9bdb
@demoray
search containers in reverse
9cd3510
@bmc-msft
Merge branch 'main' into multiple-storage-accounts
5d1f284
@bmc-msft
Merge branch 'main' into multiple-storage-accounts
46ca4a3
@demoray
add vm-scripts during deployment
518f953
@demoray
fix printed exception
499cb31
@demoray
Merge branch 'multiple-storage-accounts' of github.com:bmc-msft/onefu…
1d24c7b 
…zz into multiple-storage-accounts
@demoray
handle azure functions making /home/.azure read-only
f544d5b
@bmc-msft
Copy link
Contributor Author

bmc-msft commented Dec 9, 2020

This PR needs to add proxy-configs to the base setup.

@bmc-msft bmc-msft linked an issue Dec 16, 2020 that may be closed by this pull request
Support scaling storage by using multiple storage accounts #310
Closed
@bmc-msft bmc-msft marked this pull request as ready for review January 6, 2021 17:34
@ghost
Copy link

ghost commented Jan 6, 2021

Hello @bmc-msft!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

@ghost ghost merged commit 3b26ffe into microsoft:main Jan 6, 2021
@bmc-msft bmc-msft deleted the multiple-storage-accounts branch January 6, 2021 23:15
@ghost ghost locked as resolved and limited conversation to collaborators Apr 17, 2021
This pull request was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@ranweiler ranweiler ranweiler approved these changes

@chkeita chkeita chkeita approved these changes

Assignees
No one assigned
Labels
auto-merge passed-integration-tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support scaling storage by using multiple storage accounts
4 participants
@bmc-msft @chkeita @ranweiler @demoray