Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AzDatalake] Cleanup + Improvements #21222

Merged
merged 63 commits into from
Jul 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
e8167a2
Enable gocritic during linting (#20715)
jhendrixMSFT Apr 28, 2023
86627ae
Cosmos DB: Enable merge support (#20716)
ealsur Apr 28, 2023
8ac8c6d
[azservicebus, azeventhubs] Stress test and logging improvement (#20710)
richardpark-msft May 1, 2023
9111616
update proxy version (#20712)
azure-sdk May 1, 2023
d6bf190
Return an error when you try to send a message that's too large. (#20…
richardpark-msft May 1, 2023
e2693bd
Changes in test that is failing in pipeline (#20693)
siminsavani-msft May 2, 2023
03f0ac3
[azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20…
richardpark-msft May 2, 2023
838842d
[azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)
richardpark-msft May 3, 2023
20b4dd8
Update changelog with latest features (#20730)
jhendrixMSFT May 3, 2023
745d967
pass along the artifact name so we can override it later (#20732)
azure-sdk May 3, 2023
6dfd0cb
[azeventhubs] Fixing checkpoint store race condition (#20727)
richardpark-msft May 3, 2023
ed7f3c7
Fix azidentity troubleshooting guide link (#20736)
chlowell May 3, 2023
b2cddab
[Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (…
Alancere May 4, 2023
2a8d96d
add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)
Alancere May 4, 2023
0d22aed
add sdk/resourcemanager/eventhub/armeventhub live test (#20686)
Alancere May 4, 2023
5fa7df4
add sdk/resourcemanager/compute/armcompute live test (#20048)
Alancere May 4, 2023
c005ed6
sdk/resourcemanager/network/armnetwork live test (#20331)
Alancere May 4, 2023
36f766d
add sdk/resourcemanager/cosmos/armcosmos live test (#20705)
Alancere May 4, 2023
9c9d62a
Increment package version after release of azcore (#20740)
azure-sdk May 4, 2023
8bc3450
[azeventhubs] Improperly resetting etag in the checkpoint store (#20737)
richardpark-msft May 4, 2023
e1a6152
Eng workflows sync and branch cleanup additions (#20743)
azure-sdk May 4, 2023
04b463d
[azeventhubs] Latest start position can also be inclusive (ie, get th…
richardpark-msft May 4, 2023
8849196
Update GitHubEventProcessor version and remove pull_request_review pr…
azure-sdk May 5, 2023
27f5ee0
Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)
chlowell May 5, 2023
2eec707
fix (#20707)
Alancere May 6, 2023
22db2d4
AzFile (#20739)
souravgupta-msft May 8, 2023
0cbfd88
azfile: Fixing connection string parsing logic (#20798)
souravgupta-msft May 8, 2023
d54fb08
[azadmin] fix flaky test (#20758)
gracewilcox May 8, 2023
ad8ebd9
Prepare azidentity v1.3.0 for release (#20756)
chlowell May 8, 2023
e2a6f70
Fix broken podman link (#20801)
azure-sdk May 8, 2023
a59d912
[azquery] update doc comments (#20755)
gracewilcox May 8, 2023
bd3b467
Fixed contribution section (#20752)
bobtabor-msft May 8, 2023
132a01a
[azeventhubs,azservicebus] Some API cleanup, renames (#20754)
richardpark-msft May 8, 2023
8db51ca
Add supporting features to enable distributed tracing (#20301) (#20708)
jhendrixMSFT May 9, 2023
4a66b4f
Restore ARM CAE support for azcore beta (#20657)
chlowell May 9, 2023
7d4a3cb
Upgrade to stable azcore (#20808)
chlowell May 9, 2023
068c3be
Increment package version after release of data/azcosmos (#20807)
azure-sdk May 9, 2023
8e0f66e
Updating changelog (#20810)
souravgupta-msft May 9, 2023
ce926c4
Add fake package to azcore (#20711)
jhendrixMSFT May 9, 2023
1a145c5
Updating CHANGELOG.md (#20809)
siminsavani-msft May 9, 2023
90dfc5c
changelog (#20811)
tasherif-msft May 9, 2023
c7eda59
Increment package version after release of storage/azfile (#20813)
azure-sdk May 9, 2023
7fac0b5
Update changelog (azblob) (#20815)
siminsavani-msft May 9, 2023
498a2ef
[azquery] migration guide (#20742)
gracewilcox May 9, 2023
ccb967e
Increment package version after release of monitor/azquery (#20820)
azure-sdk May 9, 2023
f4e6a22
[keyvault] prep for release (#20819)
gracewilcox May 10, 2023
8fd8eda
Merge branch 'main' into feature/azdatalake
tasherif-msft May 11, 2023
c94fa00
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft May 11, 2023
fc0b2b5
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jun 12, 2023
6fb1694
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jun 19, 2023
4f7fe43
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jun 26, 2023
3dac9d0
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 4, 2023
a0a861b
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 7, 2023
124e27e
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 19, 2023
133e6dd
fixed datalake errors + moved to internal path
tasherif-msft Jul 19, 2023
3052a35
delegation key + constants
tasherif-msft Jul 20, 2023
a3fad77
removed test
tasherif-msft Jul 20, 2023
4048359
further cleanup
tasherif-msft Jul 20, 2023
b3f98b1
renamed support and client fixes
tasherif-msft Jul 20, 2023
42e9c62
added tests
tasherif-msft Jul 20, 2023
5a1a8ea
handle error for rename
tasherif-msft Jul 21, 2023
30bb9ba
fixed response formatting
tasherif-msft Jul 22, 2023
fc268ea
cleanup
tasherif-msft Jul 24, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sdk/storage/azdatalake/assets.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "go",
"TagPrefix": "go/storage/azdatalake",
"Tag": "go/storage/azdatalake_820b86faa9"
"Tag": "go/storage/azdatalake_db1de4a48b"
}
45 changes: 45 additions & 0 deletions sdk/storage/azdatalake/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
package azdatalake

import (
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/lease"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/exported"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/sas"
)
Expand Down Expand Up @@ -34,3 +35,47 @@ func ParseURL(u string) (URLParts, error) {
// ending at offset+count. A zero-value HTTPRange indicates the entire resource. An HTTPRange
// which has an offset but no zero value count indicates from the offset to the resource's end.
type HTTPRange = exported.HTTPRange

// ===================================== LEASE CONSTANTS ============================================================

// StatusType defines values for StatusType
type StatusType = lease.StatusType
tasherif-msft marked this conversation as resolved.
Show resolved Hide resolved

const (
StatusTypeLocked StatusType = lease.StatusTypeLocked
StatusTypeUnlocked StatusType = lease.StatusTypeUnlocked
)

// PossibleStatusTypeValues returns the possible values for the StatusType const type.
func PossibleStatusTypeValues() []StatusType {
return lease.PossibleStatusTypeValues()
}

// DurationType defines values for DurationType
type DurationType = lease.DurationType

const (
DurationTypeInfinite DurationType = lease.DurationTypeInfinite
DurationTypeFixed DurationType = lease.DurationTypeFixed
)

// PossibleDurationTypeValues returns the possible values for the DurationType const type.
func PossibleDurationTypeValues() []DurationType {
return lease.PossibleDurationTypeValues()
}

// StateType defines values for StateType
type StateType = lease.StateType

const (
StateTypeAvailable StateType = lease.StateTypeAvailable
StateTypeLeased StateType = lease.StateTypeLeased
StateTypeExpired StateType = lease.StateTypeExpired
StateTypeBreaking StateType = lease.StateTypeBreaking
StateTypeBroken StateType = lease.StateTypeBroken
)

// PossibleStateTypeValues returns the possible values for the StateType const type.
func PossibleStateTypeValues() []StateType {
return lease.PossibleStateTypeValues()
}
240 changes: 117 additions & 123 deletions sdk/storage/azdatalake/datalakeerror/error_codes.go

Large diffs are not rendered by default.

17 changes: 9 additions & 8 deletions sdk/storage/azdatalake/directory/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ func NewClient(directoryURL string, cred azcore.TokenCredential, options *Client
ClientOptions: options.ClientOptions,
}
blobClient, _ := blockblob.NewClient(blobURL, cred, &blobClientOpts)
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, nil, (*base.ClientOptions)(conOptions))
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, nil, &cred, (*base.ClientOptions)(conOptions))

return (*Client)(dirClient), nil
}
Expand Down Expand Up @@ -78,7 +78,7 @@ func NewClientWithNoCredential(directoryURL string, options *ClientOptions) (*Cl
ClientOptions: options.ClientOptions,
}
blobClient, _ := blockblob.NewClientWithNoCredential(blobURL, &blobClientOpts)
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, nil, (*base.ClientOptions)(conOptions))
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, nil, nil, (*base.ClientOptions)(conOptions))

return (*Client)(dirClient), nil
}
Expand Down Expand Up @@ -113,7 +113,7 @@ func NewClientWithSharedKeyCredential(directoryURL string, cred *SharedKeyCreden
return nil, err
}
blobClient, _ := blockblob.NewClientWithSharedKeyCredential(blobURL, blobSharedKey, &blobClientOpts)
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, cred, (*base.ClientOptions)(conOptions))
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, cred, nil, (*base.ClientOptions)(conOptions))

return (*Client)(dirClient), nil
}
Expand Down Expand Up @@ -158,6 +158,10 @@ func (d *Client) sharedKey() *exported.SharedKeyCredential {
return base.SharedKeyComposite((*base.CompositeClient[generated.PathClient, generated.PathClient, blockblob.Client])(d))
}

func (d *Client) identityCredential() *azcore.TokenCredential {
return base.IdentityCredentialComposite((*base.CompositeClient[generated.PathClient, generated.PathClient, blockblob.Client])(d))
}

// DFSURL returns the URL endpoint used by the Client object.
func (d *Client) DFSURL() string {
return d.generatedDirClientWithDFS().Endpoint()
Expand All @@ -168,6 +172,8 @@ func (d *Client) BlobURL() string {
return d.generatedDirClientWithBlob().Endpoint()
}

//TODO: create method to get file client - this will require block blob to have a method to get another block blob

// Create creates a new directory (dfs1).
func (d *Client) Create(ctx context.Context, options *CreateOptions) (CreateResponse, error) {
return CreateResponse{}, nil
Expand Down Expand Up @@ -230,8 +236,3 @@ func (d *Client) SetHTTPHeaders(ctx context.Context, httpHeaders HTTPHeaders, op
// TODO: call into blob
return SetHTTPHeadersResponse{}, nil
}

// UndeletePath restores the specified path that was previously deleted. (dfs op/blob2).
func (d *Client) UndeletePath(ctx context.Context, path string, options *UndeletePathOptions) (UndeletePathResponse, error) {
return UndeletePathResponse{}, nil
}
33 changes: 4 additions & 29 deletions sdk/storage/azdatalake/directory/constants.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,37 +7,12 @@
package directory

import (
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/generated"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/path"
)

type ResourceType = generated.PathResourceType
type EncryptionAlgorithmType = path.EncryptionAlgorithmType

// TODO: consider the possibility of not exposing this and just pass it under the hood
const (
ResourceTypeFile ResourceType = generated.PathResourceTypeFile
ResourceTypeDirectory ResourceType = generated.PathResourceTypeDirectory
)

type RenameMode = generated.PathRenameMode

// TODO: consider the possibility of not exposing this and just pass it under the hood
const (
RenameModeLegacy RenameMode = generated.PathRenameModeLegacy
RenameModePosix RenameMode = generated.PathRenameModePosix
)

type SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveMode

const (
SetAccessControlRecursiveModeSet SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveModeSet
SetAccessControlRecursiveModeModify SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveModeModify
SetAccessControlRecursiveModeRemove SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveModeRemove
)

type EncryptionAlgorithmType = blob.EncryptionAlgorithmType

const (
EncryptionAlgorithmTypeNone EncryptionAlgorithmType = blob.EncryptionAlgorithmTypeNone
EncryptionAlgorithmTypeAES256 EncryptionAlgorithmType = blob.EncryptionAlgorithmTypeAES256
EncryptionAlgorithmTypeNone EncryptionAlgorithmType = path.EncryptionAlgorithmTypeNone
EncryptionAlgorithmTypeAES256 EncryptionAlgorithmType = path.EncryptionAlgorithmTypeAES256
)
Loading