CHANGELOG-old.md

Historical Changelog

object_store_0.11.1 (2024-10-15)

Full Changelog

Implemented enhancements:

• There is no way to pass object store client options as environment variables #6333 [object-store]
• Better Document Backoff Algorithm #6324 [object-store]
• Add direction to list_with_offset #6274 [object-store]
• Support server-side encryption with customer-provided keys (SSE-C) #6229 [object-store]

Fixed bugs:

• [object-store] Requested tokio version is too old - does not compile #6458 [object-store]
• Azure SAS tokens are visible when retry errors are logged via object_store #6322 [object-store]

Merged pull requests:

• object_store: fix typo in with_connect_timeout_disabled that actually disabled non-connect timeouts #6563 [object-store] (adriangb)
• object_store: Clarify what is a prefix in list() documentation #6520 [object-store] (progval)
• object_store: enable lint unreachable_pub #6512 [object-store] (ByteBaker)
• [object_store] Retry S3 requests with 200 response with "Error" in body #6508 [object-store] (PeterKeDer)
• [object-store] Require tokio 1.29.0. #6459 [object-store] (ashtuchkin)
• feat: expose HTTP/2 max frame size in object_store #6442 [object-store] (crepererum)
• Derive Clone for object_store::aws::AmazonS3 #6414 [object-store] (ethe)
• object_score: Support Azure Fabric OAuth Provider #6382 [object-store] (RobinLin666)
• object_store::GetOptions derive Clone #6361 [object-store] (samuelcolvin)
• [object_store] Propagate env vars as object store client options #6334 [object-store] (ccciudatu)
• docs[object_store]: clarify the backoff strategy that is actually implemented #6325 [object-store] (westonpace)
• fix: azure sas token visible in logs #6323 [object-store] (alexwilcoxson-rel)
• object_store/delimited: Fix TrailingEscape condition #6265 [object-store] (Turbo87)
• fix(object_store): only add encryption headers for SSE-C in get request #6260 [object-store] (jiachengdb)
• docs: Add parquet_opendal in related projects #6236 (Xuanwo)
• feat(object_store): add support for server-side encryption with customer-provided keys (SSE-C) #6230 [object-store] (jiachengdb)
• feat: further TLS options on ClientOptions: #5034 #6148 [object-store] (ByteBaker)

object_store_0.11.0 (2024-08-12)

Full Changelog

Breaking changes:

• Make object_store errors non-exhaustive #6165 [object-store] (tustvold)
• Update snafu to 0.8.0 in object_store (#5930) #6070 [object-store] (alamb)

Merged pull requests:

• Add LICENSE and NOTICE files to object_store #6234 [object-store] (alamb)
• feat(object_store): add PermissionDenied variant to top-level error #6194 [object-store] (kyle-mccarthy)
• Update object store MSRV to 1.64 #6123 [object-store] (alamb)
• Fix clippy in object_store crate #6120 [object-store] (alamb)

object_store_0.10.2 (2024-07-17)

Full Changelog

Implemented enhancements:

• Relax WriteMultipart API to support aborting after completion #5977 [object-store]
• Make ObjectStoreScheme in the object_store crate public #5911 [object-store]
• Add BufUploader to implement same feature upon WriteMultipart like BufWriter #5834 [object-store]

Fixed bugs:

• Investigate why InstanceCredentialProvider::cache is flagged as dead code #5884 [object-store]
• [object_store] Potential race condition in list_with_delimiter on Local #5800 [object-store]

Documentation updates:

• Correct timeout in comment from 5s to 30s #6073 [object-store] (trungda)
• docs: Fix broken links of object_store_opendal README #5929 [object-store] (Xuanwo)
• docs: Add object_store_opendal as related projects #5926 [object-store] (Xuanwo)
• chore: update docs to delineate which ObjectStore lists are recursive #5794 [object-store] (wiedld)
• Document object store release cadence #5750 [object-store] (alamb)

Merged pull requests:

• Sanitize error message for sensitive requests #6074 [object-store] (tustvold)
• Update quick-xml requirement from 0.35.0 to 0.36.0 in /object_store #6032 [object-store] (dependabot[bot])
• use GCE metadata server env var overrides #6015 [object-store] (barronw)
• Update quick-xml requirement from 0.34.0 to 0.35.0 in /object_store #5983 [object-store] (dependabot[bot])
• Automatically cleanup empty dirs in LocalFileSystem #5978 [object-store] (fsdvh)
• WriteMultipart Abort on MultipartUpload::complete Error #5974 [object-store] (fsdvh)
• Update quick-xml requirement from 0.33.0 to 0.34.0 in /object_store #5954 [object-store] (dependabot[bot])
• Update quick-xml requirement from 0.32.0 to 0.33.0 in /object_store #5946 [object-store] (dependabot[bot])
• Add MultipartUpload blanket implementation for Box<W> #5919 [object-store] (fsdvh)
• Add user defined metadata #5915 [object-store] (criccomini)
• Make ObjectStoreScheme public #5912 [object-store] (orf)
• chore: Remove not used cache in InstanceCredentialProvider #5888 [object-store] (Xuanwo)
• Fix clippy for object_store #5883 [object-store] (alamb)
• Update quick-xml requirement from 0.31.0 to 0.32.0 in /object_store #5870 [object-store] (dependabot[bot])
• feat(object_store): Add put API for buffered::BufWriter #5835 [object-store] (Xuanwo)
• Fix 5592: Colon (:) in in object_store::path::{Path} is not handled on Windows #5830 [object-store] (hesampakdaman)
• Fix issue #5800: Handle missing files in list_with_delimiter #5803 [object-store] (hesampakdaman)
• Update nix requirement from 0.28.0 to 0.29.0 in /object_store #5799 [object-store] (dependabot[bot])
• Update itertools requirement from 0.12.0 to 0.13.0 in /object_store #5780 [object-store] (dependabot[bot])
• Add additional WriteMultipart tests (#5743) #5746 [object-store] (tustvold)

* This Changelog was automatically generated by github_changelog_generator

object_store_0.10.1 (2024-05-10)

Full Changelog

Implemented enhancements:

• Allow specifying PUT options when using BufWriter #5692 [object-store]
• Add more attributes to object_store::Attribute #5689 [object-store]
• feat object_store: moving tests from src/ to a tests/ folder and enabling access to test functions for enabling a shared integration test suite #5685 [object-store]
• Release Object Store 0.10.0 #5647 [object-store]

Fixed bugs:

• Using WriteMultipart::put results in 0 bytes being written #5743 [object-store]

Merged pull requests:

• Fix PutPayloadMut::push not updating content_length (#5743) #5744 [object-store] (tustvold)
• Export object_store integration tests #5709 [object-store] (tustvold)
• Add BufWriter::with_attributes and ::with_tags in object_store #5693 [object-store] (netthier)
• Add more attributes to object_store::Attribute #5690 [object-store] (netthier)

object_store_0.10.0 (2024-04-17)

Full Changelog

Breaking changes:

• Add put_multipart_opts (#5435) #5652 [object-store] (tustvold)
• Add Attributes API (#5329) #5650 [object-store] (tustvold)
• Support non-contiguous put payloads / vectored writes (#5514) #5538 [object-store] (tustvold)
• Replace AsyncWrite with Upload trait and rename MultiPartStore to MultipartStore (#5458) #5500 [object-store] (tustvold)

Implemented enhancements:

• Improve Retry Coverage #5608 [object-store]
• Zero Copy Support #5593
• ObjectStore bulk delete #5591
• Retry on Broken Connection #5589
• Inconsistent Multipart Nomenclature #5526 [object-store]
• [ObjectStore] Non-Contiguous Write Payloads #5514 [object-store]
• In Object Store, return version & etag on multipart put. #5443 [object-store]
• Release Object Store 0.9.1 #5436 [object-store]
• object_store: allow setting content-type per request #5329 [object-store]
• GCS Signed URL Support #5233

Fixed bugs:

• [object_store] minor bug: typos present in local variable #5628 [object-store]
• [arrow-csv] Schema inference requires csv on disk #5551
• Local object store copy/rename with nonexistent from file loops forever instead of erroring #5503 [object-store]
• object store ApplicationDefaultCredentials auth is not working on windows #5466 [object-store]
• MicrosoftAzure store list result omits empty objects #5451 [object-store]

Documentation updates:

• Minor: add additional documentation about BufWriter #5519 [object-store] (alamb)

Merged pull requests:

• minor-fix: removed typos in object_store sub crate #5629 [object-store] (Silemo)
• Retry on More Error Classes #5609 [object-store] (andrebsguedes)
• Fix handling of empty multipart uploads for GCS #5590 [object-store] (tustvold)
• Upgrade object_store dependency to use chrono 0.4.34 #5578 [object-store] (l1nxy)
• Fix Latest Clippy Lints for object_store #5546 [object-store] (tustvold)
• Update reqwest 0.12 and http 1.0 #5536 [object-store] (tustvold)
• Implement MultipartStore for ThrottledStore #5533 [object-store] (tustvold)
• fix: copy/rename return error if source is nonexistent #5528 [object-store] (dimbtp)
• Prepare arrow 51.0.0 #5516 (tustvold)
• Implement MultiPartStore for InMemory #5495 [object-store] (tustvold)
• Add more comprehensive documentation on testing and benchmarking to CONTRIBUTING.md #5478 (monkwire)
• add support for gcp application default auth on windows in object store #5473 [object-store] (Itayazolay)
• Update base64 requirement from 0.21 to 0.22 in /object_store #5465 [object-store] (dependabot[bot])
• Uses ResourceType for filtering list directories instead of workaround #5452 [object-store] (andrebsguedes)
• Add GCS signed URL support #5300 [object-store] (l1nxy)

object_store_0.9.1 (2024-03-01)

Full Changelog

Implemented enhancements:

• [object_store] Enable anonymous read access for Azure #5424 [object-store]
• Support for additional URL formats in object_store for Azure blob #5370 [object-store]
• Mention "Http" support in README #5320 [object-store]
• Pass Options to HttpBuilder in parse_url_opts #5310 [object-store]
• Remove Localstack DynamoDB Workaround Once Fixed Upstream #5267 [object-store]
• Can I use S3 server side encryption #5087 [object-store]

Fixed bugs:

• delete_stream fails in MinIO #5414 [object-store]
• [object_store] Completing an empty Multipart Upload fails for AWS S3 #5404 [object-store]
• Multipart upload can leave futures unpolled, leading to timeout #5366 [object-store]
• Broken Link in README (Rust Object Store) Content #5309 [object-store]

Merged pull requests:

• Expose path_to_filesystem public #5441 [object-store] (metesynnada)
• Update nix requirement from 0.27.1 to 0.28.0 in /object_store #5432 [object-store] (dependabot[bot])
• Add BufWriter for Adapative Put / Multipart Upload #5431 [object-store] (tustvold)
• Enable anonymous access for MicrosoftAzure #5425 [object-store] (andrebsguedes)
• fix(object_store): Include Content-MD5 header for S3 DeleteObjects #5415 [object-store] (paraseba)
• docds(object_store): Mention HTTP/WebDAV in README #5409 [object-store] (Xuanwo)
• [object_store] Fix empty Multipart Upload for AWS S3 #5405 [object-store] (andrebsguedes)
• feat: S3 server-side encryption #5402 [object-store] (wjones127)
• Pull container name from URL for Azure blob #5371 [object-store] (bradvoth)
• docs(object-store): add warning to flush #5369 [object-store] (wjones127)
• Minor(docs): update master to main for DataFusion/Ballista #5363 (caicancai)
• Test parse_url_opts for HTTP (#5310) #5316 [object-store] (tustvold)
• Update IOx links #5312 [object-store] (tustvold)
• Pass options to HTTPBuilder in parse_url_opts (#5310) #5311 [object-store] (tustvold)
• Bump actions/cache from 3 to 4 #5308 (dependabot[bot])
• Remove localstack DynamoDB workaround (#5267) #5307 [object-store] (tustvold)
• refactor: log server error during object store retries #5294 [object-store] (crepererum)
• Prepare arrow 50.0.0 #5291 (tustvold)
• Enable JS tests again #5287 (domoritz)

object_store_0.9.0 (2024-01-05)

Full Changelog

Breaking changes:

• Remove deprecated try_with_option methods #5237 [object-store] (tustvold)
• object_store: full HTTP range support #5222 [object-store] (clbarnes)
• feat(object_store): use http1 by default #5204 [object-store] (wjones127)
• refactor: change object_store CA handling #5056 [object-store] (crepererum)

Implemented enhancements:

• Azure Signed URL Support #5232 [object-store]
• [object-store] Make aws region optional. #5211 [object-store]
• [object_store,gcp] Document GoogleCloudStorage Default Credentials #5187 [object-store]
• Support S3 Express One Zone #5140 [object-store]
• object_store: Allow 403 Forbidden for copy_if_not_exists S3 status code #5132 [object-store]
• Add copy_if_not_exists support for AmazonS3 via DynamoDB Lock Support #4880 [object-store]
• object_store: native certs, w/o webpki-roots #4870
• object_store: range request with suffix #4611

Fixed bugs:

• ObjectStore::get_opts Incorrectly Returns Response Size not Object Size #5272 [object-store]
• Single object store has limited throughput on GCS #5194 [object-store]
• local::tests::invalid_path fails during object store release verification #5035 [object-store]
• Object Store Doctest Failure with Default Features #5025 [object-store]

Documentation updates:

• Document default value of InstanceCredentialProvider #5188 [object-store] (justinabrahms)

Merged pull requests:

• Retry Safe/Read-Only Requests on Timeout #5278 [object-store] (tustvold)
• Fix ObjectMeta::size for range requests (#5272) #5276 [object-store] (tustvold)
• docs(object_store): Mention with_allow_http in docs of with_endpoint #5275 [object-store] (Xuanwo)
• Support S3 Express One Zone #5268 [object-store] (tustvold)
• feat(object_store): Azure url signing #5259 [object-store] (roeap)
• DynamoDB ConditionalPut #5247 [object-store] (tustvold)
• Default AWS region to us-east-1 (#5211) #5244 [object-store] (tustvold)
• ci: Fail Miri CI on first failure #5243 (Jefffrey)
• Bump actions/upload-pages-artifact from 2 to 3 #5229 (dependabot[bot])
• Bump actions/setup-python from 4 to 5 #5175 (dependabot[bot])
• fix: ensure take_fixed_size_list can handle null indices #5170 (westonpace)
• Bump actions/labeler from 4.3.0 to 5.0.0 #5167 (dependabot[bot])
• object_store: fix failing doctest with default features #5161 [object-store] (Jefffrey)
• Update rustls-pemfile requirement from 1.0 to 2.0 in /object_store #5155 [object-store] (dependabot[bot])
• Allow 403 for overwrite prevention #5134 [object-store] (emcake)
• Fix ObjectStore.LocalFileSystem.put_opts for blobfuse #5094 [object-store] (RobinLin666)
• Update itertools requirement from 0.11.0 to 0.12.0 in /object_store #5077 [object-store] (dependabot[bot])
• Add a PR under "Breaking changes" in the object_store 0.8.0 changelog #5063 [object-store] (carols10cents)
• Prepare arrow 49.0.0 #5054 (tustvold)
• Fix invalid_path test #5026 [object-store] (tustvold)
• Implement copy_if_not_exist for AmazonS3 using DynamoDB (#4880) #4918 [object-store] (tustvold)

object_store_0.8.0 (2023-11-02)

Full Changelog

Breaking changes:

• Remove ObjectStore::append #5016 [object-store] (tustvold)
• Don't panic on invalid Azure access key (#4972) #4974 [object-store] (tustvold)
• Return PutResult with an ETag from ObjectStore::put (#4934) #4944 [object-store] (tustvold)
• Add ObjectMeta::version and GetOptions::version (#4925) #4935 [object-store] (tustvold)
• Add GetOptions::head #4931 [object-store] (tustvold)
• Remove Nested async and Fallibility from ObjectStore::list #4930 [object-store] (tustvold)
• Add ObjectStore::put_opts / Conditional Put #4879 [object-store] (tustvold)

Implemented enhancements:

• Relax Path Safety on Parse #5019 [object-store]
• ObjectStore: hard to determine the cause of the error thrown from retry #5013 [object-store]
• continue existing multi-part upload #4961 [object-store]
• Simplify ObjectStore::List #4946 [object-store]
• Return ETag and Version on Put #4934 [object-store]
• Support Not Signing Requests in AmazonS3 #4927 [object-store]
• Get Object By Version #4925 [object-store]
• Plans for supporting Extension Array to support Fixed shape tensor Array #4890
• Conditional Put Support #4879 [object-store]
• creates_dir_if_not_present_append Test is Flaky #4872 [object-store]
• Release object_store 0.7.1 #4858 [object-store]
• Support User-Defined Object Metadata #4754 [object-store]
• APIs for directly managing multi-part uploads and saving potential parquet footers #4608

Fixed bugs:

• ObjectStore parse_url Incorrectly Handles URLs with Spaces #5017 [object-store]
• [objects-store]: periods/dots error in GCP bucket #4991 [object-store]
• Azure ImdsManagedIdentityProvider does not work in Azure functions #4976 [object-store]
• Panic when using an azure object store with an invalid access key #4972 [object-store]
• Handle Body Errors in AWS CompleteMultipartUpload #4965 [object-store]
• ObjectStore multiple_append Test is Flaky #4868 [object-store]
• [objectstore] Problem with special characters in file path #4454

Closed issues:

• Include onelake fabric path for https #5000 [object-store]
• [object_store] Support generating and using signed upload URLs #4763 [object-store]

Merged pull requests:

• Relax path safety (#5019) #5020 [object-store] (tustvold)
• Decode URL paths (#5017) #5018 [object-store] (tustvold)
• ObjectStore: make error msg thrown from retry more detailed #5012 [object-store] (Rachelint)
• Support onelake fabric paths in parse_url (#5000) #5002 [object-store] (tustvold)
• Object tagging (#4754) #4999 [object-store] (tustvold)
• [MINOR] No need to jump to web pages #4994 (smallzhongfeng)
• Pushdown list_with_offset for GCS #4993 [object-store] (tustvold)
• Support bucket name with . when parsing GCS URL (#4991) #4992 [object-store] (tustvold)
• Increase default timeout to 30 seconds #4989 [object-store] (tustvold)
• Conditional Put (#4879) #4984 [object-store] (tustvold)
• Update quick-xml requirement from 0.30.0 to 0.31.0 in /object_store #4983 [object-store] (dependabot[bot])
• Bump actions/setup-node from 3 to 4 #4982 (dependabot[bot])
• Support ImdsManagedIdentityProvider in Azure Functions (#4976) #4977 [object-store] (tustvold)
• Add MultiPartStore (#4961) (#4608) #4971 [object-store] (tustvold)
• Split gcp Module #4956 [object-store] (tustvold)
• Add module links in docs root #4955 [object-store] (tustvold)
• Prepare arrow 48.0.0 #4948 (tustvold)
• Allow opting out of request signing (#4927) #4929 [object-store] (tustvold)
• Default connection and request timeouts of 5 seconds #4928 [object-store] (tustvold)
• Support service_account in ApplicationDefaultCredentials and Use SelfSignedJwt #4926 [object-store] (tustvold)
• Generate ETags for InMemory and LocalFileSystem (#4879) #4922 [object-store] (tustvold)
• Cleanup object_store::retry client error handling #4915 [object-store] (tustvold)
• Fix integration tests #4889 (tustvold)
• Support Parsing Avro File Headers #4888 (tustvold)
• Update ring requirement from 0.16 to 0.17 in /object_store #4887 [object-store] (dependabot[bot])
• Add AWS presigned URL support #4876 [object-store] (carols10cents)
• Flush in creates_dir_if_not_present_append (#4872) #4874 [object-store] (tustvold)
• Flush in multiple_append test (#4868) #4869 [object-store] (tustvold)
• Enable new integration tests (#4828) #4862 (tustvold)

object_store_0.7.1 (2023-09-26)

Full Changelog

Implemented enhancements:

• Automatically Cleanup LocalFileSystem Temporary Files #4778 [object-store]
• object-store: Expose an async reader API for object store #4762
• Improve proxy support by using reqwest::Proxy as configuration #4713 [object-store]

Fixed bugs:

• object-store: http shouldn't perform range requests unless accept-ranges: bytes header is present #4839
• object-store: http-store fails when url doesn't have last-modified header on 0.7.0 #4831
• object-store fails to compile for wasm32-unknown-unknown with http feature #4776 [object-store]
• object-store: could not find header in client for http feature #4775 [object-store]
• LocalFileSystem Copy and Rename Don't Create Intermediate Directories #4760 [object-store]
• LocalFileSystem Copy is not Atomic #4758 [object-store]

Closed issues:

• object_store Azure Government Cloud functionality? #4853

Merged pull requests:

• Add ObjectStore BufReader (#4762) #4857 [object-store] (tustvold)
• Allow overriding azure endpoint #4854 [object-store] (tustvold)
• Minor: Improve object_store docs.rs landing page #4849 [object-store] (alamb)
• Error if Remote Ignores HTTP Range Header #4841 [object-store] (universalmind303)
• Perform HEAD request for HttpStore::head #4837 [object-store] (tustvold)
• fix: object store http header last modified #4834 [object-store] (universalmind303)
• Prepare arrow 47.0.0 #4827 (tustvold)
• ObjectStore Wasm32 Fixes (#4775) (#4776) #4796 [object-store] (tustvold)
• Best effort cleanup of staged upload files (#4778) #4792 [object-store] (tustvold)
• Relaxing type bounds on coalesce_ranges and collect_bytes #4787 [object-store] (sumerman)
• Update object_store chrono deprecations #4786 [object-store] (tustvold)
• Make coalesce_ranges and collect_bytes available for crate users #4784 [object-store] (sumerman)
• Bump actions/checkout from 3 to 4 #4767 (dependabot[bot])
• Make ObjectStore::copy Atomic and Automatically Create Parent Directories (#4758) (#4760) #4759 [object-store] (tustvold)
• Update nix requirement from 0.26.1 to 0.27.1 in /object_store #4744 [object-store] (viirya)
• Add with_proxy_ca_certificate and with_proxy_excludes #4714 [object-store] (gordonwang0)
• Update object_store Dependencies and Configure Dependabot #4700 [object-store] (tustvold)

object_store_0.7.0 (2023-08-15)

Full Changelog

Breaking changes:

• Add range and ObjectMeta to GetResult (#4352) (#4495) #4677 [object-store] (tustvold)

Implemented enhancements:

• Add AzureConfigKey::ContainerName #4629 [object-store]
• object_store: multipart ranges for HTTP #4612
• Make object_store::multipart public #4569 [object-store]
• object_store: Export ClientConfigKey and make the HttpBuilder more consistent with other builders #4515 [object-store]
• object_store/InMemory: Make clone() non-async #4496 [object-store]
• Add Range to GetResult::File #4352 [object-store]
• Support copy_if_not_exists for Cloudflare R2 (S3 API) #4190

Fixed bugs:

• object_store documentation is broken #4683 [object-store]
• Exports are not sufficient for configuring some object stores, for example minio running locally #4530 [object-store]
• object_store: Uploading empty file to S3 results in "411 Length Required" #4514 [object-store]
• GCP doesn't fetch public objects #4417 [object-store]

Closed issues:

• [object_store] when Create a AmazonS3 instance work with MinIO without set endpoint got error MissingRegion #4617
• AWS Profile credentials no longer working in object_store 0.6.1 #4556 [object-store]

Merged pull requests:

• Add AzureConfigKey::ContainerName (#4629) #4686 [object-store] (tustvold)
• Fix MSRV CI #4671 (tustvold)
• Use Config System for Object Store Integration Tests #4628 [object-store] (tustvold)
• Prepare arrow 45 #4590 (tustvold)
• Add Support for Microsoft Fabric / OneLake #4573 [object-store] (vmuddassir-msft)
• Cleanup multipart upload trait #4572 [object-store] (tustvold)
• Make object_store::multipart public #4570 [object-store] (yjshen)
• Handle empty S3 payloads (#4514) #4518 [object-store] (tustvold)
• object_store: Export ClientConfigKey and add HttpBuilder::with_config #4516 [object-store] (thehabbos007)
• object_store: Implement ObjectStore for Arc #4502 [object-store] (Turbo87)
• object_store/InMemory: Add fork() fn and deprecate clone() fn #4499 [object-store] (Turbo87)
• Bump actions/deploy-pages from 1 to 2 #4449 (dependabot[bot])
• gcp: Exclude authorization header when bearer empty #4418 [object-store] (vrongmeal)
• Support copy_if_not_exists for Cloudflare R2 (#4190) #4239 [object-store] (tustvold)

object_store_0.6.0 (2023-05-18)

Full Changelog

Breaking changes:

• Add ObjectStore::get_opts (#2241) #4212 [object-store] (tustvold)
• Simplify ObjectStore configuration pattern #4189 [object-store] (tustvold)
• object_store: fix: Incorrect parsing of https Path Style S3 url #4082 [object-store] (roeap)
• feat: add etag for objectMeta #3937 [object-store] (Weijun-H)

Implemented enhancements:

• Object Store Authorization #4223 [object-store]
• Use XML API for GCS #4209 [object-store]
• ObjectStore with_url Should Handle Path #4199
• Return Error on Invalid Config Value #4191 [object-store]
• Extensible ObjectStore Authentication #4163 [object-store]
• object_store: When using an AWS profile, obtain the default AWS region from the active profile #4158 [object-store]
• InMemory append API #4152 [object-store]
• Support accessing ipc Reader/Writer inner by reference #4121
• [object_store] Retry requests on connection error #4119 [object-store]
• object_store: Instantiate object store from provided url with store options #4047 [object-store]
• object_store: Builders (S3/Azure/GCS) are missing the get method to get the actual configuration information #4021 [object-store]

Fixed bugs:

• ObjectStore::head Returns Directory for LocalFileSystem and Hierarchical Azure #4230 [object-store]
• object_store: different behavior from aws cli for default profile #4137 [object-store]
• ImdsManagedIdentityOAuthProvider should send resource ID instead of OIDC scope #4096 [object-store]
• Update readme to remove reference to Jira #4091
• object_store: Incorrect parsing of https Path Style S3 url #4078 [object-store]
• [object_store] local::tests::test_list_root test fails during release verification #3772 [object-store]

Merged pull requests:

• Remove AWS_PROFILE support #4238 [object-store] (tustvold)
• Expose AwsAuthorizer #4237 [object-store] (tustvold)
• Expose CredentialProvider #4235 [object-store] (tustvold)
• Return NotFound for directories in Head and Get (#4230) #4231 [object-store] (tustvold)
• Standardise credentials API (#4223) (#4163) #4225 [object-store] (tustvold)
• Extract Common Listing and Retrieval Functionality #4220 [object-store] (tustvold)
• feat(object-store): extend Options API for http client #4208 [object-store] (roeap)
• Consistently use GCP XML API #4207 [object-store] (tustvold)
• Implement list_with_offset for PrefixStore #4203 [object-store] (tustvold)
• Allow setting ClientOptions with Options API #4202 [object-store] (tustvold)
• Create ObjectStore from URL and Options (#4047) #4200 [object-store] (tustvold)
• Skip test_list_root on OS X (#3772) #4198 [object-store] (tustvold)
• Recognise R2 URLs for S3 object store (#4190) #4194 [object-store] (tustvold)
• Fix ImdsManagedIdentityProvider (#4096) #4193 [object-store] (tustvold)
• Deffered Object Store Config Parsing (#4191) #4192 [object-store] (tustvold)
• Object Store (AWS): Support dynamically resolving S3 bucket region #4188 [object-store] (mr-brobot)
• Faster prefix match in object_store path handling #4164 [object-store] (tustvold)
• Object Store (AWS): Support region configured via named profile #4161 [object-store] (mr-brobot)
• InMemory append API #4153 [object-store] (berkaysynnada)
• docs: fix the wrong ln command in CONTRIBUTING.md #4139 (SteveLauC)
• Display the file path in the error message when failed to open credentials file for GCS #4124 [object-store] (haoxins)
• Retry on Connection Errors #4120 [object-store] (kindly)
• Simplify reference to GitHub issues #4092 (bkmgit)
• Use reqwest build_split #4039 [object-store] (tustvold)
• Fix object_store CI #4037 (tustvold)
• Add get_config_value to AWS/Azure/GCP Builders #4035 [object-store] (r4ntix)
• Update AWS SDK #3993 [object-store] (tustvold)

object_store_0.5.6 (2023-03-30)

Full Changelog

Implemented enhancements:

• Document ObjectStore::list Ordering #3975 [object-store]
• Add option to start listing at a particular key #3970 [object-store]
• Implement ObjectStore for trait objects #3865 [object-store]
• Add ObjectStore::append #3790 [object-store]
• Make InMemory object store track last modified time for each entry #3782 [object-store]
• Support Unsigned S3 Payloads #3737 [object-store]
• Add Content-MD5 or checksum header for using an Object Locked S3 #3725 [object-store]

Fixed bugs:

• LocalFileSystem::put is not Atomic #3780 [object-store]

Merged pull requests:

• Add ObjectStore::list_with_offset (#3970) #3973 [object-store] (tustvold)
• Remove incorrect validation logic on S3 bucket names #3947 [object-store] (rtyler)
• Prepare arrow 36 #3935 (tustvold)
• fix: Specify content length for gcp copy request #3921 [object-store] (scsmithr)
• Revert structured ArrayData (#3877) #3894 (tustvold)
• Add support for checksum algorithms in AWS #3873 [object-store] (trueleo)
• Rename PrefixObjectStore to PrefixStore #3870 [object-store] (tustvold)
• Implement append for LimitStore, PrefixObjectStore, ThrottledStore #3869 [object-store] (tustvold)
• Supporting metadata fetch without open file read mode #3868 [object-store] (metesynnada)
• Impl ObjectStore for trait object #3866 [object-store] (Kinrany)
• Update quick-xml requirement from 0.27.0 to 0.28.0 #3857 [object-store] (dependabot[bot])
• Update changelog for 35.0.0 #3843 (tustvold)
• Cleanup ApplicationDefaultCredentials #3799 [object-store] (tustvold)
• Make InMemory object store track last modified time for each entry #3796 [object-store] (Weijun-H)
• Add ObjectStore::append #3791 [object-store] (tustvold)
• Make LocalFileSystem::put atomic (#3780) #3781 [object-store] (tustvold)
• Add support for unsigned payloads in aws #3741 [object-store] (trueleo)

object_store_0.5.5 (2023-02-27)

Full Changelog

Implemented enhancements:

• object_store: support azure cli credential #3697 [object-store]
• object_store: support encoded path as input #3651 [object-store]

Fixed bugs:

• object-store: aws_profile fails to load static credentials #3765 [object-store]
• Inconsistent Behaviour Listing File #3712 [object-store]
• object_store: bearer token is azure is used like access key #3696 [object-store]

Merged pull requests:

• object-store: fix handling of AWS profile credentials without expiry #3766 [object-store] (helmus)
• update object_store deps to patch potential security vulnerabilities #3761 [object-store] (spencerbart)
• Filter exact list prefix matches for azure gen2 accounts #3714 [object-store] (roeap)
• Filter exact list prefix matches for MemoryStore and HttpStore (#3712) #3713 [object-store] (tustvold)
• object_store: azure cli authorization #3698 [object-store] (roeap)
• object_store: add Path::from_url_path #3663 [object-store] (jychen7)

object_store_0.5.4 (2023-01-30)

Full Changelog

Implemented enhancements:

• [object_store] support more identity based auth flows for azure #3580 [object-store]
• Implement workload identity and application default credentials for GCP object store. #3533 [object-store]
• Support GCP Workload Identity #3490 [object-store]
• Allow providing service account key directly when building GCP object store client #3488 [object-store]

Closed issues:

• object_store: temporary aws credentials not refreshed? #3446 [object-store]

Merged pull requests:

• Final tweaks to 32.0.0 changelog #3618 (tustvold)
• Update AWS SDK #3617 [object-store] (tustvold)
• Add ClientOption.allow_insecure #3600 [object-store] (poelzi)
• [object_store] support azure managed and workload identities #3581 [object-store] (roeap)
• Additional GCP authentication #3541 [object-store] (winding-lines)
• Update aws-config and aws-types requirements from 0.52 to 0.53 #3539 [object-store] (viirya)
• Use GHA concurrency groups (#3495) #3538 (tustvold)
• Remove azurite test exception #3497 [object-store] (tustvold)
• feat: Allow providing a service account key directly for GCS #3489 [object-store] (scsmithr)

object_store_0.5.3 (2023-01-04)

Full Changelog

Implemented enhancements:

• Derive Clone for the builders in object-store. #3419
• Add a constant prefix object store wrapper #3328 [object-store]
• Add support for content-type while uploading files through ObjectStore API #3300 [object-store]
• Add HttpStore #3294 [object-store]
• Add support for Azure Data Lake Storage Gen2 (aka: ADLS Gen2) in Object Store library #3283
• object_store: Add Put and Multipart Upload Doc Examples #2863 [object-store]

Closed issues:

• Only flush buffered multi-part data on poll_shutdown not on poll_flush #3390 [object-store]

Merged pull requests:

• object_store: builder configuration api #3436 [object-store] (roeap)
• Derive Clone for ObjectStore builders and Make URL Parsing Stricter (#3419) #3424 [object-store] (tustvold)
• Add Put and Multipart Put doc examples #3420 [object-store] (GeauxEric)
• object_store: update localstack instructions #3403 [object-store] (wjones127)
• object_store: Flush buffered multipart only during poll_shutdown #3397 [object-store] (askoa)
• Update quick-xml to 0.27 #3395 [object-store] (tustvold)
• Add HttpStore (#3294) #3380 [object-store] (tustvold)
• add support for content-type in ClientOptions #3358 [object-store] (ByteBaker)
• Update AWS SDK #3349 [object-store] (tustvold)
• Upstream newline_delimited_stream and ChunkedStore from DataFusion #3341 [object-store] (tustvold)
• feat(object_store): add PrefixObjectStore #3329 [object-store] (roeap)
• feat(object_store): parse well-known storage urls #3327 [object-store] (roeap)
• Disable getrandom object_store #3278 [object-store] (tustvold)
• Reload token from AWS_WEB_IDENTITY_TOKEN_FILE #3274 [object-store] (tustvold)
• Minor: skip aws integration test if TEST_INTEGRATION is not set #3262 [object-store] (viirya)

object_store_0.5.2 (2022-12-02)

Full Changelog

Implemented enhancements:

• Object Store: Allow custom reqwest client #3127
• socks5 proxy support for the object_store crate #2989 [object-store]
• Cannot query S3 paths containing whitespace #2799 [object-store]

Fixed bugs:

• object_store(gcp): GCP complains about content-length for copy #3235
• object_store(aws): EntityTooSmall error on multi-part upload #3233 [object-store]

Merged pull requests:

• Add more ClientConfig Options for Object Store RequestBuilder (#3127) #3256 [object-store] (tustvold)
• Add ObjectStore ClientConfig #3252 [object-store] (tustvold)
• fix(object_store,gcp): test copy_if_not_exist #3236 [object-store] (wjones127)
• fix(object_store,aws,gcp): multipart upload enforce size limit of 5 MiB not 5MB #3234 [object-store] (wjones127)
• object_store: add support for using proxy_url for connection testing #3109 [object-store] (sum12)
• Update AWS SDK #2974 [object-store] (tustvold)
• Update quick-xml requirement from 0.25.0 to 0.26.0 #2918 [object-store] (dependabot[bot])
• Support building object_store and parquet on wasm32-unknown-unknown target #2896 [object-store] (jondo2010)
• Add experimental AWS_PROFILE support (#2178) #2891 [object-store] (tustvold)

object_store_0.5.1 (2022-10-04)

Full Changelog

Implemented enhancements:

• Allow HTTP S3 URLs #2806
• object_store: support AWS ECS instance credentials #2802
• Object Store S3 Alibaba Cloud OSS support #2777 [object-store]
• Expose option to use GCS object store in integration tests #2627 [object-store]

Fixed bugs:

• S3 Signature Error Performing List With Prefix Containing Spaces #2800 [object-store]
• Erratic Behaviour if Incorrect S3 Region Configured #2795 [object-store]

Merged pull requests:

• Support for overriding instance metadata endpoint #2811 [object-store] (wjones127)
• Allow Configuring non-TLS HTTP Connections in AmazonS3Builder::from_env #2807 [object-store] (avantgardnerio)
• Fix S3 query canonicalization (#2800) #2801 [object-store] (tustvold)
• Handle incomplete HTTP redirects missing LOCATION (#2795) #2796 [object-store] (tustvold)
• Handle S3 virtual host request type #2782 [object-store] (askoa)
• Fix object_store multipart uploads on S3 Compatible Stores #2731 [object-store] (mildbyte)

object_store_0.5.0 (2022-09-08)

Full Changelog

Breaking changes:

• Replace azure sdk with custom implementation #2509 [object-store] (roeap)
• Replace rusoto with custom implementation for AWS (#2176) #2352 [object-store] (tustvold)

Implemented enhancements:

• IMDSv1 Fallback for S3 #2609 [object-store]
• Print Response Body On Error #2572 [object-store]
• Coalesce Ranges Parallel Fetch #2562 [object-store]
• Support Coalescing Out-of-Order Ranges #2561 [object-store]
• object_store: Add TokenProvider authorization to azure #2373 [object-store]
• AmazonS3Builder::from_env to populate credentials from environment #2361 [object-store]
• AmazonS3 Support IMDSv2 #2350 [object-store]

Fixed bugs:

• Retry Logic Fails to Retry Server Errors #2573 [object-store]
• Fix multiple part uploads at once making vector size inconsistent #2681 [object-store] (gruuya)
• Fix panic in object_store::util::coalesce_ranges #2554 [object-store] (thinkharderdev)

Merged pull requests:

• update doc for object_store copy_if_not_exists #2653 [object-store] (JanKaul)
• Update quick-xml 0.24 #2625 [object-store] (tustvold)
• Add IMDSv1 fallback (#2609) #2610 [object-store] (tustvold)
• ObjectStore cleanup (#2587) #2590 [object-store] (tustvold)
• Fix retry logic (#2573) (#2572) #2574 [object-store] (tustvold)
• Improve coalesce_ranges (#2561) (#2562) #2563 [object-store] (tustvold)
• Update environment variable name for amazonS3builder in integration (#2550) #2553 [object-store] (amrltqt)
• Build AmazonS3builder from environment variables (#2361) #2536 [object-store] (amrltqt)
• feat: add token provider authorization to azure store #2374 [object-store] (roeap)

object_store_0.4.0 (2022-08-10)

Full Changelog

Implemented enhancements:

• Relax Path Validation to Allow Any Percent-Encoded Sequence #2355 [object-store]
• Support get_multi_ranges in ObjectStore #2293
• object_store: Create explicit test for symlinks #2206 [object-store]
• object_store: Make builder style configuration for object stores #2203 [object-store]
• object_store: Add example in the main documentation readme #2202 [object-store]

Fixed bugs:

• Azure/S3 Storage Fails to Copy Blob with URL-encoded Path #2353 [object-store]
• Accessing a file with a percent-encoded name on the filesystem with ObjectStore LocalFileSystem #2349 [object-store]

Documentation updates:

• Improve object_store crate documentation #2260 [object-store] (alamb)

Merged pull requests:

• Canonicalize filesystem paths in user-facing APIs (#2370) #2371 [object-store] (tustvold)
• Fix object_store lint #2367 [object-store] (tustvold)
• Relax path validation (#2355) #2356 [object-store] (tustvold)
• Fix Copy from percent-encoded path (#2353) #2354 [object-store] (tustvold)
• Add ObjectStore::get_ranges (#2293) #2336 [object-store] (tustvold)
• Remove vestigal object_store/.circleci/ #2337 (alamb)
• Handle symlinks in LocalFileSystem (#2206) #2269 [object-store] (tustvold)
• Retry GCP requests on server error #2243 [object-store] (tustvold)
• Add LimitStore (#2175) #2242 [object-store] (tustvold)
• Only trigger arrow CI on changes to arrow #2227 (alamb)
• Update instructions on how to join the Slack channel #2219 (HaoYang670)
• Add Builder style config objects for object_store #2204 [object-store] (alamb)
• Ignore broken symlinks for LocalFileSystem object store #2195 [object-store] (jccampagne)
• Change CI names to match crate names #2189 (alamb)
• Split most arrow specific CI checks into their own workflows (reduce common CI time to 21 minutes) #2168 (alamb)
• Remove another attempt to cache target directory in action.yaml #2167 (alamb)
• Run actions on push to master, pull requests #2166 (alamb)
• Break parquet_derive and arrow_flight tests into their own workflows #2165 (alamb)
• Only run integration tests when arrow changes #2152 (alamb)
• Break out docs CI job to its own github action #2151 (alamb)
• Do not pretend to cache rust build artifacts, speed up CI by ~20% #2150 (alamb)
• Port object_store integration tests, use github actions #2148 [object-store] (alamb)
• Port Add stream upload (multi-part upload) #2147 [object-store] (alamb)
• Increase upper wait time to reduce flakiness of object store test #2142 [object-store] (viirya)

* This Changelog was automatically generated by github_changelog_generator