Skip to content

Commit

Permalink
docs: service doc for s3 (#2376)
Browse files Browse the repository at this point in the history 
* docs: service doc for s3

Signed-off-by: suyanhanx <suyanhanx@gmail.com>

* try

Signed-off-by: suyanhanx <suyanhanx@gmail.com>

* fix

Signed-off-by: suyanhanx <suyanhanx@gmail.com>

* add region in via_map examples

Signed-off-by: suyanhanx <suyanhanx@gmail.com>

---------

Signed-off-by: suyanhanx <suyanhanx@gmail.com>
  • Loading branch information
@suyanhanx
suyanhanx authored May 31, 2023
1 parent 59af646 commit 174533f
Show file tree
Hide file tree
Showing 9 changed files with 367 additions and 251 deletions.
10 changes: 5 additions & 5 deletions core/src/services/azblob/docs.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -15,7 +15,7 @@ This service can be used to:
- [x] presign
- [ ] blocking

# Configuration
## Configuration

- `root`: Set the work dir for backend.
- `container`: Set the container name for backend.
Expand All @@ -25,18 +25,18 @@ This service can be used to:

Refer to public API docs for more information.

# Examples
## Examples

This example works on [Azurite](https://github.com/Azure/Azurite) for local developments.

## Start local blob service
### Start local blob service

```shell
docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite
az storage container create --name test --connection-string "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"
```

## Init OpenDAL Operator
### Init OpenDAL Operator

### Via Builder

Expand Down
10 changes: 4 additions & 6 deletions core/src/services/azdfs/docs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ As known as `abfs`, `azdfs` or `azdls`.

This service will visist the [ABFS](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver) URI supported by [Azure Data Lake Storage Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction).

# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -18,7 +18,7 @@ This service can be used to:
- [ ] presign
- [ ] blocking

# Configuration
## Configuration

- `root`: Set the work dir for backend.
- `filesystem`: Set the filesystem name for backend.
Expand All @@ -28,9 +28,7 @@ This service can be used to:

Refer to public API docs for more information.

# Example

## Init OpenDAL Operator
## Example

### Via Builder

Expand Down Expand Up @@ -69,4 +67,4 @@ async fn main() -> Result<()> {

Ok(())
}
```
```
10 changes: 5 additions & 5 deletions core/src/services/cos/docs.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -14,7 +14,7 @@ This service can be used to:
- [ ] presign
- [ ] blocking

# Configuration
## Configuration

- `root`: Set the work directory for backend
- `bucket`: Set the container name for backend
Expand All @@ -24,9 +24,9 @@ This service can be used to:

You can refer to [`CosBuilder`]'s docs for more information

# Example
## Example

## Via Builder
### Via Builder

```rust
use anyhow::Result;
Expand Down Expand Up @@ -54,4 +54,4 @@ async fn main() -> Result<()> {

Ok(())
}
```
```
4 changes: 2 additions & 2 deletions core/src/services/dashmap/docs.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -12,4 +12,4 @@ This service can be used to:
- [ ] list
- [x] scan
- [ ] presign
- [ ] blocking
- [ ] blocking
6 changes: 3 additions & 3 deletions core/src/services/fs/docs.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Capabilities
## Capabilities

This service can be used to:

Expand All @@ -15,13 +15,13 @@ This service can be used to:
- [ ] ~~presign~~
- [x] blocking

# Configuration
## Configuration

- `root`: Set the work dir for backend.

Refer to public API docs for more information.

# Example
## Example

### Via Builder

Expand Down
225 changes: 2 additions & 223 deletions core/src/services/s3/backend.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,232 +55,11 @@ static ENDPOINT_TEMPLATES: Lazy<HashMap<&'static str, &'static str>> = Lazy::new
});

const DEFAULT_WRITE_MIN_SIZE: usize = 8 * 1024 * 1024;

/// Aws S3 and compatible services (including minio, digitalocean space, Tencent Cloud Object Storage(COS) and so on) support.
/// For more information about s3-compatible services, refer to [Compatible Services](#compatible-services).
///
/// # Capabilities
///
/// This service can be used to:
///
/// - [x] stat
/// - [x] read
/// - [x] write
/// - [x] create_dir
/// - [x] delete
/// - [x] copy
/// - [ ] rename
/// - [x] list
/// - [x] scan
/// - [x] presign
/// - [ ] blocking
///
/// # Configuration
///
/// - `root`: Set the work dir for backend.
/// - `bucket`: Set the container name for backend.
/// - `endpoint`: Set the endpoint for backend.
/// - `region`: Set the region for backend.
/// - `access_key_id`: Set the access_key_id for backend.
/// - `secret_access_key`: Set the secret_access_key for backend.
/// - `security_token`: Set the security_token for backend.
/// - `default_storage_class`: Set the default storage_class for backend.
/// - `server_side_encryption`: Set the server_side_encryption for backend.
/// - `server_side_encryption_aws_kms_key_id`: Set the server_side_encryption_aws_kms_key_id for backend.
/// - `server_side_encryption_customer_algorithm`: Set the server_side_encryption_customer_algorithm for backend.
/// - `server_side_encryption_customer_key`: Set the server_side_encryption_customer_key for backend.
/// - `server_side_encryption_customer_key_md5`: Set the server_side_encryption_customer_key_md5 for backend.
/// - `disable_config_load`: Disable aws config load from env
/// - `enable_virtual_host_style`: Enable virtual host style.
///
/// Refer to [`S3Builder`]'s public API docs for more information.
///
/// # Temporary security credentials
///
/// OpenDAL now provides support for S3 temporary security credentials in IAM.
///
/// The way to take advantage of this feature is to build your S3 backend with `Builder::security_token`.
///
/// But OpenDAL will not refresh the temporary security credentials, please keep in mind to refresh those credentials in time.
///
/// # Server Side Encryption
///
/// OpenDAL provides full support of S3 Server Side Encryption(SSE) features.
///
/// The easiest way to configure them is to use helper functions like
///
/// - SSE-KMS: `server_side_encryption_with_aws_managed_kms_key`
/// - SSE-KMS: `server_side_encryption_with_customer_managed_kms_key`
/// - SSE-S3: `server_side_encryption_with_s3_key`
/// - SSE-C: `server_side_encryption_with_customer_key`
///
/// If those functions don't fulfill need, low-level options are also provided:
///
/// - Use service managed kms key
/// - `server_side_encryption="aws:kms"`
/// - Use customer provided kms key
/// - `server_side_encryption="aws:kms"`
/// - `server_side_encryption_aws_kms_key_id="your-kms-key"`
/// - Use S3 managed key
/// - `server_side_encryption="AES256"`
/// - Use customer key
/// - `server_side_encryption_customer_algorithm="AES256"`
/// - `server_side_encryption_customer_key="base64-of-your-aes256-key"`
/// - `server_side_encryption_customer_key_md5="base64-of-your-aes256-key-md5"`
///
/// After SSE have been configured, all requests send by this backed will attach those headers.
///
/// Reference: [Protecting data using server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html)
///
/// # Example
///
/// ## Basic Setup
///
/// ```no_run
/// use std::sync::Arc;
///
/// use anyhow::Result;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// // Create s3 backend builder.
/// let mut builder = S3::default();
/// // Set the root for s3, all operations will happen under this root.
/// //
/// // NOTE: the root must be absolute path.
/// builder.root("/path/to/dir");
/// // Set the bucket name, this is required.
/// builder.bucket("test");
/// // Set the endpoint.
/// //
/// // For examples:
/// // - "https://s3.amazonaws.com"
/// // - "http://127.0.0.1:9000"
/// // - "https://oss-ap-northeast-1.aliyuncs.com"
/// // - "https://cos.ap-seoul.myqcloud.com"
/// //
/// // Default to "https://s3.amazonaws.com"
/// builder.endpoint("https://s3.amazonaws.com");
/// // Set the access_key_id and secret_access_key.
/// //
/// // OpenDAL will try load credential from the env.
/// // If credential not set and no valid credential in env, OpenDAL will
/// // send request without signing like anonymous user.
/// builder.access_key_id("access_key_id");
/// builder.secret_access_key("secret_access_key");
///
/// let op: Operator = Operator::new(builder)?.finish();
///
/// Ok(())
/// }
/// ```
///
/// ## S3 with SSE-C
///
/// ```no_run
/// use anyhow::Result;
/// use log::info;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// let mut builder = S3::default();
///
/// // Setup builders
///
/// // Enable SSE-C
/// builder.server_side_encryption_with_customer_key("AES256", "customer_key".as_bytes());
///
/// let op = Operator::new(builder)?.finish();
/// info!("operator: {:?}", op);
///
/// // Writing your testing code here.
///
/// Ok(())
/// }
/// ```
///
/// ## S3 with SSE-KMS and aws managed kms key
///
/// ```no_run
/// use anyhow::Result;
/// use log::info;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// let mut builder = S3::default();
///
/// // Setup builders
///
/// // Enable SSE-KMS with aws managed kms key
/// builder.server_side_encryption_with_aws_managed_kms_key();
///
/// let op = Operator::new(builder)?.finish();
/// info!("operator: {:?}", op);
///
/// // Writing your testing code here.
///
/// Ok(())
/// }
/// ```
///
/// ## S3 with SSE-KMS and customer managed kms key
///
/// ```no_run
/// use anyhow::Result;
/// use log::info;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// let mut builder = S3::default();
///
/// // Setup builders
///
/// // Enable SSE-KMS with customer managed kms key
/// builder.server_side_encryption_with_customer_managed_kms_key("aws_kms_key_id");
///
/// let op = Operator::new(builder)?.finish();
/// info!("operator: {:?}", op);
///
/// // Writing your testing code here.
///
/// Ok(())
/// }
/// ```
///
/// ## S3 with SSE-S3
///
/// ```no_run
/// use anyhow::Result;
/// use log::info;
/// use opendal::services::S3;
/// use opendal::Operator;
///
/// #[tokio::main]
/// async fn main() -> Result<()> {
/// let mut builder = S3::default();
///
/// // Setup builders
///
/// // Enable SSE-S3
/// builder.server_side_encryption_with_s3_key();
///
/// let op = Operator::new(builder)?.finish();
/// info!("operator: {:?}", op);
///
/// // Writing your testing code here.
///
/// Ok(())
/// }
/// ```
///
/// # Compatible Services
#[doc = include_str!("docs.md")]
#[doc = include_str!("compatible_services.md")]
#[derive(Default)]
pub struct S3Builder {
Expand Down
Loading

0 comments on commit 174533f

Please sign in to comment.