Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README and INSTALL documentation for 1.0.0 release #439

Merged
merged 1 commit into from
Aug 9, 2023
Merged

Update README and INSTALL documentation for 1.0.0 release #439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/bug-report.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ body:
attributes:
label: Mountpoint for Amazon S3 version
description: |
Which version of the mountpoint are you using? You can find out using `mount-s3 --version`.
Which version of Mountpoint are you using? You can find out using `mount-s3 --version`.
If you're building from source or a fork, please state that.
placeholder: mountpoint-s3 x.y.z
placeholder: mount-s3 x.y.z
validations:
required: true
- type: input
Expand Down
126 changes: 37 additions & 89 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,118 +1,66 @@
# Mountpoint for Amazon S3

[![CI Status](https://github.com/awslabs/mountpoint-s3/actions/workflows/integration_main.yml/badge.svg?branch=main)](https://github.com/awslabs/mountpoint-s3/actions/workflows/integration_main.yml)
[![Tests](https://github.com/awslabs/mountpoint-s3/actions/workflows/tests.yml/badge.svg)](https://github.com/awslabs/mountpoint-s3/actions/workflows/tests.yml)
[![Integration tests](https://github.com/awslabs/mountpoint-s3/actions/workflows/integration_main.yml/badge.svg?branch=main)](https://github.com/awslabs/mountpoint-s3/actions/workflows/integration_main.yml)

Mountpoint for Amazon S3 is a simple, high-throughput file client for [mounting an Amazon S3 bucket as a local file system](https://aws.amazon.com/blogs/storage/the-inside-story-on-mountpoint-for-amazon-s3-a-high-performance-open-source-file-client/). With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through file operations like `open` and `read`. Mountpoint for Amazon S3 automatically translates these operations into S3 object API calls, giving your applications access to the elastic storage and throughput of Amazon S3 through a file interface.

Mountpoint for Amazon S3 is optimized for read-heavy workloads that need high throughput, and does not implement the full POSIX specification for file systems. For example, Mountpoint for Amazon S3 only supports write operations to new files, and writes must be made to the file sequentially. See [SEMANTICS.md](doc/SEMANTICS.md) for a detailed description of Mountpoint for Amazon S3's behavior and POSIX support and how they could affect your application. For workloads that need full POSIX support, we recommend [Amazon FSx for Lustre](https://aws.amazon.com/fsx/lustre/) and its [support for linking S3 buckets](https://docs.aws.amazon.com/fsx/latest/LustreGuide/create-dra-linked-data-repo.html).
Mountpoint for Amazon S3 is optimized for applications that need high read throughput to large objects, potentially from many clients at once, and to write new objects sequentially from a single client at a time. This means it's a great fit for applications that use a file interface to:
* read large objects from S3, potentially from many instances concurrently, without downloading them to local storage first
* access only some S3 objects out of a larger data set, but can't predict which objects in advance
* upload their output to S3 directly, or upload files from local storage with tools like `cp`

## Current status

Mountpoint for Amazon S3 is **currently an alpha release and should not be used in production**. We're tracking its production readiness and future features on the [Mountpoint for Amazon S3 public roadmap](https://github.com/orgs/awslabs/projects/84). We're especially interested in early feedback on features, performance, and compatibility. Please send feedback by [opening a GitHub issue](https://github.com/awslabs/mountpoint-s3/issues/new/choose).

This release has some notable restrictions:
* The only way to install the alpha release of Mountpoint for Amazon S3 is by compiling from source (see [Installation](#installation) below). This will change in a future release.
* Manual endpoint configuration might be required for some S3 customers (see [Configuration and usage](#configuration-and-usage) below).
* Objects written with Mountpoint for Amazon S3 are [always stored in the S3 Standard storage class](https://github.com/awslabs/mountpoint-s3/issues/34).

## Getting started

The alpha release of Mountpoint for Amazon S3 is only available by building from source, and only supports Linux.

### Installation
but probably not the right fit for applications that:
* use file operations that S3 doesn't natively support, like directory renaming or symlinks
* make edits or appends to existing files in-place (don't work on your Git repository or SQLite database in Mountpoint 😄)

First, install the necessary dependencies. For RPM-based distributions (Amazon Linux 2 and 2023, Fedora, CentOS, etc):
Mountpoint for Amazon S3 does not implement all the features of a POSIX file system, and there are some differences that may affect compatibility with your application. See [Mountpoint file system behavior](doc/SEMANTICS.md) for a detailed description of Mountpoint for Amazon S3's behavior and POSIX support and how they could affect your application.

sudo yum install fuse fuse-devel cmake3 clang git pkg-config

or for apt-based distributions (Debian, Ubuntu, etc):

sudo apt install fuse libfuse-dev cmake clang git pkg-config

Second, install the Rust compiler via [rustup](https://rustup.rs/):

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

Now clone this repository and its submodules:

git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git
## Current status

Finally, compile the client:
Mountpoint for Amazon S3 is generally available! We're tracking future feature development on the [Mountpoint for Amazon S3 public roadmap](https://github.com/orgs/awslabs/projects/84). We're always interested in feedback on features, performance, and compatibility. Please send feedback by [opening a new GitHub issue](https://github.com/awslabs/mountpoint-s3/issues/new/choose) or adding your input to an existing roadmap issue.

cd mountpoint-s3
cargo build --release
## Getting started

The final binary will be at `target/release/mount-s3`.
The following instructions assume you have the binary on your path.
Add it to a location on your path now, or prefix `mount-s3` commands with the path to the binary.
Run these two commands to install Mountpoint for Amazon S3 on your Amazon Linux EC2 instance (for Graviton instances, replace `x86_64` with `arm64` in the URL):

### Configuration and usage
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
sudo yum install -y ./mount-s3.rpm

To use Mountpoint for Amazon S3, your host needs [access to valid AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). For example, you could [create a new IAM user and add its credentials to the `~/.aws/credentials` file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html), or [associate your EC2 instance with an IAM role](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
On Ubuntu, use these commands instead (for Graviton instances, replace `x86_64` with `arm64` in the URL):

Launch Mountpoint for Amazon S3 by running the `mount-s3` command.
Replace `DOC-EXAMPLE-BUCKET` with the name of your bucket,
and specify the directory in which your S3 bucket should be mounted (here, `~/mnt`):
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
sudo apt-get install -y ./mount-s3.deb

mkdir ~/mnt
mount-s3 DOC-EXAMPLE-BUCKET ~/mnt
See [Installing Mountpoint for Amazon S3](doc/INSTALL.md) for detailed instructions and other installation options (including Docker or building from source).

The client will run in the background by default, and the `~/mnt` directory now gives access to the objects in your S3 bucket.
Once you've got Mountpoint for Amazon S3 installed, you can mount your Amazon S3 bucket.
You'll need valid AWS credentials to access your bucket;
for example, Mountpoint will automatically use credentials from [an IAM role associated with your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html),
or you can use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.

To unmount your S3 bucket and exit Mountpoint for Amazon S3, run the `umount` command:
To mount your bucket, run this command,
replacing `DOC-EXAMPLE-BUCKET` with the name of your bucket
and `/path/to/mount` with the directory you want to mount the bucket to:

umount ~/mnt
mount-s3 DOC-EXAMPLE-BUCKET /path/to/mount

### Additional configuration
Now you can work with your bucket contents as if they were a local file system:

Mountpoint for Amazon S3 supports the same [configuration options for AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) as the AWS CLI. For example, to use a different profile from your `~/.aws/config` configuration file, set the `AWS_PROFILE` environment variable or pass the `--profile` argument to `mount-s3`.
ls /path/to/mount
echo "Hello World!" > /path/to/mount/Data.txt
cat /path/to/mount/Data.txt

To use Mountpoint for Amazon S3 with [S3 Access Points](https://aws.amazon.com/s3/features/access-points/), use the [bucket-style access point alias](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-alias.html) of the access point as the bucket name to mount.
When you're finished accessing your bucket, you can unmount it (you might need `sudo`):

For the alpha release, additional configuration is required to use Mountpoint for Amazon S3 in some circumstances:
umount /path/to/mount

* **AWS GovCloud, China (Beijing), and China (Ningxia) regions**: manually specify the S3 endpoint with the `--endpoint-url` command-line argument **and** the region name with the `--region` argument. For example, for the AWS GovCloud (US-West) region, specify `--endpoint-url https://s3.us-gov-west-1.amazonaws.com --region us-gov-west-1`. See [the S3 endpoints documentation](https://docs.aws.amazon.com/general/latest/gr/s3.html) for more details. These regions will be supported via only the `--region` command-line argument in the future (https://github.com/awslabs/mountpoint-s3/issues/4).
* **FIPS, dualstack, and transfer acceleration endpoints**: manually specify the S3 endpoint with the `--endpoint-url` command-line argument. See [the S3 endpoints documentation](https://docs.aws.amazon.com/general/latest/gr/s3.html) for more details. These endpoints will be configurable in the future (https://github.com/awslabs/mountpoint-s3/issues/4).
See [Configuring Mountpoint for Amazon S3](doc/CONFIGURATION.md) for more details on how to configure and use Mountpoint, including options for providing AWS credentials.

#### Compatibility with other storage services
### Compatibility with other storage services

Mountpoint for Amazon S3 is designed for high-performance access to the Amazon S3 service. While it may be functional against other storage services that use S3-like APIs, we aren't able to provide support for those use cases, and they may inadvertently break when we make changes to better support Amazon S3. We welcome contributions of minor compatibility fixes or performance improvements for these services if the changes can be tested against Amazon S3.

### Alternative installation options

#### Mountpoint for Amazon S3 using Docker

A sample [Dockerfile](docker/Dockerfile) is provided, using Amazon Linux 2023 as a base.

First, build the container image:

cd mountpoint-s3/docker
docker build -t mountpoint-s3 .

Before running the container, you'll need to provide AWS credentials as [described above](#additional-configuration).
We recommend using short-term credentials whenever possible.
For example, if you run your containers on an EC2 instance,
you can [associate an IAM role with the instance](https://docs.aws.amazon.com/sdkref/latest/guide/access-iam-roles-for-ec2.html)
and Mountpoint for Amazon S3 will automatically assume that role at startup.
[Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) also supports short-term credentials
by [associating an IAM role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) with your ECS task.
If you need to use long-term credentials, set the [`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html), and pass those variables to Docker with the `--env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY` arguments.

To launch the container, run the following command,
replacing `DOC-EXAMPLE-BUCKET` with the name of your S3 bucket:

docker run -it --cap-add SYS_ADMIN --device /dev/fuse \
mountpoint-s3 /bin/bash -c "./mount-s3 DOC-EXAMPLE-BUCKET /mnt"

Inside the container, your bucket will be mounted at the `/mnt` path.

#### Arch Linux AUR (unofficial)

If you're using Arch Linux, you can use the unofficial [AUR](https://aur.archlinux.org/packages/mountpoint-s3-git) package. Using [Yay](https://github.com/Jguer/yay):

yay -S mountpoint-s3-git

## Contributing

We welcome contributions to Mountpoint for Amazon S3! Please see [CONTRIBUTING.md](doc/CONTRIBUTING.md) for more information on how to report bugs or submit pull requests. We especially welcome contributions to issues tagged as [good first issues to work on](https://github.com/awslabs/mountpoint-s3/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
Expand All @@ -132,6 +80,6 @@ This project has adopted the [Amazon Open Source Code of Conduct](https://aws.gi

This project is licensed under the Apache-2.0 License. It builds on a number of other awesome projects with open source licenses, primarily:
* The [fuser](https://github.com/cberner/fuser) bindings for FUSE (MIT license)
* The [AWS Common Runtime](https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html) for interacting with S3 (Apache-2.0 license)
* The [AWS Common Runtime](https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html) for interacting with S3 (Apache 2.0 license)

See [deny.toml](deny.toml) for a list of licenses used by our dependencies.
Loading