-
Notifications
You must be signed in to change notification settings - Fork 236
/
Copy pathREADME.md
181 lines (133 loc) · 7.79 KB
/
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
# S3Proxy
[](https://github.com/gaul/s3proxy/releases/)
[](https://hub.docker.com/r/andrewgaul/s3proxy/)
[](https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22s3proxy%22)
[](https://twitter.com/S3Proxy)
S3Proxy implements the
[S3 API](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services)
and *proxies* requests, enabling several use cases:
* translation from S3 to Backblaze B2, EMC Atmos, Google Cloud, Microsoft Azure, and OpenStack Swift
* testing without Amazon by using the local filesystem
* extension via middlewares
* embedding into Java applications
## Usage with Docker
[Docker Hub](https://hub.docker.com/r/andrewgaul/s3proxy/) hosts a Docker image
and has instructions on how to run it.
## Usage without Docker
Users can [download releases](https://github.com/gaul/s3proxy/releases)
from GitHub. Developers can build the project by running `mvn package` which
produces a binary at `target/s3proxy`. S3Proxy requires Java 11 or newer to
run.
Configure S3Proxy via a properties file. An example using the local
file system as the storage backend with anonymous access:
```
s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy
```
First create the filesystem basedir:
```
mkdir /tmp/s3proxy
```
Next run S3Proxy. Linux and Mac OS X users can run the executable jar:
```
chmod +x s3proxy
s3proxy --properties s3proxy.conf
```
Windows users must explicitly invoke java:
```
java -jar s3proxy --properties s3proxy.conf
```
Finally test by creating a bucket then listing all the buckets:
```
$ curl --request PUT http://localhost:8080/testbucket
$ curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>CustomersName@amazon.com</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>
```
## Usage with Java
Maven Central hosts S3Proxy artifacts and the wiki has
[instructions on Java use](https://github.com/gaul/s3proxy/wiki/Using-S3Proxy-in-Java-projects).
## Supported storage backends
* atmos
* aws-s3 (Amazon-only)
* azureblob
* azureblob-sdk (newer but lacks multi-part upload, see [Azure/azure-sdk-for-java#42603](https://github.com/Azure/azure-sdk-for-java/issues/42603))
* b2
* filesystem (on-disk storage, deprecated)
* filesystem-nio2 (on-disk storage, recommended)
* google-cloud-storage
* openstack-swift
* rackspace-cloudfiles-uk and rackspace-cloudfiles-us
* s3 (all implementations)
* transient (in-memory storage, deprecated)
* transient-nio2 (in-memory storage, recommended)
See the wiki for [examples of configurations](https://github.com/gaul/s3proxy/wiki/Storage-backend-examples).
## Assigning buckets to backends
S3Proxy can be configured to assign buckets to different backends with the same
credentials. The configuration in the properties file is as follows:
```
s3proxy.bucket-locator.1=bucket
s3proxy.bucket-locator.2=another-bucket
```
In addition to the explicit names, [glob syntax](https://docs.oracle.com/javase/tutorial/essential/io/fileOps.html#glob) can be used to configure many
buckets for a given backend.
A bucket (or a glob) cannot be assigned to multiple backends.
## Middlewares
S3Proxy can modify its behavior based on middlewares:
* [bucket aliasing](https://github.com/gaul/s3proxy/wiki/Middleware-alias-blobstore)
* [bucket locator](https://github.com/gaul/s3proxy/wiki/Middleware-bucket-locator)
* [eventual consistency modeling](https://github.com/gaul/s3proxy/wiki/Middleware---eventual-consistency)
* [large object mocking](https://github.com/gaul/s3proxy/wiki/Middleware-large-object-mocking)
* [read-only](https://github.com/gaul/s3proxy/wiki/Middleware-read-only)
* [regex rename blobs](https://github.com/gaul/s3proxy/wiki/Middleware-regex)
* [sharded backend containers](https://github.com/gaul/s3proxy/wiki/Middleware-sharded-backend)
* [storage class override](https://github.com/gaul/s3proxy/wiki/Middleware-storage-class-override)
* [user metadata replacer](https://github.com/gaul/s3proxy/wiki/Middleware-user-metadata-replacer)
## SSL Support
S3Proxy can listen on HTTPS by setting the `secure-endpoint` and [configuring a keystore](http://wiki.eclipse.org/Jetty/Howto/Configure_SSL#Generating_Keys_and_Certificates_with_JDK_keytool). You can read more about how configure S3Proxy for SSL Support in [the dedicated wiki page](https://github.com/gaul/s3proxy/wiki/SSL-support) with Docker, Kubernetes or simply Java.
## Limitations
S3Proxy has broad compatibility with the S3 API, however, it does not support:
* ACLs other than private and public-read
* BitTorrent hosting
* bucket logging
* bucket policies
* [CORS bucket operations](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html#how-do-i-enable-cors) like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration (see below).
* hosting static websites
* object server-side encryption
* object tagging
* object versioning, see [#74](https://github.com/gaul/s3proxy/issues/74)
* POST upload policies, see [#73](https://github.com/gaul/s3proxy/issues/73)
* requester pays buckets
* [select object content](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html)
S3Proxy emulates the following operations:
* copy multi-part objects, see [#76](https://github.com/gaul/s3proxy/issues/76)
S3Proxy has basic CORS preflight and actual request/response handling. It can be configured within the properties
file (and corresponding ENV variables for Docker):
```
s3proxy.cors-allow-origins=https://example\.com https://.+\.example\.com https://example\.cloud
s3proxy.cors-allow-methods=GET PUT
s3proxy.cors-allow-headers=Accept Content-Type
s3proxy.cors-allow-credential=true
```
CORS cannot be configured per bucket. `s3proxy.cors-allow-all=true` will accept any origin and header.
Actual CORS requests are supported for GET, PUT, POST, HEAD and DELETE methods.
The wiki collects
[compatibility notes](https://github.com/gaul/s3proxy/wiki/Storage-backend-compatibility)
for specific storage backends.
## Support
* [GitHub issues](https://github.com/gaul/s3proxy/issues)
* [Stack Overflow](https://stackoverflow.com/questions/tagged/s3proxy)
* [commercial support](mailto:andrew@gaul.org)
## References
* [Apache jclouds](https://jclouds.apache.org/) provides storage backend support for S3Proxy
* [Ceph s3-tests](https://github.com/ceph/s3-tests) help maintain and improve compatibility with the S3 API
* [fake-s3](https://github.com/jubos/fake-s3), [gofakes3](https://github.com/johannesboyne/gofakes3), [minio](https://github.com/minio/minio), [S3 ninja](https://github.com/scireum/s3ninja), and [s3rver](https://github.com/jamhall/s3rver) provide functionality similar to S3Proxy when using the filesystem backend
* [GlacierProxy](https://github.com/bouncestorage/glacier-proxy) and [SwiftProxy](https://github.com/bouncestorage/swiftproxy) provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
* [s3mock](https://github.com/adobe/S3Mock) - Adobe's s3 mock implementation
* [sbt-s3](https://github.com/localytics/sbt-s3) runs S3Proxy via the Scala Build Tool
* [swift3](https://github.com/openstack/swift3) provides an S3 middleware for OpenStack Swift
* [Zenko](https://www.zenko.io/) provide similar multi-cloud functionality
## License
Copyright (C) 2014-2024 Andrew Gaul
Licensed under the Apache License, Version 2.0