Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add instructions for seaweedfs and s3 connectivity #8678

Merged
merged 4 commits into from
May 9, 2022
Merged

add instructions for seaweedfs and s3 connectivity #8678

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
205b9f5
add instructions for seaweedfs and s3 connectivity
carlsonp May 6, 2022
40c73e9
spelling
carlsonp May 6, 2022
b87eed6
format SeaweedFS docs, small tweaks #8678
pdurbin May 6, 2022
9cbd30f
Merge pull request #1 from pdurbin/seaweedfs
carlsonp May 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 43 additions & 2 deletions doc/sphinx-guides/source/installation/config.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -613,9 +613,50 @@ For direct uploads and downloads, Dataverse redirects to the proxy-url but presi
Additional configuration (appropriate CORS settings, proxy caching/timeout configuration, and proxy settings to pass headers to/from S3 and to avoid adding additional headers) will also be needed to enable use of a proxy with direct upload and download.
For Amazon AWS, see comments in the edu.harvard.iq.dataverse.dataaccess.S3AccessIO class about support for AWS's bucket-specific DNS names.

**HINT:** If you are successfully using an S3 storage implementation not yet listed above, please feel free to
`SeaweedFS <https://github.com/chrislusf/seaweedfs>`_
SeaweedFS is a distributed storage system that has S3 compatibility. Set the S3 storage options as explained above. Make sure to set ``dataverse.files.<id>.path-style-access`` to ``true``. You will need to create the bucket beforehand. You can do this with the filer API using curl commands. For example, to create an empty bucket called ``dataverse``:

.. code-block:: bash

curl -X POST "http://localhost:8888/buckets/"
curl -X POST "http://localhost:8888/buckets/dataverse/"

You will also need to set an access and secret key. One way to do this is via a `static file <https://github.com/chrislusf/seaweedfs/wiki/Amazon-S3-API#static-configuration>`_. As an example, your ``config.json`` might look like this if you're using a bucket called ``dataverse``:

.. code-block:: json

{
"identities": [
{
"name": "anonymous",
"credentials": [
{
"accessKey": "secret",
"secretKey": "secret"
}
],
"actions": [
"Read:dataverse",
"List:dataverse",
"Tagging:dataverse",
"Write:dataverse"
]
}
]
}

And lastly, to start up the SeaweedFS server and various components you could use a command like this:

.. code-block:: bash

weed server -s3 -metricsPort=9327 -dir=/data -s3.config=/config.json

**Additional Reported Working S3-Compatible Storage**

If you are successfully using an S3 storage implementation not yet listed above, please feel free to
`open an issue at Github <https://github.com/IQSS/dataverse/issues/new>`_ and describe your setup.
We will be glad to add it here.
We will be glad to add it.


Migrating from Local Storage to S3
##################################
Expand Down