Guidance help

[craftzneko]

Hi

I have been trying to implement this project in Azure app Service docs.getoutline.com. It requires s3 storage. I have setup s3proxy as another container in Azure i am trying to set it up to function. It appears outline requires the following variables to be filled in or it will not attempt to send files to s3

AWS_REGION=us-east-2
AWS_S3_FORCE_PATH_STYLE=false
AWS_S3_UPLOAD_BUCKET_NAME=my-bucket-name
AWS_S3_UPLOAD_BUCKET_URL=https://my-bucket-name.s3.us-east-2.amazonaws.com
AWS_S3_UPLOAD_MAX_SIZE=26214400
AWS_ACCESS_KEY_ID=AK66L9HZpTtfrpFFgtVxcxOUTn
AWS_SECRET_ACCESS_KEY=97L9HfssZpTtfrxOUTVxcnpgtSa

I note that the bucket is part of the url, but in order to use s3proxy i am unsure what i need to change above for it to work. I have created an app registration in azure and have the secret as well as the storage account key but what is the bucket URL i need, also what would i put in AWS_S3_UPLOAD_BUCKET_NAME?

Any help would be very much appreciated, spent 3 weeks so far getting this setup and this is my last challenge now, hoping it wont all have been for waste

I have got much further now, i did not realise that it was javascript running in the browser that is sending the requests. I configured CORS and some other bits and i have got further now but i am getting a 501 response back

A header you provided implies functionality that is not implemented

@gaul i see there is another issue with the same error #73

I can provide the post headers and payload if it can help. It could be i have missed some settings as i see this in log

2023-08-02T21:56:35.165188289Z [s3proxy] E 08-02 21:56:35.155 S3Proxy-Jetty-14 o.gaul.s3proxy.S3ProxyHandler:769 |::] Unknown method POST with URI /mybucket

[alifirat]

Hi, to be sure are you trying to use s3proxy with an Azure backend ?

[craftzneko]

@alifirat hi, yes with an azure backend

[alifirat]

Please check https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Just set the authentication method to none and also the one about that ignores unknown headers.

[craftzneko]

@alifirat set as above, server still returns 501 not implemented,

If i navigate to endpoint with bucket in path https://itestproxy.azurewebsites.net/testBucketname i see this response

The logs from S3proxy state

Unknown method POST with URI /Testbucketname

[alifirat]

Did you try to add a folder after testBucket ? Also can you share your s3proxy configuration ?

[craftzneko]

Cheers @alifirat Forgive my ignorance S3 is new to me. so if i misunderstand please correct me. I created a folder in the backend called Uploads as that is what the client is trying to save to (along with random guid string folders). I can see this at the endpoint now so i know it can see my azure side

https://testproxy.azurewebsites.net/testbucketname

Config is created by environment variables here
JCLOUDS_AZUREBLOB_AUTH=azureKey
JCLOUDS_CREDENTIAL=REDACTED But IS my Azure KEY
JCLOUDS_ENDPOINT=https://MyStorageAccount.blob.core.windows.net/
JCLOUDS_IDENTITY=REDACTED but is storage account name
JCLOUDS_PROVIDER=azureblob
S3PROXY_AUTHORIZATION=none
S3PROXY_CORS_ALLOW_HEADERS=*
S3PROXY_CORS_ALLOW_METHODS=GET PUT POST HEAD
S3PROXY_CORS_ALLOW_ORIGINS=https://MySite.azurewebsites.net
S3PROXY_CREDENTIAL=TestCred
S3PROXY_IDENTITY=TestPass
S3PROXY_ENDPOINT=http://0.0.0.0:8080
S3PROXY_IGNORE_UNKNOWN_HEADERS=true

[craftzneko]

Looking at the code where the unsupported gets thrown, none of the headers in the request match the list but the form data does have policy in

In s3proxyhandler.java
UNSUPPORTED_PARAMETERS = ImmutableSet.of(
"accelerate",
"analytics",
"cors",
"inventory",
"lifecycle",
"list-type",
"logging",
"metrics",
"notification",
"policy",
"replication",
"requestPayment",
"restore",
"tagging",
"torrent",
"versioning",
"versions",
"website"}

my request headers

POST /mytestbucket HTTP/1.1
Accept: /
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9,en-US;q=0.8
Connection: keep-alive
Content-Length: 22860
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary9LQ2gN3Nv9OqJiQA
DNT: 1
Host: testproxy.azurewebsites.net
Origin: https://testwebsite.azurewebsites.net
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.200
sec-ch-ua: "Not/A)Brand";v="99", "Microsoft Edge";v="115", "Chromium";v="115"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"

Formdata payload

Cache-Control: max-age=31557600
Content-Type: image/png
Content-Disposition: attachment
key: uploads/7bbca2fd-69eb-48e5-baee-60afd34c4505/e5876117-7741-4d25-9350-dab2d1f305f6/2bc62b04-fb9b-4c91-ab78-402c02a3a3ef.png
acl: private
bucket: mytestbucket
X-Amz-Algorithm: AWS4-HMAC-SHA256
X-Amz-Credential: testcred/20230810/us-east-1/s3/aws4_request
X-Amz-Date: 20230810T082529Z
Policy: eyJleHBpcmF0aW9uIjoiMjAyMy0wOC0xMFQwOToyNToyOVoiLCJjb25kaXRpb25zIjpbWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsMCwxMDAwMDAwMDBdLFsic3RhcnRzLXdpdGgiLCIkQ24tgs9udGVudC1UeXBlIiwiaW1hZ2UvcG5nIl0sWyJzdGFydHMtd2l0aCIsIiRDYWNoZS1Db250cm9sIiwiIl0seyJDb250ZW50LURpc3Bvc2l0aW9uIjoiYXR0YWNobWVudCJ9LHsia2V5IjoidXBsb2Fkcy83YmJjYTJzYyYjA0LWZiOWItNGM5MS1hYjc4LTQwMmMwMmEzYTNlZi5wbmcifSx7ImFjbCI6InByaXZhdGUifSx7ImJ1Y2tldCI6InRlY2hub3Rlc21lZGlhIn0seyJYLUFtei1BbGdvcsdfsd0aG0iOiJBV1M0LUhNQUcdMtU0hBMjU2In0seyJYLUFtei1DcmVkZW50aWFsIjoiMTIzNDU2NzgvMjAyMzA4MTAvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsiWC1BbXotRsdsdcsdcsdcsdcsdvcMwODEwVDA4MjUyOVoifV19
X-Amz-Signature: 43373e09647af6399d896754c214bccbc9f014a56726491c9fd15622df1d711e
file: (binary)

I believe the client is using presigned URLS but not sure what policy is, is there any way of working around this @gaul

[craftzneko]

I am really desperate now to get this resovled. Its been 4 weeks of hell trying to get outline to work with s3 storage and i am no closer to a solution. As far as i can see above outline makes a valid S3 request but 501 is returned and i cant see how. If it is never gonna work with S3proxy then i just need to move on, anybody able to see what the issue is?

[alifirat]

I'm having a look tomorrow

[gaul]

Could you try testing with the new azureblob-sdk provider from #606?

[craftzneko]

I ended up using minio hosted in a vm but when I get chance I will try this again as if I can get it to work I won't need to host a vm in azure. I will close this for now as it may a while before I can try