Add support for saving files to AWS S3

[skinp]

Using Mongo's GridStore for file storage is fine for basic usage, but it won't scale much. Heavy production users should use a proper file/object storage system. Let's at least officially add support for S3 storage for Parse files.

To use this:

var api = new ParseServer({
  ...
  filesAdapter: new S3Adapter(
    "<AWS_ACCESS_KEY>",
    "<AWS_SECRET_ACCESS_KEY>",
    {bucket: "<BUCKET_NAME>", bucketPrefix: "files/", directAccess: true}
  ),
  ...
});

Once this is accepted, I'll add docs to the wiki on how to use this properly and explain all the available options.

[skinp]

I'm curious if this is the best/most idiomatic way of doing this. I couldn't reference this object within the Promise function.

[skinp]

Also, the AWS node library can load it's config from a JS object. I'm debating whether the S3Adapter should just take such an object as parameter or something. That would potentially allow more configuration of the S3 object, although would probably make it less errr simple. Again, what's more idiomatic?

[gfosco]

You should be able to access this, as ExportAdapter uses it heavily...

[skinp]

Ok discussed offline, apparently ES6 arrow function (=>) is the way to go

[lucianmat]

👍

[skinp]

hmmm I clearly messed up something in that rebase. Let me try to fix this

[skinp]

Ok, I fixed it. Should be good to merge when accepted.

[edvincandon]

@skinp looking forward to your wiki/docs on your AWS implementation, can't seem to get it working..

[skinp]

@edvincandon the S3Adapter object is not exposed directly just yet. If you're trying to use my example from earlier, you need one more require line to get access to S3Adapter object:

var S3Adapter = require('parse-server/S3Adapter');

Docs to come, and this object should be exposed in some next versions.

[nitrag]

@skinp This is great! Looking forward to a guide!

[edvincandon]

Hey @skinp - Stil trying to figure out what's wrong,
FilesAdapter is catching an error and throwing me this on the client side.
{"code":130,"error":"Could not store file."}

Server-side logs empty for this issue.

I'm fairly new to S3 and the AWS SDK, but I saw they had their own error handling codes and messages - would it be possible to pass those to Parse.Error instead of just throwing this in files.js :

...
}).catch((error) => {
next(new Parse.Error(Parse.Error.FILE_SAVE_ERROR,
'Could not store file.'));
});

Would be a nice implementation for debugging purposes, no?
Thanks for your time!

EDIT - Error was on my side with my bucket configuration.

[skinp]

@edvincandon glad you got it figured out. Yeah I was printing that error when developing this module but the errors coming from the AWS SDK ended up being too verbose. Maybe we could print them to the server log instead. I'm going to first write docs for this. Let's see if having clear docs helps reduce the amount of errors/confusion people get with this Adapter and we'll see after if we need better errors/logging.

[leobortolotti]

@edvincandon I'm having the same error here
[Error]: Could not store file. (Code: 130, Version: 1.12.0)
Could you tell us what was the problem you were having in the bucket configuration?
Thanks

[edvincandon]

@leobortolotti
I had to edit the bucket policies on S3 for it to work. Use the bucket policy generator on AWS (found under Properties > Permissions once you select your target bucket on AWS or http://awspolicygen.s3.amazonaws.com/policygen.html )

Next select S3 Bucket Policy as your desired type of policy
Select Bucket Policy
Enter * in the Principal field,
SelectGetObject as your desired Action
Enter your Amazon Resource Name -> Your resource name for a S3 bucket : arn:aws:s3:::<YOUR_BUCKET_NAME>/*

Generate the policy and add it to your bucket!

Hope it helps! Also make sure your bucket is CORS enabled, your CORS configuration should look something like this:

<CORSConfiguration>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
</CORSConfiguration>

[leobortolotti]

@edvincandon Thank you very much!!!
I did everything you said, but still getting the error, seems that this problem was one but not the only that I have to solve!! hahaha

[edvincandon]

Are you using the latest parse-server release? If not, try updating..
Also, you're probably inserting that Parse.file in some object, try deleting the _schema for that particular object (worked for me..)

[leobortolotti]

I did this but I'm still getting this error! But thanks again!
I think that I will wait for the Docs Guide...

[leobortolotti]

Ohhhh, I got it!!! Seems like my Access Key and Secret Key was not working/inactive, I don't know why!!
I generated new ones and now it is working!! Thanks man!!!

[davidruisinger]

As far as I understand the code this does not support deleting the file via REST API, right?
Any plans on implementing the delete functionality as well?

In my setup I'm deleting files using Parse Cloud:

  Parse.Cloud.httpRequest({
    method: 'DELETE',
    url: imageUrl,
    headers: {
      "X-Parse-Application-Id": "******"
    },
    success: function(httpResponse) {
      console.log('Delete succeeded  ' + httpResponse.text);
    },
    error: function(httpResponse) {
      console.log('error');
    }
  });

which returns an Access Denied error.

[gfosco]

@flavordaaave Delete will be implemented in #354 and will require the master key.

[facebook-github-bot]

@skinp updated the pull request.

[davidruisinger]

@gfosco still getting an Access Denied message with the latest update (2.0.8):

  Parse.Cloud.httpRequest({
    method: 'DELETE',
    url: imageUrl,
    headers: {
      "X-Parse-Application-Id": "******",
      "X-Parse-Master-Key": "******"
    },
    success: function(httpResponse) {
      console.log('Delete succeeded  ' + httpResponse.text);
    },
    error: function(httpResponse) {
      console.log('error');
    }
  });

[facebook-github-bot]

@skinp updated the pull request.