Status of legacy files on Parse S3

[hashmapped]

What will happen to file migration? I can't really find instructions on how to transfer/migrate over files to your own bucket.

[gfosco]

We'll be working on a plan for this. Right now when you migrate data, the files stay on Parse's S3 bucket, but parse-server knows how to continue serving them..

[shahzeb1]

It would be so nice to have a config setting with our own S3 settings which end up getting used for file uploads.

[hashmapped]

So we are assured that file migration will be happening? So that we can continue develop using PFFile before migration?

[franksierra]

@hashmapped Yes

[chrisfoulds]

Without PFFile migration my apps are dead in the water, please prioritise this as it would allow me to migrate alot sooner

[chrisfoulds]

So if I do a database migration right now, when a user creates a new PFFile - will the MongoDB entry just point to a new file hosted by Parse or will something else happen ?

[gfosco]

New files uploaded to parse-server are hosted in GridStore(Mongo), only files uploaded through api.parse.com are still hosted by Parse. We are definitely going to solve this, and an awesome engineer here is working on an great update.

[pungme]

+1 we really need this. Do we at least have a timeframe for this? like when it likely will be available?

[Abderezai]

Also would be good if the client SDK can directly upload to S3 without hoping on first to parse server, abstracted in the client so we can still use the parse file api.

[fatuhoku]

I would love to see a script that helps the user (idempotently) download every file from Parse's S3 http://files.parsetfss.com/ endpoint and from GridFS, and uploads them to a third location of the user's choice (authentication might be an issue here of course — parse-server will need a way to handle this): whether this be an S3 bucket, or Dropbox, or Cloudinary, or whatever.

The script should also update the URL held in the relevant Mongo file field for each class.

It's scary to think that after a year, all files held on Parse will disappear unless action is taken!

[chrisfoulds]

@fatuhoku that is very easy, I have already written one to copy all my files off Parse to an alternative server for backup purposes. Just use the javascript or php api's and you will get it done in minutes.

The real issue is that until I move to my own server files uploaded to api.parse.com are hosted by Parse.
What is needed is two fold.

1. When I tell api.parse.com that I want to move to MongoDB - to move all my existing files over as well as the database and store them with MongoDB Gridstore.
2. When (1) above is complete and I have moved to my own MongoDB for api.parse.com to store all new files to MongoDB gridstore and STOP hosting the files yourself.

This way I can follow Parse's guide of step 1) point at own database , 2) build own server, 3) issue app update.

Currently with files this is not possible.

An alternative (or additional) process could be telling both api.parse.com and the new parse server to host files on a S3 instance instead of MongoDB gridstore.
Either way this is a glaring omission that needs resolving asap.

[fatuhoku]

@chrisfoulds do you have that script to hand by any chance? I'd be grateful if you stuck it on a gist so that we all have something to work off. Re. new files hitting api.parse.com being stored there, I imagine you could write a Cloud Code function (for every class that has files...!) that saves them to GridDB explicitly.

[gateway]

Im not sure if its finished but in the FilesAdaptor.js you should abe able to set this to S3 soon. https://github.com/ParsePlatform/parse-server/blob/master/FilesAdapter.js see this bit of code for the s3 implementation https://github.com/ParsePlatform/parse-server/blob/master/S3Adapter.js .. I wish someone would document how to use this over gridstore. I personally haven't had time to test it out.

[barnaclejive]

@gateway The S3Adapter seems to be working. I have not seen official documentation but here is a sample of my index.js. Note the additional require at the top to bring in the S3Adapter. It is noted in #113 that this may eventually be exposed by default so the require() won't be needed.

What happens when you use this adapter is that any new files added will go to your S3 bucket instead of GridFS in your MongoDB database.

var express = require('express');
var ParseServer = require('parse-server').ParseServer;
var S3Adapter = require('parse-server/S3Adapter');
...
var api = new ParseServer({
  databaseURI: databaseUri || 'mongodb://.....',
  cloud: process.env.CLOUD_CODE_MAIN || __dirname + '/cloud/main.js',
  appId: 'MY_APP_ID',
  clientKey: 'MY_CLIENT_KEY', 
  masterKey: 'MY_MASTER_KEY',
  filesAdapter: new S3Adapter(
    "KEY",
    "SECRET_KEY",
    {bucket: "my-bucket-name", bucketPrefix: "some-folder/", directAccess: true}
  )
});

[barnaclejive]

@gfosco When can we expect an update on migrating existing files to a self-hosted S3 bucket? I have tested migration (and am using my fileKey) but all existing files result in a 404. The PFFile.url ends up pointing to a self hosted file instead of the parse hosted file.
Example URL:
http://foo.herokuapp.com/parse/files/dWOKPEJ9ihbPYARGV7lwBUz3vQOASuhfWrNl7PCtxrau/78a0a3aa-2ef3-46c8-be1b-14059308cc4a-my_image.jpg
I should note I am using the S3Adapter so that new files get stored to a self-hosted S3 bucket instead of GridFS. Is this resulting in the parse-server trying to also Retrieve files from my S3 instead of the existing parse hosted file?

The file migration solution will need to account for both existing files and new files. Existing files will ideally be able to be migrated to the same S3 bucket/folder as new files. If they cannot be migrated to the same self-hosted S3 bucket/folder as new files, the parse-server should work seamlessly for those of us opting to use the S3Adapter fileAdapter by delegating to some other S3 bucket for existing files and to the self-hosted S3 bucket for new files.

Thanks for helping us. So far things are working pretty smooth in general, but the PFFile question is a roadblock to finalizing migration.

[barnaclejive]

@fatuhoku I am sure we'll get an awesome solution for migrating files from the Parse team.
In the meantime (and for paranoid backup) I whipped a quick Python script to manually download files. To use it you'll need to export your collections to json using the "Export app data" option in your apps App Settings.
Then you'll need to run the script for each file that contains pointers to files. It is a quick and dirty script that met my needs but it covers the basic requirements of a brute force file downloader. You'll very likely need to make some modifications to it to process your exported json files.
https://gist.github.com/barnaclejive/bac711c85e5e9a94a8b1

[jaydeep82]

@drew-gross @nlutsenko
Thank you for sharing this.

Do we have plan to add adapter for other file storage services like Dropbox or Azure file storage ?

[nlutsenko]

Immediate plans - no, but since it's fairly easy to create your own file adapter we would love to see the community build the ones that they think are the most helpful and we'll happily merge them.

[natanrolnik]

@poojanjhaveri
you have the following options available: https://mv.sashido.io or this command line tool

[JeremyPlease]

The parse-server-modules/parse-files-utils tool suggests two strategies for migrating parse files.

README has been recently updated to describe the two options:
https://github.com/parse-server-modules/parse-files-utils#parse-file-migrations

[poojanjhaveri]

@JeremyPlease Thanks. @natanrolnik @vladiGtr Have you guys used mv.sashido ?

[RafaRuiz]

Finally I didn't want to use any alternative since they didn't seem what I was looking for.

I'm showing my code here just in case you need it. It's working on me (migrating really slow, but it's going on!)

Good luck!

[iamvajid]

Is someone already working on this ? I would like to work on if not started already

[gateway]

@RafaRuiz I think your code is broken with the last parse php sdk..

$parseFile->getName();

2016/09/13 18:56:27 [error] 25386#0: *329857 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Call to a member function getName() on a non-object in /var/www/aaa.com/tests/data/migrate.php on line 34" while reading response header from upstream, client: , server: aaa.com, request: "GET /tests/data/migrate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "aaa.com"

Also I dont think this takes into account if you have more than one image in the row.. like a profile pic, small, med, large etc.. I might be able to fix it just letting you know.

[iamvajid]

@gfosco Any updates on this ? Things are moving to danger zone if this is further delayed. Is there any active development on this ?

[flovilmart]

Look at https://github.com/parse-server-modules/parse-files-utils

[woodardj]

@flovilmart so is parse-files-utils the official answer or not? This ticket appears to be closed on the basis of that utility existing, but it comes with a giant warning claiming that it's "unofficial" and might break things.

[flovilmart]

Yeah, it's as good as it gets, not sure if anything else will be released.

[FeleciaGenet]

Thanks for all of these comments. They have helped me thus far. However I have added the File Key to my index.js as mentioned above :

var api = new ParseServer({
databaseURI: databaseUri || 'mongodb://xxxxxxx',
cloud: process.env.CLOUD_CODE_MAIN || __dirname + '/cloud/main.js',
appId: process.env.APP_ID || 'myAppID',
masterKey: process.env.MASTER_KEY || 'MyMasterKey', //Add your master key here. Keep it secret!
serverURL: process.env.SERVER_URL || 'http://myServerURL', // Don't forget to change to https if needed
fileKey: process.env.PARSE_FILE_KEY || 'myFileKey',
liveQuery: {
classNames: ["Posts", "Comments"] // List of classes to support for query subscriptions
}
});
// Client-keys like the javascript key or the .NET key are not necessary with parse-server
// If you wish you require them, you can set them as options in the initialization above:
// javascriptKey, restAPIKey, dotNetKey, clientKey

and I am still not able to access my imageFiles that were stored to Parse prior to migration. When I run my app, I get the following error :

[Error]: Response status code was unacceptable: 404

Has anyone else ran into this? What have I missed? Thanks in advance.

[vinaykaitha]

Can someone please answer this:
http://stackoverflow.com/questions/41737158/migrating-parse-files-from-gridstore-to-s3

Description:
We migrated the data(without files) to mLab and Heroku. So the old files are still on Parse.
Since then, any new file added goes into Gridstore, which is the default file storage for mLab.
Now I migrated old parse files from Parse to an S3 Bucket using sashido
The files are migrated and are accessible using S3Adapter in Heroku.
But the files on Gridstore are not accessible now. How can I migrate them to the same S3 bucket and change references in mLab?

[pivanov]

Open a ticket in SashiDo an mention Pavel :)

[RafaRuiz]

@vinaykaitha I just answered it

[hspindell]

This might be useful for some of you:
https://github.com/modernistik/parse-file-migration

[MarsVard]

aaaaaannddd it's gone.

[mortizbey]

Hello everyone! We have just realized that lots of our files did not migrate completely to our servers and, as expected, we cannot access files on http://files.parsetfss.com/. This is core to our business and we need to get those files back even if we have to pay. Please, please, please, is there a way we can access them? We'll do anything that is necessary.

[mkll]

@ortimanu
Today I received a message from back4app. Maybe it will help you.

[MarsVard]

How come they can still access those files?

[mkll]

I have no idea

[davimacedo]

Hi, guys. We are doing like this:
parse-server-modules/parse-files-utils#67

It is already merged to the master so you can still use parse-file-utils.

Parse.com announced it will stop anytime though.

[MarsVard]

I migrated using sashido, just the files with mv.sashido.io.
Everything went fine and I can access the files in s3. but I can't access the files using parse dashboard, or my apps. what am I missing?

[pivanov]

@ortimanu ping me on support@sashido.io

[iamvajid]

It might be permission issue sometimes.

…

On 31 January 2017 at 17:56, Manuel Ortiz ***@***.***> wrote: Hello everyone! We have just realized that lots of our files did not migrate completely to our servers and, as expected, we cannot access files on http://files.parsetfss.com/. This is core to our business and we need to get those files back even if we have to pay. Please, please, please, is there a way we can access them? We'll do anything that is necessary. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADRXACci3cXsU_RcEQ2s5NKKG5eZht_qks5rXyhvgaJpZM4HO2Bk> .

-- alvajindia..... "nthng spzl..nthng mattrzz..."

[iamvajid]

@mars If you are seeing access denied message, its because files were uploaded with limited permission. You can make the files public and then it will be available

…

On 1 February 2017 at 00:14, Mars ***@***.***> wrote: I migrated using sashido, just the files with mv.sashido.io. Everything went fine and I can access the files in s3. but I can't access the files using parse dashboard, or my apps. what am I missing? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADRXAI6dy2aNUzcFMLPh0lad4rwLu6ocks5rX4DygaJpZM4HO2Bk> .

-- alvajindia..... "nthng spzl..nthng mattrzz..."

[MarsVard]

@alvajindia no, its giving me a "file not found" the urls have been updated with the mv-prefix do I need to set up a file adapter to make this work ?

[arturogutierrez]

Did you set the Parse Public URL in your Parse-Server? I forgot to add it and all my file URLs started with localhost instead of the public URL. Can you check the URL that is giving your Parse-Server when accessing to files? El mié., 1 feb. 2017 7:45, Mars <notifications@github.com> escribió:

…

@alvajindia <https://github.com/alvajindia> no, its giving me a "file not found" the urls have been updated with the mv-prefix do I need to set up a file adapter to make this work ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAsPn5473tJ-lTrjhB1xLY0-wEt_Vgd2ks5rYCoXgaJpZM4HO2Bk> .

[MarsVard]

@arturogutierrez the file urls are pointing to the right domain as far as I can see in the parse-dashboard

a migrated file looks like this
https://[mydomain]/apps/[myapp]/files/[filekey]/mv-b0acb1ae-d6f7-44c9-921d-0f332366692a-file
newly uploaded and working files look like this, these were uploaded to my local parse
https://[mydomain]/apps/[myapp]/files/[filekey]/7213555b291d0c4ca7f99942c460a9ee_file

Could the file adapter cause this?

[mortizbey]

Guys, thanks to parse-server-modules/parse-files-utils#67 I could download all my files and transfer them to my S3 bucket. Thanks guys for everything!