S3 Files Uploaded by Parse Server Inaccessible to api.parse.com clients

[benishak]

We migrated our database before the deadline, it is working fine,
We are hosting now ParseServer on our server and want to use it, we will release a new version of our iOS App that points to our ParseServer.
People using the new version of the App won't see any problem, but the user who still using the older version, won't be able to see the new data stored on our own S3.

Are you going to release a tool that can solve this problem ? something like configuration in the dashboard where we can configure a Storage Adapter?

Files Migration is in itself no problem, we can go through all files and copy them to our own S3, it is all about users who use our old version can see the objects uploaded to the new S3.

How fast do you think we are going to see this problem solved?

[flovilmart]

I'm not sure (and I may be wrong) about that problem. Your newly stored files should be publicly accessible and the URL's for the new files on your S3 will be stored in your DB alongside the old ones. So given that, as long as your files are properly uploaded to S3, an 'old' client would just get the new URL, pointing to your S3. Is it an issue you're seeing now?

[benishak]

The current situation now, the users who are using the app that point to parse.com are not able to see files stored in our DB or our S3 because the look like this

http://files.parsetfss.com/$filekey/$filename

and not like this

https://api.parse.com/files/$appid/$filename // if file is stored in the DB

or am I wrong?

[flovilmart]

that should not matter as the File objects store the full URL. Are your newly stored files accessible via your web browser?

[benishak]

no because it forward me to http://files.parsetfss.com

[flovilmart]

for the new files? How did you configure the S3 Adapter?

[benishak]

The old App uses : api.parse.com
The new App uses: api.ourparseServer.com

People using the "new App" can see files hosted on our S3 because we configured our S3Adapter and also the files hosted on Parse S3 ('ttfs')

But lets assume not all users upgrade the App so some users who use OLD APP, which use api.parse.com are not able to see or access object upload to our OUR S3

[hramos]

This is a known issue. api.parse.com has no knowledge of your specific Parse Server configuration, it only knows about files.parsetfss.com and files.parse.com. If Parse Server stores only the filename, api.parse.com will assume it is being hosted on files.parse.com and it will return a files.parse.com URL, which expectedly will fail to load the resource.

[flovilmart]

Don't we store the full URL if not, that's a big deal

[hramos]

My understanding is that only a filename is stored in the database, and it is up to the file adapter to look up the static asset or construct the direct URL when direct access is enabled.

See https://github.com/parse-server-modules/parse-server-s3-adapter/blob/master/index.js#L144 as reference.

[benishak]

Ok is Parse.com working on any solution or a tool on the dashboard that allow us to configure our own S3 or any kind of solutions? If yes how long will it take to see something?

[hramos]

We're heads down on database migration tasks at the moment. Improving the file migration story is something we have on our sights as we enter Phase 2 on April 28. No promises but this seems something that would be useful to look into.

See #8 and #1521 for related issues.

[flovilmart]

That Would be fairly Easy to make sure we store the full URL's. I believe the controller asks the adapter for the custom location, so it should in that case be stored as a full URL. I'll have a look.

[kvreem]

@benishak any luck?

[kvreem]

When using the default file grid adapter or amazon s3 older versions of the app cannot see the newly uploaded files. Users on new versions can see both. Did anyone figure a workaround or managed to get it working on grid adapter? Was surprised when the default adapter wasnt working either.

[MattiaConfalonieri]

i have the same problem too, old users cannot see new file upload with the default file grid adapter, that's a real big problem.

[kkgelu]

As @kkhattab @MattiaConfalonieri pointed out, this issue is holding us from going production with the new hosted Parse Server.

[benishak]

We pop up a dialog asking people to upgrade app and we will not support this version anymore and they will notice that

[MattiaConfalonieri]

@kkgelu same here, this have nonsense... the migration term suggested is 28th July... how we can do that?

@benishak not very elegant, but this is a solution... the only one at the moment...

[benishak]

Yeah unfortunately. ..

[kkgelu]

The MongoDB only keeps the file name, it has no knowledge about where the file system is.

Somehow the new server knows where the old files are, probably by prefix.

The problem is to let the Parse.com Server understand the new files. We have no way to upgrade api.parse.com but the Parse team.

[kvreem]

@kkgelu is it possible to just add a prefix automatically so that old clients can see the new files?

[kkgelu]

@kkhattab the clients, old or new, just go and fetch a file a from its url. The url, as I understood is constructed by the server (I have not looked into the implementation of ParseFile), it's the old server/api.parse.com couldn't understand the new files (the mongodb only keeps the names) and give its clients the wrong urls. (I'll make a concrete example next month when we get back to this issue again)

[JeremyPlease]

I've run into this issue while migrating a relatively large app.

The issue is that Parse Server only stores the filename and not the full url.

@flovilmart, @hramos If I submitted a PR to update Parse Server (and/or file adapters) to store the full urls, could api.parse.com be updated to handle it accordingly (ie. files without full url get url set to files.parse.com and files with full urls use those instead)?

[flovilmart]

I believe api.parse.com won't be updated now as it's winding down. We should aim for a solution that would work correctly, not expecting the original api would change to accommodate the update.

[JeremyPlease]

@flovilmart @hramos
If api.parse.com won't be updated then there is no possible way (that I see) for it to return the proper url for files uploaded to Parse Server.

The only possible solution I see is to write a file adapter to proxy file upload requests to api.parse.com/1/files. That way all files are stored in Parse's bucket and url will be set properly from both hosted and open source Parse.

Any other ideas or solutions you can think of?

[hramos]

I can't comment on whether api.parse.com will be updated to handle files uploaded by clients using Parse Server at this time. What I can say is that this issue only affects old clients that use api.parse.com, therefore the issue can be mitigated by updating the app to the newer version that uses Parse Server. We're almost 50% of the way through the sunset period, so I'd encourage everyone to focus on getting that updated version of the app out ASAP.

This issue seems to be a duplicate of #8. Can we close this issue and continue the thread there?

[JeremyPlease]

@hramos I definitely agree with getting updated versions of apps out ASAP. I'm just trying to mitigate the issue of users on old app versions (before they upgrade) unable to access files uploaded through Parse Server.

If there's no recommended solution for this, then I can accept it as a "migration side effect." We should document it somewhere with something like the following:

"Files uploaded through Parse Server will not be accessible by clients using api.parse.com."

After we document this, we can close this issue 😃

[hramos]

@JeremyPlease it is mentioned under Files in the Migration Guide:

"Files uploaded by Parse Server clients are not yet accessible by Parse.com clients as of this writing."

[JeremyPlease]

Thanks @hramos. Sorry I missed that in the migration guide!

I think it's fair to close this issue now and continue discussion of migrating legacy files on #8.

[acinader]

@hramos should we expect that there will be no solution to support parse.com and parse-server file access simultaneously?

At the risk of being obvious ;) it would be super nice not to break parse.com immediately when pushing the parse-server client.

"Files uploaded by Parse Server clients are not yet accessible by Parse.com clients as of this writing."

Since addressing this issue would require a change to parse.com, I'm asking here so we can plan accordingly. We'll have to deal either way, but would be interested to know if a decision has been made already?

The "as of this writing" is giving me hope, so better to dash those hopes now if necessary (in which case the readme should probably be updated too?)

[hramos]

@acinader I can confirm that there are currently no plans to update the legacy parse.com service.

[sud80]

@hramos Then essentially are we saying that, the migration guide that has been provided where we can support running older clients till end of January without breaking functionalities is gone for a toss. As you can see here in step 12 of migration guide https://www.parse.com/migration, the new clients and old clients work together for a period of time.
I don't understand why facebook is taking all its customers for granted. This is really sad state of affairs.

[hramos]

@sud80 the process of migrating an app is non-trivial and there will be some bumps in the road for some apps. The earlier you release an updated app that uses something other than api.parse.com, the earlier your user base will be able to transition and get past these bumps.

[sud80]

@hramos I do understand it is non-trivial but the point is to see if there is a way to avoid customer impact. Even however early I move, the way it is currently, the existing users with old app is going to get affected. But yeah we are left with no choice.