-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added object cache for Swift ext storage #7897
Conversation
looks good but not tested 👍 |
* @param string $path | ||
* @return \OpenCloud\OpenStack\ObjectStorage\Resource\DataObject |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this return the fetched object or a bool?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the object exists, it returns it.
If the object does not exist, it returns false, which is also cached. The point being that we don't want to check X times whether a file doesn't exist when we already knew it doesn't. (check is_dir/filetype, that one is nasty)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add the bool in the doc
This improvement was excellent, greatly reduced the cURL requests and response time of the application. Good job! |
I've updated it. For some parts, objects need to be removed from the cache to force refetch/repopulate. Please review/test. |
The branch should be ready now (rebased/squased). I've done some testing and fixed the remaining issue with caching. |
Rebased. Please review @icewind1991 @schiesbn |
Changes look good 👍 |
testing .... |
@PVince81 testStat failed:
|
debugging .... |
the touch operation succeeds, but when retrieving the mtime the value is not as expected |
Ok, I'll look into this. |
I'll also check master, because the touch operation was fixed in a separate PR. |
@DeepDiver1975 I ran the unit tests and I'm getting something different on this branch:
Not sure if it's due to connection speed being different from yours. |
@DeepDiver1975 I have the feeling that OpenStack doesn't automatically update the mtime of its root. Also even stranger it always returns the same timestamp, even when I delete the bucket and re-create it. It seems to be the bucket's creation time, seen this with a newly created bucket. Do you get the same results on master ? If yes I suggest to merge this first and I'll have a look at that issue separately (with potential backport). Maybe we need to "touch" the root as well (this adds our custom mtime attribute) to manually propagate the change. |
I also realized that mtime won't propagate to parent folders either since OpenStack is a flat object hierarchy. To fix this we'd need to touch (update the mtime) every parent folder up to the root... |
@PVince81 I'll test this and master tomorrow - I've been asking myself if we should put each external fs into it's own repo and prepare travis to execute the tests on - as far as possible - locally installer services (e.g. Ceph as S3 backend etc) |
Thanks. I'd like to get this merged first, then possibly look into making the mtime propagation and update detection work. What do you mean with own repo ? Having each external storage as separate GitHub repo app ? |
Yes - divide and conquer! |
That would be quite a lot of backends. Might make sense for some of them, or at least the "exotic" ones like object store. The FTP, SFTP ones could stay here. Or group them somehow. Changes will be needed in the app to make it possible for external apps to register their backends, which makes sense as this is currently not supported and makes it impossible for 3rd party to provide their own ext storages. |
@@ -53,6 +53,12 @@ class Swift extends \OC\Files\Storage\Common { | |||
private static $tmpFiles = array(); | |||
|
|||
/** | |||
* Object cache, map of path to object |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This description does not correctly consider https://github.com/owncloud/core/pull/7897/files#diff-2ab592dfd95e06115a358d1b5b20cdc5R86
It seems that Flysystem performs no caching at all... but if we were to have a single storage API provided by Flysystem, implementing generic caching would be a possibility |
|
@DeepDiver1975 Or that 🙈 |
101312b
to
ecba66e
Compare
Rebased. Has failing tests:
|
ecba66e
to
39046a8
Compare
Rebased and fixed unit tests. Please review @owncloud/filesystem |
also @tjdett @MorrisJobke in case you are interested |
👍 |
This avoid calling the remote API for repeated calls to is_dir, filetype, etc
39046a8
to
f77a11a
Compare
Rebased due to the PSR-4 changes. (no conflicts, automatically resolved) |
Ran the Swift tests locally and they passed, merging. |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
This avoid calling the remote API for repeated calls to is_dir,
filetype, etc
I haven't fully tested this yet, but it does seem to be a bit faster.
I'm starting to think we might want to provide a generic cache for remote storages, because the Amazon implementation would also need it. The Dropbox one also has something similar.
@icewind1991