Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add uploads_path param #38587

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

tanganellilore
Copy link
Contributor

@tanganellilore tanganellilore commented Jun 1, 2023

Summary

Alternative solution instead usage of symlink proposed here: #38506.

In this solution I'm not use the symlink but directly new virtual mount to configuration passed.

TODO

  • ...

Checklist

apps/dav/lib/Upload/UploadHome.php Fixed Show fixed Hide fixed
apps/dav/lib/Upload/UploadHome.php Fixed Show fixed Hide fixed
apps/dav/lib/Upload/UploadHome.php Fixed Show fixed Hide fixed
@szaimen szaimen added the 3. to review Waiting for reviews label Jun 1, 2023
@szaimen szaimen added this to the Nextcloud 28 milestone Jun 1, 2023
@szaimen szaimen requested review from blizzz, come-nc, a team and icewind1991 and removed request for a team June 1, 2023 23:05
@tanganellilore tanganellilore force-pushed the add_uploads_path branch 3 times, most recently from daf4334 to 20159b2 Compare June 2, 2023 09:54
@tanganellilore
Copy link
Contributor Author

Recommit with correct approach suggested in last comment of #38506 from @icewind1991.

Path now it's splitted from cache_path to cache_path and uploads_path . In that way we can choice different paths.

@tanganellilore tanganellilore mentioned this pull request Jun 2, 2023
5 tasks
@kesselb
Copy link
Contributor

kesselb commented Jun 2, 2023

Recommit with correct approach suggested in last comment of #38506 from @icewind1991.

I don't think we suggested introducing another configuration option. The feature you are looking for is already there.

Would you mind explaining why you need to separate uploads and cache? Did you run into problems?

@tanganellilore
Copy link
Contributor Author

My only concern is to have a clear segmentation of folder/mountpoint/disk associated.
For example for uploads we need to be sure that we have space on it for chunks uploads (ad we can stimate it), for cache folder, for my knowledge, if not exists or no space left shouldn't be a problem.

Moreover we can decide to use a different mount point in different path, associated to different disk (and size).

But this is an idea/features, and I notice only after my PR and reading code that cache_path is used for cache path and uploads path (and this is not document ed well).

@kesselb
Copy link
Contributor

kesselb commented Jun 6, 2023

Does it work?

If cache_path === uploads_path the expected /uploads/ subfolder is created in the cache_path block. But if you define a different uploads_path (not inside the cache_path) who is going to create the uploads subfolder for the user?

@kesselb
Copy link
Contributor

kesselb commented Jun 6, 2023

	/**
	 * Get the cache mount for a user
	 *
	 * @param IUser $user
	 * @param IStorageFactory $loader
	 * @return \OCP\Files\Mount\IMountPoint[]
	 */
	public function getMountsForUser(IUser $user, IStorageFactory $loader) {
		$cacheBaseDir = $this->config->getSystemValueString('cache_path', '');
		$uploadsBaseDir = $this->config->getSystemValueString('uploads_path', '');

		if ($cacheBaseDir === '' && $uploadsBaseDir === '') {
			return [];
		}

		$cacheDir = rtrim($cacheBaseDir, '/') . '/' . $user->getUID();
		if (!file_exists($cacheDir)) {
			mkdir($cacheDir, 0770, true);
		}

		if ($uploadsBaseDir === '') {
			$uploadsDir = $cacheDir . '/uploads';
		} else {
			$uploadsDir = rtrim($uploadsBaseDir, '/') . '/' . $user->getUID();
		}

		if (!file_exists($uploadsDir)) {
			mkdir($uploadsDir, 0770, true);
		}

		return [
			new MountPoint('\OC\Files\Storage\Local', '/' . $user->getUID() . '/cache', ['datadir' => $cacheDir], $loader, null, null, self::class),
			new MountPoint('\OC\Files\Storage\Local', '/' . $user->getUID() . '/uploads', ['datadir' => $uploadsDir], $loader, null, null, self::class)
		];
	}

I guess this version should cover the old case and new case.
Yet, we should first merge #38633 and then add a test for uploads_path in this pull requests.

@tanganellilore
Copy link
Contributor Author

tanganellilore commented Jun 7, 2023

Thanks @kesselb for highlight it. You are correct, I missing the removal of creation uploads user path in the cache if, and creation of it on upload if.

Everything working in my previous commit, because uplaod function check and create uploads folder if not exists.

Now should be fixed, if uploads_path is defined we use it (and create relative user path), if not we use the value of cache_path (and create user uploads folder), if also cache_path is empty, we skip it.

@tanganellilore tanganellilore force-pushed the add_uploads_path branch 2 times, most recently from bd243b3 to 60ceaca Compare June 7, 2023 06:34
@tanganellilore
Copy link
Contributor Author

@kesselb Any news on this PR?

@come-nc come-nc requested a review from kesselb August 28, 2023 13:09
@skjnldsv skjnldsv mentioned this pull request Nov 1, 2023
This was referenced Nov 6, 2023
@skjnldsv skjnldsv mentioned this pull request Mar 28, 2024
81 tasks
@skjnldsv skjnldsv modified the milestones: Nextcloud 29, Nextcloud 30 Mar 28, 2024
Signed-off-by: Lorenzo Tanganelli <lorenzo.tanganelli@hotmail.it>
@kesselb kesselb added 2. developing Work in progress and removed 3. to review Waiting for reviews labels Jun 18, 2024
@kesselb
Copy link
Contributor

kesselb commented Jun 18, 2024

Hi @tanganellilore,

Sorry, I apparently missed your ping last year.

#38633 was merged, and therefore we have unit tests now for the component.

Please ping me again if the linter and nodb tests are happy, I will trigger another review then. Note some tests (e.g., cypress) are failing for forks, don't worry.

Signed-off-by: Lorenzo Tanganelli <lorenzo.tanganelli@hotmail.it>
This was referenced Jul 30, 2024
This was referenced Aug 5, 2024
@skjnldsv skjnldsv mentioned this pull request Aug 13, 2024
@skjnldsv skjnldsv closed this Aug 14, 2024
@skjnldsv skjnldsv removed this from the Nextcloud 30 milestone Aug 14, 2024
@tanganellilore

This comment was marked as resolved.

@skjnldsv

This comment was marked as resolved.

@skjnldsv
Copy link
Member

@susnux @juliushaertl can I get your opinion on this?
That seems interesting, but there might be some underlying issues. Do you have any insights?

/**
* Override where Nextcloud stores uploaded user files while uploading (chunks). Useful in situations
* where the default `<user_id>/uploads` is on network disk like NFS.
* Defaults to the value of '' if unset.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Defaults to the value of '' if unset.
* Defaults to `cache_path` if set, otherwise ''

@juliushaertl
Copy link
Member

I think this is reasonable, though I currently lack insight on what we actually use the cache folder for. Tend to agree with @kesselb that two separate config values seem to much.

Needs a rebase and resolving the conflict.

@tanganellilore
Copy link
Contributor Author

Cache folder is used for chunk upload if i remeber well.
It's possible that in some configuration we wanto have cache folder with ssd storage (to be very fast on chunk assembling).

See issue linkedin to this MR.

I think that i can do rebase, but before tell me if this will be merged

@kesselb
Copy link
Contributor

kesselb commented Aug 14, 2024

image

@skjnldsv
Copy link
Member

I think that i can do rebase, but before tell me if this will be merged

Would be nice yeah! We're about to start the 31 release schedule. Maybe have this merged soon to be safe?

I would also like some tests for this
If possible 😬

@nextcloud-command nextcloud-command added the stale Ticket or PR with no recent activity label Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2. developing Work in progress feature: filesystem needs info performance 🚀 stale Ticket or PR with no recent activity
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support to custom user uploads dir
8 participants