Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Use file cache query builder for searching for favorites #39303

Closed
wants to merge 2 commits into from
Closed

feat: Use file cache query builder for searching for favorites #39303

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b230e54
feat: Use file cache query builder for searching for favorites
juliusknorr Jul 11, 2023
dfd21de
feat: Pass limit/offset for searchByTag
juliusknorr Jul 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 47 additions & 32 deletions apps/dav/lib/Connector/Sabre/FilesReportPlugin.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
use OCP\Files\Node as INode;
use OCP\IGroupManager;
use OCP\ITagManager;
use OCP\ITags;
use OCP\IUserSession;
use OCP\SystemTag\ISystemTagManager;
use OCP\SystemTag\ISystemTagObjectMapper;
Expand All @@ -49,6 +50,10 @@
// namespace
public const NS_OWNCLOUD = 'http://owncloud.org/ns';
public const NS_NEXTCLOUD = 'http://nextcloud.org/ns';

public const NS_OWNCLOUD_PREFIX = '{' . self::NS_OWNCLOUD. '}';
public const NS_NEXTCLOUD_PREFIX = '{' . self::NS_NEXTCLOUD. '}';

public const REPORT_NAME = '{http://owncloud.org/ns}filter-files';
public const SYSTEMTAG_PROPERTYNAME = '{http://owncloud.org/ns}systemtag';
public const CIRCLE_PROPERTYNAME = '{http://owncloud.org/ns}circle';
Expand Down Expand Up @@ -186,15 +191,13 @@
return;
}

$ns = '{' . $this::NS_OWNCLOUD . '}';
$ncns = '{' . $this::NS_NEXTCLOUD . '}';
$requestedProps = [];
$filterRules = [];

// parse report properties and gather filter info
foreach ($report as $reportProps) {
$name = $reportProps['name'];
if ($name === $ns . 'filter-rules') {
if ($name === self::NS_OWNCLOUD_PREFIX . 'filter-rules') {
$filterRules = $reportProps['value'];
} elseif ($name === '{DAV:}prop') {
// propfind properties
Expand All @@ -205,7 +208,7 @@
foreach ($reportProps['value'] as $propVal) {
if ($propVal['name'] === '{DAV:}nresults') {
$limit = (int)$propVal['value'];
} elseif ($propVal['name'] === $ncns . 'firstresult') {
} elseif ($propVal['name'] === self::NS_NEXTCLOUD_PREFIX . 'firstresult') {
$offset = (int)$propVal['value'];
}
}
Expand Down Expand Up @@ -292,36 +295,18 @@
* @return array array of unique file id results
*/
protected function processFilterRulesForFileIDs(array $filterRules): array {
$ns = '{' . $this::NS_OWNCLOUD . '}';
$resultFileIds = [];
$circlesIds = [];
$favoriteFilter = null;
foreach ($filterRules as $filterRule) {
if ($filterRule['name'] === self::CIRCLE_PROPERTYNAME) {
$circlesIds[] = $filterRule['value'];
}
if ($filterRule['name'] === $ns . 'favorite') {
$favoriteFilter = true;
}
}

if ($favoriteFilter !== null) {
$resultFileIds = $this->fileTagger->load('files')->getFavorites();
if (empty($resultFileIds)) {
return [];
}
}

if (!empty($circlesIds)) {
$fileIds = $this->getCirclesFileIds($circlesIds);
if (empty($resultFileIds)) {
$resultFileIds = $fileIds;
} else {
$resultFileIds = array_intersect($fileIds, $resultFileIds);
}
return $this->getCirclesFileIds($circlesIds);
}

return $resultFileIds;
return [];
}

protected function processFilterRulesForFileNodes(array $filterRules, ?int $limit, ?int $offset): array {
Expand All @@ -345,27 +330,57 @@
foreach ($tags as $tag) {
$tagName = $tag->getName();
$tmpNodes = $this->userFolder->searchBySystemTag($tagName, $this->userSession->getUser()->getUID(), $dbLimit, $dbOffset);
if (count($nodes) === 0) {
$nodes = $tmpNodes;
} else {
$nodes = array_uintersect($nodes, $tmpNodes, function (INode $a, INode $b): int {
return $a->getId() - $b->getId();
});
}
$nodes = $this->intersectNodes($nodes, $tmpNodes);
if ($nodes === []) {
// there cannot be a common match when nodes are empty early.
return $nodes;
}
}

if (!$oneTagSearch && ($limit !== null || $offset !== null)) {
$nodes = array_slice($nodes, $offset, $limit);
$nodes = array_slice($nodes, $offset ?? 0, $limit);
}
}

if ($this->hasFilterFavorites($filterRules)) {
$tmpNodes = $this->userFolder->searchByTag(ITags::TAG_FAVORITE, $this->userSession->getUser()->getUID(), $limit ?? 0, $offset ?? 0);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this this thing again where we slice multiple times and then intersect the results? Does this give proper results? I forgot.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

limit = 3
offset = 2

a = [0,3,6,9,12,15,18,21,24]
b = [0,2,4,6,8,10,12,14,16,18,20,22,24]

slice(a) // 6,9,12
slice(b) // 4,6,8

intersect(slice(a), slice(b)) // 6

intersect(a,b) // 0,6,12,18,24
slice(intersect(a,b)) // 12,18,24

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, it can give too little results. Sometimes even none when some would be expected.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, will see what could be done about that

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since both the searchBySystemTag and searchByTag end up doing a search with their specific search queries anyway. You can instead build a single search query that performs all the filtering in one go.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and you can probably also include the filtering for the fileids from processFilterRulesForFileIDs in the same query to remove the need for the findNodesByFileIds later down

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can instead build a single search query that performs all the filtering in one go.

Search stuff is not in OCP, though. Would be awesome if it was.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using it in the dav app is fine since it's releases are coupled with core

$nodes = $this->intersectNodes($nodes, $tmpNodes);
if ($nodes === []) {
// there cannot be a common match when nodes are empty early.
return $nodes;
}
}

if ($limit !== null || $offset !== null) {
$nodes = array_slice($nodes, $offset ?? 0, $limit);
}

return $nodes;
}

private function intersectNodes(array $nodes, array $newNodes): array {
if (count($nodes) === 0) {
$nodes = $newNodes;
} else {
$nodes = array_uintersect($nodes, $newNodes, function (INode $a, INode $b): int {
return $a->getId() - $b->getId();
});
}

return $nodes;
}

private function hasFilterFavorites(array $filterRules): bool {
$favoriteFilter = null;
foreach ($filterRules as $filterRule) {
if ($filterRule['name'] === self::NS_OWNCLOUD_PREFIX . 'favorite') {
return true;
}
}

return false;
}

/**
* @suppress PhanUndeclaredClassMethod
* @param array $circlesIds
Expand Down
24 changes: 19 additions & 5 deletions apps/dav/tests/unit/Connector/Sabre/FilesReportPluginTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -495,7 +495,7 @@ public function testProcessFilterRulesSingle(): void {
->with('OneTwoThree')
->willReturn([$filesNode1, $filesNode2]);

$this->assertEquals([$filesNode1, $filesNode2], $this->invokePrivate($this->plugin, 'processFilterRulesForFileNodes', [$rules, 0, 0]));
$this->assertEquals([$filesNode1, $filesNode2], $this->invokePrivate($this->plugin, 'processFilterRulesForFileNodes', [$rules, null, null]));
}

public function testProcessFilterRulesAndCondition(): void {
Expand Down Expand Up @@ -934,11 +934,25 @@ public function testProcessFavoriteFilter(): void {
['name' => '{http://owncloud.org/ns}favorite', 'value' => '1'],
];

$this->privateTags->expects($this->once())
->method('getFavorites')
->willReturn(['456', '789']);
$filesNode1 = $this->createMock(File::class);
$filesNode1->expects($this->any())
->method('getId')
->willReturn(111);

$filesNode2 = $this->createMock(File::class);
$filesNode2->expects($this->any())
->method('getId')
->willReturn(222);

$this->userFolder->expects($this->exactly(1))
->method('searchByTag')
->with('_$!<Favorite>!$_')
->willReturn([
$filesNode2,
$filesNode1,
]);

$this->assertEquals(['456', '789'], array_values($this->invokePrivate($this->plugin, 'processFilterRulesForFileIDs', [$rules])));
$this->assertEquals([$filesNode1, $filesNode2], array_values($this->invokePrivate($this->plugin, 'processFilterRulesForFileNodes', [$rules, null, null])));
}

public function filesBaseUriProvider() {
Expand Down
4 changes: 2 additions & 2 deletions lib/private/Files/Node/Folder.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,8 +295,8 @@ public function searchByMime($mimetype) {
* @param string $userId owner of the tags
* @return Node[]
*/
public function searchByTag($tag, $userId) {
$query = $this->queryFromOperator(new SearchComparison(ISearchComparison::COMPARE_EQUAL, 'tagname', $tag), $userId);
public function searchByTag($tag, $userId, int $limit = 0, int $offset = 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose the interface needs to be extended as well? Though I don't see a psalm warning.
LazyFolder should look the same, alone for consistency.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjusted, would need another adjustment in groupfolders again once merged.

For potential backports this is a bit more tricky but we may just use an ugly direct call on the private class implementation of a separate method then to avoid the interface adjustment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The foundational PRs were also backported and there the Folder changes were indeed made private. Since the signature is different, it's likely taking more to make at least tooling accept that state.

$query = $this->queryFromOperator(new SearchComparison(ISearchComparison::COMPARE_EQUAL, 'tagname', $tag), $userId, $limit, $offset);
return $this->search($query);
}

Expand Down
2 changes: 1 addition & 1 deletion lib/private/Files/Node/LazyFolder.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -480,7 +480,7 @@ public function searchByMime($mimetype) {
/**
* @inheritDoc
*/
public function searchByTag($tag, $userId) {
public function searchByTag($tag, $userId, int $limit = 0, int $offset = 0) {
return $this->__call(__FUNCTION__, func_get_args());
}

Expand Down
2 changes: 1 addition & 1 deletion lib/private/Files/Node/NonExistingFolder.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ public function searchByMime($mimetype) {
throw new NotFoundException();
}

public function searchByTag($tag, $userId) {
public function searchByTag($tag, $userId, int $limit = 0, int $offset = 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That class is a good finding, i shall add searchBySystemTag here also – will be a separate PR.

throw new NotFoundException();
}

Expand Down
4 changes: 3 additions & 1 deletion lib/public/Files/Folder.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,10 +136,12 @@ public function searchByMime($mimetype);
*
* @param string|int $tag tag name or tag id
* @param string $userId owner of the tags
* @param int $limit since 28.0.0
* @param int $offset since 28.0.0
* @return \OCP\Files\Node[]
* @since 8.0.0
*/
public function searchByTag($tag, $userId);
public function searchByTag($tag, $userId, int $limit = 0, int $offset = 0);

/**
* search for files by system tag
Expand Down
Loading