Add time window in search requests #2126

artonge · 2023-11-13T13:04:04Z

Rational

The current SEARCH request is done on all the files. Which means that for each requests, the server has to get all the files, order them and pick the first 200. This is really heavy when the user as a lot of photos.

The goal of this PR is to limit the query to a certain time window, so the server works with a subset of the user's photos.
The front-end now sends an upper and a lower time bound.
The front-end now also need to be smart about the requests in order to limit the number of requests:

First make a request on the last 4 months
If nothing is found, increase to the last 16 months
If nothing is found, increase to the last 64 months
If nothing is found, remove the lower bound
If nothing is found, remove the upper bound

Limitation:

If the file does not have the original date time metadata, then it will only be returned when the lower bound is removed.
If the file has an original date time superior to the current date time, then it will be return when the upper bound is removed

Test requests

Test request with both condition applied:

curl -k 'https://louis:password@cloud.nextcloud.com/remote.php/dav/' \
  -X 'SEARCH' \
  -H 'content-type: text/xml' \
  -H 'depth: infinity' \
  --data '<?xml version="1.0" encoding="UTF-8"?>
  <d:searchrequest xmlns:d="DAV:"
    xmlns:oc="http://owncloud.org/ns"
    xmlns:nc="http://nextcloud.org/ns"
    xmlns:ns="https://github.com/icewind1991/SearchDAV/ns"
    xmlns:ocs="http://open-collaboration-services.org/ns">
    <d:basicsearch>
      <d:select>
        <d:prop>
          <d:getcontentlength />
          <d:getcontenttype />
          <d:getetag />
          <d:getlastmodified />
          <d:resourcetype />
          <nc:face-detections />
          <nc:face-preview-image />
          <nc:metadata-photos-size />
          <nc:metadata-photos-original_date_time />
          <nc:has-preview />
          <nc:realpath />
          <oc:favorite />
          <oc:fileid />
          <oc:permissions />
          <nc:nbItems />
        </d:prop>
      </d:select>
      <d:from>
        <d:scope>
          <d:href>/files/louis/</d:href>
          <d:depth>infinity</d:depth>
        </d:scope>
      </d:from>
      <d:where>
        <d:and>
          <d:or>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/png</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/jpeg</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/heic</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/webp</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>video/mp4</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>video/quicktime</d:literal>
            </d:eq>
          </d:or>
          <d:or>
            <d:and>
              <d:lte>
                <d:prop><nc:metadata-photos-original_date_time /></d:prop>
                <d:literal>1703090557</d:literal>
              </d:lte>
              <d:gt>
                <d:prop><nc:metadata-photos-original_date_time /></d:prop>
                <d:literal>1700498557</d:literal>
              </d:gt>
            </d:and>
            <d:and>
              <d:not>
                <d:is-defined>
                  <d:prop><nc:metadata-photos-original_date_time /></d:prop>
                </d:is-defined>
              </d:not>
              <d:lte>
                <d:prop><d:getlastmodified /></d:prop>
                <d:literal>1703090557</d:literal>
              </d:lte>
              <d:gt>
                <d:prop><d:getlastmodified /></d:prop>
                <d:literal>1700498557</d:literal>
              </d:gt>
            </d:and>
          </d:or>
        </d:and>
      </d:where>
      <d:orderby>
        <d:order>
          <d:prop><nc:metadata-photos-original_date_time/></d:prop>
          <d:descending/>
        </d:order>
        <d:order>
          <d:prop><d:getlastmodified/></d:prop>
          <d:descending/>
        </d:order>
      </d:orderby>
      <d:limit>
        <d:nresults>5</d:nresults>
        <ns:firstresult>0</ns:firstresult>
      </d:limit>
    </d:basicsearch>
  </d:searchrequest>' | xq '.["d:multistatus"]["d:response"][] | .["d:href"]'

Test request with condition applied on getlastmodified and check that metadata is NULL:

curl -k 'https://louis:password@cloud.nextcloud.com/remote.php/dav/' \
-X 'SEARCH' \
-H 'content-type: text/xml' \
-H 'depth: infinity' \
--data '<?xml version="1.0" encoding="UTF-8"?>
<d:searchrequest xmlns:d="DAV:"
  xmlns:oc="http://owncloud.org/ns"
  xmlns:nc="http://nextcloud.org/ns"
  xmlns:ns="https://github.com/icewind1991/SearchDAV/ns"
  xmlns:ocs="http://open-collaboration-services.org/ns">
  <d:basicsearch>
    <d:select>
      <d:prop>
        <d:getcontentlength />
        <d:getcontenttype />
        <d:getetag />
        <d:getlastmodified />
        <d:resourcetype />
        <nc:face-detections />
        <nc:face-preview-image />
        <nc:metadata-photos-size />
        <nc:metadata-photos-original_date_time />
        <nc:has-preview />
        <nc:realpath />
        <oc:favorite />
        <oc:fileid />
        <oc:permissions />
        <nc:nbItems />
      </d:prop>
    </d:select>
    <d:from>
      <d:scope>
        <d:href>/files/louis/</d:href>
        <d:depth>infinity</d:depth>
      </d:scope>
    </d:from>
    <d:where>
      <d:and>
        <d:or>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>image/png</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>image/jpeg</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>image/heic</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>image/webp</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>video/mp4</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>video/quicktime</d:literal>
          </d:eq>
        </d:or>
        <d:and>
          <d:not>
            <d:is-defined>
              <d:prop><nc:metadata-photos-original_date_time /></d:prop>
            </d:is-defined>
          </d:not>
          <d:lte>
            <d:prop><d:getlastmodified /></d:prop>
            <d:literal>1703090557</d:literal>
          </d:lte>
          <d:gt>
            <d:prop><d:getlastmodified /></d:prop>
            <d:literal>1700498557</d:literal>
          </d:gt>
        </d:and>
      </d:and>
    </d:where>
    <d:orderby>
      <d:order>
        <d:prop>
          <nc:metadata-photos-original_date_time />
        </d:prop>
        <d:descending />
      </d:order>
      <d:order>
        <d:prop>
          <d:getlastmodified />
        </d:prop>
        <d:descending />
      </d:order>
    </d:orderby>
    <d:limit>
      <d:nresults>5</d:nresults>
      <ns:firstresult>0</ns:firstresult>
    </d:limit>
  </d:basicsearch>
</d:searchrequest>' | xq '.["d:multistatus"]["d:response"][] | .["d:href"]'

Test request with condition applied on metadata:

curl -k 'https://louis:password@cloud.nextcloud.com/remote.php/dav/' \
  -X 'SEARCH' \
  -H 'content-type: text/xml' \
  -H 'depth: infinity' \
  --data '<?xml version="1.0" encoding="UTF-8"?>
  <d:searchrequest xmlns:d="DAV:"
    xmlns:oc="http://owncloud.org/ns"
    xmlns:nc="http://nextcloud.org/ns"
    xmlns:ns="https://github.com/icewind1991/SearchDAV/ns"
    xmlns:ocs="http://open-collaboration-services.org/ns">
    <d:basicsearch>
      <d:select>
        <d:prop>
          <d:getcontentlength />
          <d:getcontenttype />
          <d:getetag />
          <d:getlastmodified />
          <d:resourcetype />
          <nc:face-detections />
          <nc:face-preview-image />
          <nc:metadata-photos-size />
          <nc:metadata-photos-original_date_time />
          <nc:has-preview />
          <nc:realpath />
          <oc:favorite />
          <oc:fileid />
          <oc:permissions />
          <nc:nbItems />
        </d:prop>
      </d:select>
      <d:from>
        <d:scope>
          <d:href>/files/louis/</d:href>
          <d:depth>infinity</d:depth>
        </d:scope>
      </d:from>
      <d:where>
        <d:and>
          <d:or>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/png</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/jpeg</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/heic</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/webp</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>video/mp4</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>video/quicktime</d:literal>
            </d:eq>
          </d:or>
          <d:lte>
            <d:prop>
              <nc:metadata-photos-original_date_time/>
            </d:prop>
            <d:literal>1703090557</d:literal>
          </d:lte>
          <d:gt>
            <d:prop>
            <nc:metadata-photos-original_date_time /></d:prop>
            <d:literal>1700498557</d:literal>
          </d:gt>
        </d:and>
      </d:where>
      <d:orderby>
        <d:order>
          <d:prop><nc:metadata-photos-original_date_time/></d:prop>
          <d:descending/>
        </d:order>
        <d:order>
          <d:prop><d:getlastmodified/></d:prop>
          <d:descending/>
        </d:order>
      </d:orderby>
      <d:limit>
        <d:nresults>5</d:nresults>
        <ns:firstresult>0</ns:firstresult>
      </d:limit>
    </d:basicsearch>
  </d:searchrequest>' | xq '.["d:multistatus"]["d:response"][] | .["d:href"]'

artonge · 2023-11-13T13:49:45Z

/compile amend /

artonge · 2023-11-13T22:03:09Z

Note: maybe I can send the request as:

originalDateTime > lowerBound || (originalDateTime === null && modtime > lowerBound)
Same with upper bound. This would get rid of limitation n°1.

artonge · 2023-11-13T22:03:59Z

To get rid of limitation n°2, I can send the request without upper bound first.

artonge · 2023-11-14T09:54:14Z

Check favorites and on this day views

Signed-off-by: Louis Chemineau <louis@chmn.me>

artonge self-assigned this Nov 13, 2023

artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch 2 times, most recently from f5b38a2 to 265ac3a Compare November 13, 2023 13:36

artonge added enhancement New feature or request 3. to review Waiting for reviews javascript Javascript related ticket performances Performances issues and optimisations labels Nov 13, 2023

artonge added this to the Nextcloud 28 milestone Nov 13, 2023

artonge requested review from susnux, Pytal and ArtificialOwl November 13, 2023 13:51

artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch from 265ac3a to 791761f Compare November 13, 2023 14:10

Base automatically changed from artonge/sort_by_taken_date to master November 13, 2023 14:26

artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch 2 times, most recently from d4d15b3 to 498ecdb Compare November 13, 2023 17:51

artonge enabled auto-merge November 13, 2023 21:40

blizzz mentioned this pull request Nov 14, 2023

28.0.0 beta 3 nextcloud/server#41379

Merged

artonge marked this pull request as draft November 14, 2023 09:59

auto-merge was automatically disabled November 14, 2023 09:59
Pull request was converted to draft

blizzz mentioned this pull request Nov 14, 2023

28.0.0 beta 4 nextcloud/server#41477

Merged

artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch from 18582ec to 0e654e1 Compare November 15, 2023 15:40

blizzz mentioned this pull request Nov 20, 2023

28.0.0 RC1 nextcloud/server#41621

Merged

5 tasks

blizzz modified the milestones: Nextcloud 28, Nextcloud 29 Nov 23, 2023

artonge added 2 commits December 21, 2023 13:41

Add time window in search requests

5fe4f3e

Signed-off-by: Louis Chemineau <louis@chmn.me>

WIP

3239928

Signed-off-by: Louis Chemineau <louis@chmn.me>

WIP

b003a1a

Signed-off-by: Louis Chemineau <louis@chmn.me>

artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch from 4e8148c to b003a1a Compare December 21, 2023 12:42

artonge added 2 commits December 21, 2023 15:52

WIP

9a2fe26

Signed-off-by: Louis Chemineau <louis@chmn.me>

WIP

9bfc44b

Signed-off-by: Louis Chemineau <louis@chmn.me>

This was referenced Mar 12, 2024

29.0.0 beta 2 nextcloud/server#44152

Merged

29.0.0 beta 3 nextcloud/server#44186

Merged

29.0.0 beta 4 nextcloud/server#44275

Merged

Altahrim mentioned this pull request Mar 20, 2024

29.0.0 beta 5 nextcloud/server#44361

Merged

Altahrim modified the milestones: Nextcloud 29, Nextcloud 29.0.1 Apr 24, 2024

skjnldsv modified the milestones: Nextcloud 29.0.1, Nextcloud 29.0.2 May 28, 2024

blizzz modified the milestones: Nextcloud 29.0.2, Nextcloud 29.0.4 Jun 25, 2024

Altahrim modified the milestones: Nextcloud 29.0.4, Nextcloud 29.0.5 Jul 19, 2024

skjnldsv modified the milestones: Nextcloud 29.0.5, Nextcloud 29.0.6 Aug 21, 2024

blizzz modified the milestones: Nextcloud 29.0.6, Nextcloud 29.0.7 Sep 3, 2024

blizzz modified the milestones: Nextcloud 29.0.7, Nextcloud 29.0.9 Oct 10, 2024

Altahrim modified the milestones: Nextcloud 29.0.9, Nextcloud 29.0.10 Nov 7, 2024

blizzz modified the milestones: Nextcloud 29.0.10, Nextcloud 29.0.11 Dec 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add time window in search requests #2126

Add time window in search requests #2126

artonge commented Nov 13, 2023 •

edited

Loading

artonge commented Nov 13, 2023

artonge commented Nov 13, 2023 •

edited

Loading

artonge commented Nov 13, 2023 •

edited

Loading

artonge commented Nov 14, 2023

Add time window in search requests #2126

Are you sure you want to change the base?

Add time window in search requests #2126

Conversation

artonge commented Nov 13, 2023 • edited Loading

Rational

Test requests

artonge commented Nov 13, 2023

artonge commented Nov 13, 2023 • edited Loading

artonge commented Nov 13, 2023 • edited Loading

artonge commented Nov 14, 2023

artonge commented Nov 13, 2023 •

edited

Loading

artonge commented Nov 13, 2023 •

edited

Loading

artonge commented Nov 13, 2023 •

edited

Loading