[RFC] FileInfo API

[skjnldsv]

Goals

Provides a way to add new properties to a Node and have a parity between the DAV properties and what a dedicated method could output via an OCS API request

Use case

Whether I am processing a list of files from the dav Endpoint or returning a list of files from an ocs API endpoint, I expect the same kind of data to be available with the same property names for a back-front parity. Reducing the frontend complexity and increase code readibility.

Issue example

Photos currently gets the list of files via SEARCH request, but get the gallery files set via ocs API, forcing us to do something like that:
https://github.com/nextcloud/photos/blob/aebb6e6695885cd5adf5d41c06a6b5c917f6cf7e/lib/Controller/AlbumsController.php#L107-L117

$result[] = [
	'basename' => $isRoot ? '' : $node->getName(),
	'etag' => $node->getEtag(),
	'fileid' => $node->getId(),
	'filename' => $path,
	'lastmod' => $node->getMTime(),
	'mime' => $node->getMimetype(),
	'size' => $node->getSize(),
	'type' => $node->getType(),
	'hasPreview' => $this->previewManager->isAvailable($node),
];

https://github.com/nextcloud/photos/blob/aebb6e6695885cd5adf5d41c06a6b5c917f6cf7e/src/utils/fileUtils.js#L99-L122

const genFileInfo = function(obj) {
	const fileInfo = {}

	Object.keys(obj).forEach(key => {
		const data = obj[key]

		// flatten object if any
		if (!!data && typeof data === 'object') {
			Object.assign(fileInfo, genFileInfo(data))
		} else {
			// format key and add it to the fileInfo
			if (data === 'false') {
				fileInfo[camelcase(key)] = false
			} else if (data === 'true') {
				fileInfo[camelcase(key)] = true
			} else {
				fileInfo[camelcase(key)] = isNumber(data)
					? Number(data)
					: data
			}
		}
	})
	return fileInfo
}

Proposal

Create a dedicated php API for apps allowing to extend a file with some custom properties for a given namespace (e.g http://nextcloud.org/ns)

The API will ensure the following

• Register those new properties to the DAV ServerPlugin

  $propFind->handle(self::MOUNT_TYPE_PROPERTYNAME, function () use ($node) {
      return $node->getFileInfo()->getMountPoint()->getMountType();
  });

• Register those properties to the FileInfo API
• Allow a node to be directly serialised to a similar output as the dav requests after processing from the webdav API

  {
    basename: 'file.txt',
    path: '/folder/file.txt',
    ...
    'nc/has-preview': true,
  }

• A node library should be created to help matching both back and front I assume. Helping making sure all apps can manage that easily.

Questions

• Should we drop or keep the namespace for the final serialised object ?
• If namespaced, what format do we use for the keys?
  • @nc/has-preview
  • nc/has-preview
  • nc-has-preview
  • nc_has-preview
  • @nc/hasPreview
  • nc/hasPreview
  • nc_hasPreview
  • nc-hasPreview
  • ...
• ...

Comments, questions, arguments? @ChristophWurst @juliushaertl @nextcloud/server @nickvergessen

[ChristophWurst]

Objectives

• Standardized representation of file info objects, independent of their source

Standardized representation

Front end

On the front end it will make most sense if we use JSON as the the data format for file info objects, as that makes the representation technology-agnostic.

The standard has to define all possible properties, their types and allowed values/ranges.

E.g.

{
    basename: 'cat.jpg', // string, can't be empty
    lastmod: null, // int|null
}

Extended file info

Apps may extend a file info object with proprietary properties. Since more than one app might extend a single object, the property name will be namespaced by the app ID.

The details of the namespace are to be defined. As an example, proprietary props of the Deck app could extend the base file info standard like this:

{
    basename: 'cat.jpg', // string, can't be empty
    lastmod: null, // int|null,
    deck_projectId: 13,
}

Unification of DAV, OCS and REST

No matter where a file info object comes from, we want to be able to convert it into the standard representation. OCS and REST and likely already transport the file info in the standard representation, if the back end is aware of the structure.

On the front end only DAV objects have to be converted.

import { fileInfoFromDav } from '@nextcloud/files';

const dav = ...
const fileInfo = fileInfoFromDav(dav)

From an implementation perspective the function will need a map of XML namespaces and property names that are translated to the standardized JSON properties.

For proprietary properties a map can extend the translation.

import { fileInfoFromDav } from '@nextcloud/files';

const dav = ...
const fileInfo = fileInfoFromDav(dav, {
    '{http://nextcloud.com/ns}deck-project-id': ['deck', 'projectId'],
})

Back end

To be able to write a standardized JSON file info object it will make sense to have an OCP helper so that this functionality doesn't have to be implemented in each app.

The helper will look something like

interface IProperty {
    public function getAppId(): string;
    public function getValue(): mixed;
}

/**
 * @param IProperty[] $proprietary
 */
createStandardFileInfo(FileInfo $fileInfo, array $proprietary): JsonSerializable;

Promotion of props & versioning

The standardized representation will likely undergo a series of changes as we add or change the default properties. Moreover if some properties of one app are to be used in other apps we can promote them to a standardized property.

To be able to reason about a given file info object and the available props it will make sense to version the objects.

E.g. when you get a v1.0 file info object you know that there is a basename prop. However, if you expect a metadata property then only v1.3 has it.

File objects created dynamically will be easy to handle. Where the versioning will help is whenever a file info is serialized (to cache, database, other) for later use.

Old brain dump

What we discussed was different. We talked about standardizing the format of a file info object. The goal was to get the same object from any endpoint.

Data is from OCS/REST -> standardized JSON representation
Data is from DAV -> XML representation -> standardized JSON representation

The standard file info object and its default properties has to be defined somewhere, e.g. our dev docs. The definition would specify what the JSON representation looks like.

E.g.

{
    basename: '', // string
    lastmod: null, // int|null
}

Then we discussed how to handle custom props. Those will be possible to add to this object. Due to likely naming conflicts there should be a prefix to name space them.

The conversion from the DAV XML representation to the standardized JSON representation can be offered as a helper function in a node package. To work with custom props the helper function also needs to accept custom mappings of namespaces. E.g. so that an app can extend the file info for its purposes and use the extende file info object in the front-end.

On the back-end we can also offer a helper to generate a standard and extended file info object with the namespaced props.

Another point about the standardization. When a custom prop is used by more than one app, then it should be considered to be promoted as standard property, without the namespace. All helpers have to be adapted for this. So we probably need (semantic) versioning on the file info API.

[ChristophWurst]

I've refined my comment from yesterday.

[skjnldsv]

E.g. when you get a v1.0 file info object you know that there is a basename prop. However, if you expect a metadata property then only v1.3 has it.

File objects created dynamically will be easy to handle. Where the versioning will help is whenever a file info is serialized (to cache, database, other) for later use.

This is interesting.
I initially thought about using the same way DAV does it. If you request a prop that is not her,e it's in the 404 section and not here. If the front end knows it straight away, it's easy to turn on and off features based on what prop is available (or namespaces)
If you see the metadata key, then you know it's here, and usable, else: 🤷

[ChristophWurst]

Guess it comes down to preferences. A versioned, typed object would give you static guarantees but if you check for existence every time you use a property and also code the patch for the lack of it then it's fine to do duck typing.

[skjnldsv]

First front implementation done in
https://nextcloud.github.io/nextcloud-files/classes/File.html
https://nextcloud.github.io/nextcloud-files/classes/Folder.html

[CarlSchwan]

Register those properties to the FileInfo API

What is the plan to how to do it? I would heavily prefers if this was done in the backend (app register the custom dav properties they handle but also register that for the file listing they need this properties)

Currently this is done in the frontend and it means we need to parse every js files before we can do the PROPFIND, this cause a 300ms second delays on the page load before we can start fetching the file listing: see #31160

[skjnldsv]

@CarlSchwan ideally, for all dav-related props, the backend should dictate the frontend yep.
Having a library or mappings between what exists and what can be requested to the backend would be nice.

For the new files app, there will be no plugins system.
Each view is responsible for their own data fetch and will have to return an array of File and Folder.