API: No way to retrieve events later than a certain point

[dragotin]

The current API only takes the parameters page number and number of entries to influence the subset of entries returned.

For a client that is not suitable. We need an API that provides all new entries that came after a certain point of time. That can either be done through ID or timestamp. The client needs a way to specify a number of events to be returned at a time (already there), but also a way to query how many new entries are new after a certain point.

This blocks owncloud/client#3732, owncloud/client#3728

[DeepDiver1975]

For a client that is not suitable. We need an API that provides all new entries that came after a certain point of time.

Reading this I assume you intend to store the activities locally in the clients and want to check in a given interval for new activities - right?

So basically this would be an activity sync mechanism.

[nickvergessen]

Don't really care too much, which route we go. Either OCS or WebDAV

OCS has the pro that it exists already, con it needs to be adjusted a lot to allow starting by timestamp and more, so we either break the existing endpoint or need to make a new one, so the already-existing benefit is gone.

Anyone with power please make the call.

[DeepDiver1975]

Don't really care too much, which route we go. Either OCS or WebDAV

Protocol is to be choosen at a later stage - for now I want to understand the client's expectations and it's operations.

@dragotin

[dragotin]

I was thinking about this:

• Client maintains a local storage of the events to be able to display them quickly and also in offline mode
• From that local storage, client can find out the latest event it knows, ie. the highest timestamp, or maybe better id.
• to fetch new events from the server, it sends the last known highest timestamp along and the server returns the new events that came after that.
• To not get completely flooded client sends a maximum of events per call, and server replies accordingly with that amount maximum. If there are more that has to be indicated somehow.

But maybe I misunderstood the API completely?

[nickvergessen]

Well the current API is like a paged search. Just like you would get from a book or in the web ui.
However the local storing and offline thing makes sense to me and I also see no other way then having a "I know everything up to activity #X" approach.

[karlitschek]

@nickvergessen OCS please. Let's stay at the current route with the current API unless a change is absolutely necessary.

[nickvergessen]

Endpoint

1. Specify the last known ID
2. Sorts ascending
3. Header specifies the first known ID (in case the given one was pruned (time))
4. Header specifies if there are more activities (only bool, not a number or something)
5. Client reads header from 3 and throws away all activities before it.
6. Client reads header from 4 and then calls again with the newest ID from the previous call

Questions

• What happens when disabling an activity type in the personal settings? There might be invalid activities in the middle, that need to be deleted, but the server does not know which activities are in the client that should be deleted as well.
  • Should we also send a list of "activity types to keep", and the client drops activities that are not set anymore?
  • Enabling a type is not that big of a problem, because the server may not have stored the action, but it can happen that an activity is in the WebUI which the client is missing, because it didn't sync back then. But I'd say "sorry, this setting may not be retro-active"

Request for all activities

GET /cloud/activity

With type filter

GET /cloud/activity/:filter

Parameters

Name	Type	Description
since	int (Optional)	The integer ID of the last activity that you’ve seen.
limit	int (Optional)	How many activities should be returned (Default: 50)
object_type	string (Optional)	Allows to filter the activities to a given object. May only appear together with object_id
object_id	string (Optional)	Allows to filter the activities to a given object. May only appear together with object_type
sort	string - asc or desc	Should activities be given ascending or descending (from the since) (Default: desc)

Status

Status Code	Response	Description
200 OK	[{Activity element}(,{Activity element})]	Activities
304 Not Modified	[]	No new activities
403 Forbidden		The offset activity belongs to a different user
404 Not Found		The filter is unknown

Headers

Link for the next request

Already includes all parameters

Link: <http(s)://localhost/ocs/v1.php/cloud/activity?since=364>; rel="next"

First known activity

In case the since parameter was not known, the header gives the first known activity ID

X-Activity-First-Known: 370

Activity element

Field name	Type	Value description
activity_id	int	Autoincrement value from the database
timestamp	int	UNIX timestamp of the activity
app	string	App that created the activity (e.g. 'files', 'files_sharing', etc.)
type	string	For most files related activities this is the action that was performed on the file/folder (e.g. 'file_changed', 'file_created' (same is used for both, file and folder)), other apps use other strings (e.g. 'announcementcenter')
user	string (Optional)	User ID of the user that triggered/created this activity (can also be empty in case of public link/remote share action)
affecteduser	string	User ID of the user that received this activity (always the same)
subject	string	Untranslated arbitrary subject (e.g. 'changed_self', 'changed_by')
subjectparams	array	Array with one array per parameter, each parameter array has the key 'type' and 'value'. Value is an array-of-parameters in case of type collection and a string otherwise
message	string (Optional)	Untranslated arbitrary message (unused by core apps)
messageparams	array	See subjectparams
link	string (Optional)	A full URL pointing to a suitable location (e.g. 'http://localhost/ownCloud/master/core/index.php/apps/files/?dir=%2Ffolder' in case the folder got created)
object_type	string (Optional)	Type of the object this activity is about (e.g. 'files' is used for files and folders)
object_id	string (Optional)	ID of the object this activity is about (e.g. ID in the file cache is used for files and folders)
object_name	string (Optional)	Name of the object this activity is about (e.g. for files it's the relative path to the user's root: '/folder/.travis.yml')
subjectformatted	array
subjectformatted.trimmed	string	Short version of the subject (only filename instead of path, etc. You changed abc.txt)
subjectformatted.full	string	Long version of the subject (e.g. You changed path/to/file/abc.txt)
subjectformatted.markup	array
subjectformatted.markup.trimmed	string	Short version of the subject (only filename instead of path, etc. You changed <a class="filename has-tooltip" href="/ownCloud/master/core/index.php/apps/files?dir=%2Ffolder&scrollto=abc.txt" title="in path/to/file">abc.txt</a>)
subjectformatted.markup.full	string	Short version of the subject (e.g. You changed <a class="filename has-tooltip" href="/ownCloud/master/core/index.php/apps/files?dir=%2Ffolder&scrollto=abc.txt">path/to/fileabc.txt</a>)
messageformatted	array	See subjectformatted and it's children (unused by core apps, values can be very long)
activity_ids	array of int (Optional)	Array of activity_ids that have been grouped into this activity

In case the endpoint returns more fields, they should be ignored and are deprecated (only for backwards compatibility usage) or internal.

Example

  {
    "activity_id": 1,
    "timestamp": 1446137950,
    "app": "files",
    "type": "file_deleted",
    "user": "admin",
    "affecteduser": "admin",
    "subject": "deleted_self",
    "subjectparams": [
      {
        "value": "\/welcome.txt",
        "type": "file"
      }
    ],
    "message": "",
    "messageparams": [

    ],
    "link": "",
    "object_type": "files",
    "object_id": 3,
    "object_name": "\/welcome.txt",
    "subjectformatted": {
      "trimmed": "You deleted welcome.txt",
      "full": "You deleted path/to/file/welcome.txt",
      "markup": {
        "trimmed": "You deleted <a class=\"filename\" href=\"\/ownCloud\/master\/core\/index.php\/apps\/files?dir=%2F&scrollto=welcome.txt.d1446137950&view=trashbin\" title="in path/to/file">welcome.txt<\/a>",
        "full": "You deleted <a class=\"filename\" href=\"\/ownCloud\/master\/core\/index.php\/apps\/files?dir=%2F&scrollto=welcome.txt.d1446137950&view=trashbin\"path/to/file/>welcome.txt<\/a>"
      }
    },
    "messageformatted": {
      "trimmed": "",
      "full": "",
      "markup": {
        "trimmed": "",
        "full": ""
      }
    }
  }

Usages

• Web UI uses subjectformatted.markup.trimmed and messageformatted.markup.trimmed
• RSS feed uses subjectformatted.full and messageformatted.full

Notes

The endpoint does not yet output that information

[DeepDiver1975]

THX @nickvergessen

We need to specify the url parameters of the request as well.

May I suggest to add some example code like https://developer.github.com/v3/repos/#get ? THX

[nickvergessen]

Updated

[dragotin]

Nice, this looks reasonable.

Some questions/remarks:

• How do we specify how many results come back with a call to the new api?
• I don't think using headers here is nice. Can we add the next-link and the there-is-more to the json doc?
• What is the difference between trimmed and full for message and subject?
• We should not duplicate the strings with markups in the result. Let's use an Accept header to let each client specify the format it wants back.
• I think object_type and object_name are redundant, as they are already there as "subjectparams". How about adding the object_id to the subject params as well?

[DeepDiver1975]

I don't think using headers here is nice. Can we add the next-link and the there-is-more to the json doc?

this approach follows a common standard - https://tools.ietf.org/html/rfc5988 - used by e.g. Github as well

[DeepDiver1975]

We should not duplicate the strings with markups in the result. Let's use an Accept header to let each client specify the format it wants back.

Accept is to specify the full body format - not individual elements

[nickvergessen]

• How do we specify how many results come back with a call to the new api?

Missed it, added now:

Name	Type	Description
limit	int (Optional)	How many activities should be returned (Default: 50)

• I don't think using headers here is nice. Can we add the next-link and the there-is-more to the json doc?

Well then we need to wrap the result once more, because stuff can not be added to the meta setion, resulting in: ocs > data > activities > element+

• What is the difference between trimmed and full for message and subject?

As per description above:

• trimmed - Short version of the subject (only filename instead of path, etc. You changed abc.txt)
• full - Long version of the subject (e.g. You changed path/to/file/abc.txt)

• We should not duplicate the strings with markups in the result. Let's use an Accept header to let each client specify the format it wants back.

Not possible, because the header is used by OCS to decide if json or xml is to be returned.

• I think object_type and object_name are redundant, as they are already there as "subjectparams". How about adding the object_id to the subject params as well?

No, only in some cases. When stuff is grouped (instead of a single activity) or when there are more parameters, they are not equal anymore. But yes, object_id will most likely be added to the path for future activities

[dragotin]

@DeepDiver1975 @nickvergessen ok, how do we resolve the problem with the mark-up cluttering? We do not want to add mark ups for all use cases (webIF, RSS, desktop, mobile, no idea..), see that for example the css class-definitions do not make sense for clients.

How about a reduced "meta markup" - maybe rst-based - that needs to be converted to the final on each client?

[nickvergessen]

Well that is why we have versions completly without markup:

    "subjectformatted": {
      "trimmed": "You deleted welcome.txt",
      "full": "You deleted path/to/file/welcome.txt",

But I see that you don't really want to have that either.

I mean we can do something like <user>admin</user> changed <file>path/to/file.txt</file> so you can parse it your self.
In case of RSS we'd then just remove the markup altogether to have admin changed path/to/file.txt while for the webui we turn it into something like the stuff that is returned atm.

Grouping could be an collection item: <user>admin</user> changed <collection><file>path/to/file1.txt</file><file>path/to/file2.txt</file>....</collection> which then translates to admin changed path/to/file1.txt and path/to/file2.txt or admin changed path/to/file1.txt, path/to/file2.txt and 5 more.

Would work for me.

Is that would you had in mind?

[dragotin]

Well, no markup at all is not useful as well.

I like @nickvergessen idea of the simplified markup. However, the solution should completely preserve the translation, otherwise the texts become completely useless.

Another alternative would be to add a parameter to the request which markups should be added, ie. markup=rss or markup=plain or so. That would mean that the server would have to maintain a list of possibly requested markups.

[nickvergessen]

Another alternative would be to add a parameter to the request which markups should be added, ie. markup=rss or markup=plain or so.

I mean we do that indirectly atm anyway (that's how we ended up with those options (full path, html, grouping, ...)
I have no problem in turning them into markup groups. The question is, which markup would you like to have?

[dragotin]

@nickvergessen lets keep it simple for now:

• link to mark complete links
• relativelink to mark relative links on the same ownCloud (do we need that?)
• user the principal name of the user - to be fed into at API
• file a file relative on the cloud - or on the client.

I do not really see the need for the collection thing you mentioned. What do I overlook?

[nickvergessen]

We have "other" parameters as well, like calendar names, federated cloud ids, for the sake of sanity I'd just put them into tags as well, define a list somewhere and you can just strip them out if you feel like it.

[nickvergessen]

Merged #434

[dragotin]

The API turns out to not be suitable to fulfill the needs of WebUI and client. That is for the following reasons:

• field subjectparams does not need the collection type
• the object_* fields are not of use
• the *_prepared fields do not really contain displayable strings with marked object user, file and such, as it was agreed on before.

@nickvergessen saw more issues. Unfortunately the required changes can not go into 9.0 any more.

[karlitschek]

Argh. @nickvergessen @dragotin Not sure how this happened. Any suggestions how we can fi this mess?
@DeepDiver1975 any input?

[nickvergessen]

Well we need this scheduled in the same time frame as the client so we can work on it concurrently.

Most likely we have to break the activity manager api, as it currently allows for random output, even html works. We need to define a structure that works for more api users than just the web ui and then all activity creating apps need to obey this new format.

We also need to define what we do with objects we don't know, so objects should come with a type and id additional to the name, but in a defined format. When the api consumer then does not know the type it simply strips out the tag and only retains the name and the result should be usable, that's what I understood from Klaas' request?

[ghost]

does 9.0.1 work?

[karlitschek]

This sounds like more changes are needed then just a bugfix. So 9.1 it seems. :-( Again. Not sure how this disconnect could have happened on the architecture side. @nickvergessen @DeepDiver1975 Can you please schedule a call with me to discuss a possible way out? Thanks.

[DeepDiver1975]

@karlitschek so the original v2 OCS API was removed from the public api. So no unused API will be released - which is good.

I suggest we discuss the API to be used from 9.1 on during the design week in Nürnberg - with @dragotin being there as well we can get this sorted out.

[karlitschek]

agreed. let restart this for 9.1

[dragotin]

@karlitschek mentioned something about a public standard that could be used here.

[MTRichards]

In the interim between 9 and 9.1, a draft standard from W3C emerged. There is the opportunity to use this standard for this activity stream, rather than implementing our own. As it is in draft, we can put in feature requests.
Note: Types of activity stream are defined by W3C, and we need to correlate ownCloud values to W3C pre-defined types.

Plan for oC

• Support the old API with the new W3C API, and also introduce a way to jump back and fetch activities from a certain time.

Any other W3C API specification support is for a version 2.0.

[MTRichards]

Estimation: 2 weeks

[PVince81]

I'm moving this to backlog for now until this gets rescheduled.

cc @guruz to recheck this old requirement

cc @pmaier1