Why the library always passes timestamp to the server?

[george-kozlov]

Documentation says that timestamp is optional on the server side - https://docs.influxdata.com/influxdb/v0.10/guides/writing_data/

Why is the library passing timestamp always? There is a code in the build() method of Point class that validates the incoming value and if NULL passed, it uses System.currentTimeMillis().
As the result, the library is passing timestamp to the server, not depend on user's choice.
Additional validation is in formatedTime() method of Point class.

We would like to don't pass timestamp to the server to let InfluxDB generate timestamp on its own. There are more than 100 machines on our side which will write data to the database so we would like server to guarantee uniqueness of each record.

[andrewdodd]

Hi @george-kozlov

Yes, I can see that what you are saying is true. Maybe if I get some time I can look into this.

However, for me it makes sense that the library takes the timestamps and that it does not rely on the server for the following reasons:

• The time at which the library calls System.currentTimeMillis() is as close as possible (temporally) to when the event occurred (i.e. it is only a few CPU cycles between when your code created the point and when the timestamp it recorded).
• The time is stored with the recording, so if the 'write' fails it could be retried at a later point in time but would be recorded against the correct time.
• The effects of network latency on the recorded time would be difficult to account for and would (in my opinion) make testing/troubleshooting difficult.

On a side note, I have had success with setting up my InfluxDB server as an NTP server and having my clients sync to it (though this would obviously depend a lot on the time resolution you are after).

Additionally, I have a query about the comment 'guarantee uniqueness'... I don't think I understand this point? Are you tagging your points with something from each of the different 100 machines? (i.e. a tag like "machineId=1" / "machineId=2" or "machineMAC="aabbcc112233"?)

Thanks

[george-kozlov]

Hi Andrew,

Thanks for reply.
No, we don't tag our points with any unique value, at least for this data.
Theoretically, if many machines are running, two machines can push the data with the same timestamp and similar tags, so two records will be considered as the same by the server and it won't save both.
I know that such case is the exception rather than the rule, but we already run 150 machines in the cloud, the more machine we run, the more probable the problem become.
That's why we would like to don't pass the timestamp and allow the server to generate value on its own.

In the library, API you allow the null value for the timestamp, but generate and pass timestamp in any case, no matter whether a caller is passing it or not. I believe this is wrong, and the caller should choose whether to pass the timestamp or not while working with the library.

Regards,
George

[andrewdodd]

I don't think this change would be too hard to do, so if you want to submit a PR I'll have a look. I imagine it would be adding a method public Point build(boolean omitTimestamp); to the Point class.

You could also tag your points with a unique value. ;-)

[george-kozlov]

We write more than a million records daily and I don't think add unique value to every point just, it's the exception rather than the rule.
I strongly believe that if the server supports that feature (timestamp is not mandatory), the client libraries should expose it through their APIs as well.

We've already made a fix in our own version of the client and use modified version now. Meanwhile I would like to have that fix in the client library out of the box, this way we can easy replace the library instead of making the fix with every client update.

Can you please add the overloaded method like you suggested?

[andrewdodd]

I'll have a go when I get the time. Maybe this weekend?

(I was unclear about the 'unique' tag...I meant add a tag that is unique for each client that you have, that way the combination of {tag & timestamp} would be unique from InfluxDBs perspective, thus records with identical timestamps but from different machines will be kept).

[zanella]

I've implemented a tentative solution in the MR #168

[george-kozlov]

Hi Rafael,

Thanks for update!
We will replace homemade version with this one.

Regards,
George Kozlov Senior Cloud Architect

Lexalytics Read Between The Lines™

http://bd.semantria.com/a7f5c9324d3f737a5cwww.lexalytics.comhttp://www.lexalytics.com/

This message (including any attachments) may contain confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and contact the sender.

From: Rafael Zanella <notifications@git.luolix.topmailto:notifications@github.com>
Reply-To: influxdata/influxdb-java <reply@reply.git.luolix.topmailto:reply@reply.github.com>
Date: Thursday, April 28, 2016 at 5:12 AM
To: influxdata/influxdb-java <influxdb-java@noreply.git.luolix.topmailto:influxdb-java@noreply.github.com>
Cc: Heorhiy Kozlov <george.kozlov@lexalytics.commailto:george.kozlov@lexalytics.com>, Mention <mention@noreply.git.luolix.topmailto:mention@noreply.github.com>
Subject: Re: [influxdata/influxdb-java] Why the library always passes timestamp to the server? (#154)

I've implemented a tentative solution in the MR #168#168

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/154#issuecomment-215288011

[fmachado]

Fixed by #350 .