feat: `POST` query variants serializes `'q'` parameter into HTTP body #765

bednar · 2021-08-27T08:52:10Z

Proposed Changes

The query parameter for post variants of query is serialized into HTTP body.

It is usefull when you hit the nginx limit for Request-URI length, eq. HTTP error: 414 Request-URI Too Large.

Following query variants also respects the requiresPost settings:

QueryResult query(Query, TimeUnit)
void query(Query, int, BiConsumer<Cancellable, QueryResult>, Runnable, Consumer<Throwable>)

Checklist

CHANGELOG.md updated
Rebased/mergeable
A test has been added if appropriate
compile-and-test.sh completes successfully
Commit messages are in semantic format
Sign CLA (if not already signed)

majst01 · 2021-08-27T09:34:30Z

Can you update the CI Test matrix with influxdb-2 as well ?

src/main/java/org/influxdb/impl/InfluxDBImpl.java

bednar · 2021-08-27T13:53:39Z

Can you update the CI Test matrix with influxdb-2 as well ?

Yes I will do. It will be a little bit tricky but I will do my best.

codecov-commenter · 2021-08-30T08:54:53Z

Codecov Report

Merging #765 (0b036a8) into master (c0e60d1) will decrease coverage by 4.34%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #765      +/-   ##
============================================
- Coverage     58.45%   54.11%   -4.35%     
+ Complexity      414      383      -31     
============================================
  Files            70       70              
  Lines          2571     2574       +3     
  Branches        269      268       -1     
============================================
- Hits           1503     1393     -110     
- Misses         1002     1116     +114     
+ Partials         66       65       -1

Impacted Files	Coverage Δ
src/main/java/org/influxdb/impl/InfluxDBImpl.java	`75.39% <100.00%> (-7.11%)`	⬇️
.../influxdb/msgpack/MessagePackConverterFactory.java	`0.00% <0.00%> (-100.00%)`	⬇️
...uxdb/msgpack/MessagePackResponseBodyConverter.java	`0.00% <0.00%> (-100.00%)`	⬇️
...ava/org/influxdb/msgpack/QueryResultModelPath.java	`0.00% <0.00%> (-90.00%)`	⬇️
...ava/org/influxdb/msgpack/MessagePackTraverser.java	`0.00% <0.00%> (-87.24%)`	⬇️
src/main/java/org/influxdb/dto/Query.java	`65.00% <0.00%> (-2.50%)`	⬇️
src/main/java/org/influxdb/dto/Point.java	`92.97% <0.00%> (+0.82%)`	⬆️
...n/java/org/influxdb/impl/InfluxDBResultMapper.java	`89.36% <0.00%> (+2.83%)`	⬆️
...rc/main/java/org/influxdb/impl/InfluxDBMapper.java	`88.57% <0.00%> (+88.57%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c0e60d1...0b036a8. Read the comment docs.

bednar · 2021-08-30T14:14:52Z

@majst01, InfluxDB 2 compatibility endpoints requires Database and retention policy mapping for queries and writes. So there is no easy way how to use existing test cases with InfluxDB 2.

I've prepared separete test case for InfluxDB 2 - InfluxDB2Test

You can start this testcase by: INFLUXDB_VERSION=2.0 ./compile-and-test.sh

compile-and-test.sh

src/test/java/org/influxdb/InfluxDB2Test.java

src/main/java/org/influxdb/impl/InfluxDBImpl.java

bednar · 2021-09-09T08:16:51Z

@majst01, I had to change how is PR implemented:

All query variants respects requiresPost settings on Query
POST variants serializes query into HTTP body

Can you review this changes?

Thx

rhajek

LGTM

majst01

So, does this mean that influxdb-java is now compatible with influxdb v2 ? If yes this must be reflected in the Readme, currently it says:

Note: This library is for use with InfluxDB 1.x. For connecting to InfluxDB 2.x instances, please use the influxdb-client-java client.

Please adjust accordingly

bednar · 2021-09-10T07:19:43Z

@majst01

So, does this mean that influxdb-java is now compatible with influxdb v2 ?

The influxdb-java is able to use InfluxDB 1.x compatibility API, but this API supports only subset of v1 features, e.g. queries:

https://docs.influxdata.com/influxdb/v2.0/query-data/influxql/#influxql-support

I think that current text is correct, because the client doesn't supports v2 API.

majst01 · 2021-09-10T07:22:43Z

@majst01

So, does this mean that influxdb-java is now compatible with influxdb v2 ?

The influxdb-java is able to use InfluxDB 1.x compatibility API, but this API supports only subset of v1 features, e.g. queries:

https://docs.influxdata.com/influxdb/v2.0/query-data/influxql/#influxql-support

I think that current text is correct, because the client doesn't supports v2 API.

Understood, but even that must be clarified in the Readme, please add a short description with a link to the official documentation.

bednar · 2021-09-10T11:00:42Z

@majst01, note changed to Note: This library is for use with InfluxDB 1.x and 2.x compatibility API. For full supports of InfluxDB 2.x features, please use the influxdb-client-java client.

majst01 · 2021-09-10T12:38:10Z

@bednar merged, and now all builds fail ?? Could you please check why.

majst01 · 2021-09-13T05:30:34Z

@bednar it still does not compile anymore, please fix this, otherwise i have to revert this

bednar · 2021-09-13T05:36:28Z

@majst01 currently I'am fixing that issue, stay tuned.

The problem is caused by definition of GitHub PRs workflow:

All 1.x builds are running agains 1.8...

bednar · 2021-09-15T06:02:17Z

The PR with fix is prepared: #773

bentatham · 2022-05-20T20:41:08Z

With this change, I am now seeing failures in 2.22 that I didn't see before in 2.21.

missing required parameter "q"

Using InfluxDB 1.8.10.
I don't think I'm doing anything too fancy. Things like this:

BoundedParameterQuery.QueryBuilder
        .newQuery(queryString)
        .forDatabase(database)
        .bind("startTime", startTime)
        .bind("endTime", endTime)
        .create();

(which defaults to requiresPost=true)

Is this expected? What do I need to change?

bednar · 2022-05-23T09:09:09Z

Hi @bentatham,

can you share a little bit more about your code? How do you initialise the client? How do you run your query?

I just tried following code and everything works fine:

try (InfluxDB client = InfluxDBFactory.connect("http://localhost:8086")) {

  BoundParameterQuery query = QueryBuilder
		  .newQuery("SELECT * FROM cpu WHERE time >= $startTime AND time <= $endTime")
		  .forDatabase("my_database")
		  .bind("startTime", "2015-08-18T00:00:00Z")
		  .bind("endTime", "2015-08-18T00:30:00Z")
		  .create();

  QueryResult result = client.query(query);

  System.out.println("result = " + result);
}

Regards

bentatham · 2022-05-24T13:24:44Z

@bednar Thanks for getting back to me:

We are using influxDB behind vertx, but I don't think that should really matter unless something in the query object is massively not thread-safe and is consistently hitting a volatility issue or something really obscure.

Initialization:

influxDb = InfluxDBFactory.connect(
            configuration.getUrl().toString(), 
            configuration.getUser(), 
            configuration.getPassword(), 
            new OkHttpClient().newBuilder() 
              .connectTimeout(CONNECT_TIMEOUT_SECONDS_DEFAULT, TimeUnit.SECONDS) 
              .readTimeout(READ_TIMEOUT_SECONDS_DEFAULT, TimeUnit.SECONDS)
              .writeTimeout(WRITE_TIMEOUT_SECONDS_DEFAULT, TimeUnit.SECONDS) 
enableGzip();

Batching is not enabled.

And then invoked with:

influxDb.query(query, TimeUnit.NANOSECONDS)

Maybe something is different about the query that specifies units? (My unit tests still fail with removing that, but that might be because of the units themselves, so I'll keep digging here too.)

bednar · 2022-05-25T07:27:28Z

@bednar Thanks for getting back to me:

We are using influxDB behind vertx, but I don't think that should really matter unless something in the query object is massively not thread-safe and is consistently hitting a volatility issue or something really obscure.

So the vertx performs as a proxy? Is there a support for query in the HTTP body?

Initialization:

influxDb = InfluxDBFactory.connect(
            configuration.getUrl().toString(), 
            configuration.getUser(), 
            configuration.getPassword(), 
            new OkHttpClient().newBuilder() 
              .connectTimeout(CONNECT_TIMEOUT_SECONDS_DEFAULT, TimeUnit.SECONDS) 
              .readTimeout(READ_TIMEOUT_SECONDS_DEFAULT, TimeUnit.SECONDS)
              .writeTimeout(WRITE_TIMEOUT_SECONDS_DEFAULT, TimeUnit.SECONDS) 
enableGzip();

Batching is not enabled.

I tried it with a direct connection to InfluxDB and everything works fine.

And then invoked with:
influxDb.query(query, TimeUnit.NANOSECONDS)
Maybe something is different about the query that specifies units? (My unit tests still fail with removing that, but that might be because of the units themselves, so I'll keep digging here too.)

It doesn't matter.

mschwore · 2022-07-01T15:26:41Z

I'm seeing the same behavior as @bentatham

I submitted issue !848 describing the problem. It seems to be related (at least for me) to enableGzip being set to true.

sranka reviewed Aug 27, 2021

View reviewed changes

src/main/java/org/influxdb/impl/InfluxDBImpl.java Outdated Show resolved Hide resolved

bednar changed the title ~~feat: streaming query serializes 'q' parameter into HTTP body for InfluxDB 2~~ feat: streaming query serializes 'q' parameter into HTTP body Aug 27, 2021

bednar marked this pull request as ready for review August 30, 2021 14:15

bednar requested review from majst01 and rhajek August 30, 2021 14:15

majst01 reviewed Aug 31, 2021

View reviewed changes

compile-and-test.sh Outdated Show resolved Hide resolved

majst01 reviewed Aug 31, 2021

View reviewed changes

src/test/java/org/influxdb/InfluxDB2Test.java Outdated Show resolved Hide resolved

majst01 approved these changes Aug 31, 2021

View reviewed changes

bednar commented Aug 31, 2021

View reviewed changes

src/main/java/org/influxdb/impl/InfluxDBImpl.java Outdated Show resolved Hide resolved

bednar marked this pull request as draft September 9, 2021 07:34

bednar changed the title ~~feat: streaming query serializes 'q' parameter into HTTP body~~ feat: query parameter for POST variants is serialized into HTTP body Sep 9, 2021

feat: query parameter for POST variants is serialized into HTTP body

8e425fd

bednar force-pushed the feat/query-as-body-for-influxdb-v2 branch from 666ebb5 to 8e425fd Compare September 9, 2021 08:06

bednar changed the title ~~feat: query parameter for POST variants is serialized into HTTP body~~ feat: POST query variants serializes 'q' parameter into HTTP body Sep 9, 2021

docs: update CHANGELOG.md

608689c

bednar marked this pull request as ready for review September 9, 2021 08:16

bednar requested a review from majst01 September 9, 2021 08:16

rhajek approved these changes Sep 9, 2021

View reviewed changes

majst01 requested changes Sep 9, 2021

View reviewed changes

docs: add note about 2.x compatibility API

0b036a8

majst01 approved these changes Sep 10, 2021

View reviewed changes

majst01 merged commit d73aa32 into master Sep 10, 2021

majst01 deleted the feat/query-as-body-for-influxdb-v2 branch September 10, 2021 12:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `POST` query variants serializes `'q'` parameter into HTTP body #765

feat: `POST` query variants serializes `'q'` parameter into HTTP body #765

bednar commented Aug 27, 2021 •

edited

Loading

majst01 commented Aug 27, 2021

bednar commented Aug 27, 2021

codecov-commenter commented Aug 30, 2021 •

edited

Loading

bednar commented Aug 30, 2021

bednar commented Sep 9, 2021

rhajek left a comment

majst01 left a comment

bednar commented Sep 10, 2021

majst01 commented Sep 10, 2021

bednar commented Sep 10, 2021

majst01 commented Sep 10, 2021

majst01 commented Sep 13, 2021

bednar commented Sep 13, 2021

bednar commented Sep 15, 2021

bentatham commented May 20, 2022

bednar commented May 23, 2022

bentatham commented May 24, 2022

bednar commented May 25, 2022

mschwore commented Jul 1, 2022 •

edited

Loading

feat: POST query variants serializes 'q' parameter into HTTP body #765

feat: POST query variants serializes 'q' parameter into HTTP body #765

Conversation

bednar commented Aug 27, 2021 • edited Loading

Proposed Changes

Checklist

majst01 commented Aug 27, 2021

bednar commented Aug 27, 2021

codecov-commenter commented Aug 30, 2021 • edited Loading

Codecov Report

bednar commented Aug 30, 2021

bednar commented Sep 9, 2021

rhajek left a comment

Choose a reason for hiding this comment

majst01 left a comment

Choose a reason for hiding this comment

bednar commented Sep 10, 2021

majst01 commented Sep 10, 2021

bednar commented Sep 10, 2021

majst01 commented Sep 10, 2021

majst01 commented Sep 13, 2021

bednar commented Sep 13, 2021

bednar commented Sep 15, 2021

bentatham commented May 20, 2022

bednar commented May 23, 2022

bentatham commented May 24, 2022

bednar commented May 25, 2022

mschwore commented Jul 1, 2022 • edited Loading

feat: `POST` query variants serializes `'q'` parameter into HTTP body #765

feat: `POST` query variants serializes `'q'` parameter into HTTP body #765

bednar commented Aug 27, 2021 •

edited

Loading

codecov-commenter commented Aug 30, 2021 •

edited

Loading

mschwore commented Jul 1, 2022 •

edited

Loading