Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strange behavior when passing SelectArg for orderByRaw(). #169

Closed
gogos-venge opened this issue Nov 11, 2019 · 8 comments
Closed

Strange behavior when passing SelectArg for orderByRaw(). #169

gogos-venge opened this issue Nov 11, 2019 · 8 comments

Comments

@gogos-venge
Copy link

gogos-venge commented Nov 11, 2019

I am re-posting this issue here on github, as there is already an open question in SO. I stumbled upon some strange behavior regarding orderByRaw. I was trying to model this query, which resembles a distinct:

SELECT `occupation` FROM `cards` GROUP BY `occupation` ORDER BY `occupation` IS NULL ASC.

In a populated database, this query should group by the occupation value, and order by the IS NULL value. IS NULL is either 1 or 0, which depends on whether the value is null or not correspondingly. Now if you order by IS NULL value ascending, then you ask SQL to return all the zeroes first, and all the ones in the end, literally giving a result of NULLS in the end.
In order to model the above query using ORMlite, I did this:

String column = "occupation"; //This comes from an unknown source
SelectArg selectArg = new SelectArg(SqlType.STRING, column);
qBuilder.selectColumns(column).groupBy(column).orderByRaw("? IS NULL ASC", selectArg);

Apparently one should expect the null values to be returned after all the non-null ones, but instead the query seems to be run as is:

SELECT * FROM `cards` GROUP BY `occupation` ORDER BY ? IS NULL ASC.

The above query is a perfectly valid one. I've never seen a logically valid usage of ? in this context but still it's syntactically correct, and therefore the driver just accepts it without errors, but the ordering is unaffected.
I created a JUnit test with an assert test in my github. You can check it here.

@j256
Copy link
Owner

j256 commented Dec 2, 2020

Sorry for the delay @gogos-venge . I've grabbed your unit test code but it doesn't fail for me. Should it?

@gogos-venge
Copy link
Author

It's impossible not to get this assert to fail:

expected:<[apple
orange
raspberry
null]> but was:<[null
apple
orange
raspberry]>

If this:
orderByRaw("? IS NULL ASC", new SelectArg(SqlType.STRING, "hello"));

Does not produce something like this:

... ORDER BY `hello` IS NULL ASC

Then this documentation is, in my opinion, misleading:

public QueryBuilder<T, ID> orderByRaw(String rawSql, com.j256.ormlite.stmt.ArgumentHolder... args)
Add raw SQL "ORDER BY" clause to the SQL query statement.

Params:
rawSqlThe raw SQL order by clause. This should not include the "ORDER BY".
argsOptional arguments that correspond to any ? specified in the rawSql. Each of the arguments must have the sql-type set.

Or, there's something really wrong with my perception

@j256
Copy link
Owner

j256 commented Dec 2, 2020

It's not supposed to generate ... ORDER BY hello IS NULL ASC. It's supposed to generate ... ORDER BY ? IS NULL ASC and then pass in an argument when preparing the query. I've improved the log output to show that there are arguments. The ? is then handled by the underlying JDBC code.

I tried to run your test but the assert does not fail. I see the same order in the list whether with SelectArg or the column hardcoded. I've imported your test and tweaked it a bit. See: #185

Does your test code still fail for you? If so, what version of ORMLite are you running and on what OS?

@j256 j256 changed the title Strange behavior when passing SelectArg for orderByRaw(String, SelectArg). Strange behavior when passing SelectArg for orderByRaw(). Dec 2, 2020
@georgios-tsoumas
Copy link

This is, really strange. It fails all the times for me, I'm 100% positive. Here let me append the output:

Testing started at 10:43 AM ...

> Task :compileJava
> Task :processResources NO-SOURCE
> Task :classes
> Task :compileTestJava
> Task :processTestResources NO-SOURCE
> Task :testClasses
> Task :test FAILED
2020-12-03 10:43:49,050 [DEBUG] DaoManager created dao for class class ORMlite_UnitTest.TestObject with reflection
2020-12-03 10:43:49,052 [INFO] TableUtils creating table 'testobject'
2020-12-03 10:43:49,158 [DEBUG] JdbcConnectionSource opened connection to jdbc:h2:mem:test got #1089999931
2020-12-03 10:43:49,167 [INFO] TableUtils executed create table statement changed 0 rows: CREATE TABLE `testobject` (`testColumn` VARCHAR(255) ) 
2020-12-03 10:43:49,170 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,171 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,171 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,172 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,172 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,172 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,173 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,173 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,174 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,174 [DEBUG] BaseMappedStatement insert data with statement 'INSERT INTO `testobject` (`testColumn` ) VALUES (?)' and 1 args, changed 1 rows
2020-12-03 10:43:49,176 [DEBUG] StatementBuilder built statement SELECT `testColumn` FROM `testobject` 
2020-12-03 10:43:49,182 [DEBUG] BaseMappedStatement prepared statement 'SELECT `testColumn` FROM `testobject` ' with 0 args
2020-12-03 10:43:49,194 [DEBUG] SelectIterator starting iterator @596646147 for 'SELECT `testColumn` FROM `testobject` '
2020-12-03 10:43:49,194 [DEBUG] SelectIterator closed iterator @596646147 after 10 rows
2020-12-03 10:43:49,195 [DEBUG] StatementExecutor query of 'SELECT `testColumn` FROM `testobject` ' returned 10 results
2020-12-03 10:43:49,195 [INFO] OrderByTest apple-null-orange-null-raspberry-null-apple-null-orange-null
2020-12-03 10:43:49,196 [DEBUG] StatementBuilder built statement SELECT `testColumn` FROM `testobject` GROUP BY `testColumn` ORDER BY ? IS NULL ASC 
2020-12-03 10:43:49,197 [DEBUG] BaseMappedStatement prepared statement 'SELECT `testColumn` FROM `testobject` GROUP BY `testColumn` ORDER BY ? IS NULL ASC ' with 1 args
2020-12-03 10:43:49,198 [DEBUG] SelectIterator starting iterator @314422829 for 'SELECT `testColumn` FROM `testobject` GROUP BY `testColumn` ORDER BY ? IS NULL ASC '
2020-12-03 10:43:49,199 [DEBUG] SelectIterator closed iterator @314422829 after 4 rows
2020-12-03 10:43:49,199 [DEBUG] StatementExecutor query of 'SELECT `testColumn` FROM `testobject` GROUP BY `testColumn` ORDER BY ? IS NULL ASC ' returned 4 results
2020-12-03 10:43:49,199 [DEBUG] StatementBuilder built statement SELECT `testColumn` FROM `testobject` GROUP BY `testColumn` ORDER BY `testColumn` IS NULL ASC 
2020-12-03 10:43:49,199 [DEBUG] BaseMappedStatement prepared statement 'SELECT `testColumn` FROM `testobject` GROUP BY `testColumn` ORDER BY `testColumn` IS NULL ASC ' with 0 args
2020-12-03 10:43:49,200 [DEBUG] SelectIterator starting iterator @2097756821 for 'SELECT `testColumn` FROM `testobject` GROUP BY `testColumn` ORDER BY `testColumn` IS NULL ASC '
2020-12-03 10:43:49,200 [DEBUG] SelectIterator closed iterator @2097756821 after 4 rows
2020-12-03 10:43:49,200 [DEBUG] StatementExecutor query of 'SELECT `testColumn` FROM `testobject` GROUP BY `testColumn` ORDER BY `testColumn` IS NULL ASC ' returned 4 results

expected:<[apple
orange
raspberry
null]> but was:<[null
apple
orange
raspberry]>
<Click to see difference>

org.junit.ComparisonFailure: expected:<[apple
orange
raspberry
null]> but was:<[null
apple
orange
raspberry]>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at ORMlite_UnitTest.OrderByTest.test(OrderByTest.java:93)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.base/java.lang.Thread.run(Thread.java:829)

ORMlite_UnitTest.OrderByTest > test FAILED
    org.junit.ComparisonFailure at OrderByTest.java:93
1 test completed, 1 failed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///home/<hidden>/Tools/ORMlite_JUnit/build/reports/tests/test/index.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 3s
3 actionable tasks: 3 executed

I'm using version 5.1 (please check it here in the build.gradle)
I tested both on Windows 10 and Pop OS! 20.10
The output is from a fresh Pop installation

@j256
Copy link
Owner

j256 commented Dec 3, 2020

Huh. Thanks for looking into it. This seems to be an issue with H2. Looks like they changed their behavior between version 1.2.128 that ORMLite is using and 1.4.200 that you are using. If I upgrade ORMLite to 1.4.200 my test fails as well.

In any case, this doesn't seem to be an ORMLite issue. It does mean that I should support the NULLS FIRST and LAST constructs however.

@j256 j256 closed this as completed Dec 3, 2020
@gogos-venge
Copy link
Author

Sorry I have to insist here. I didn't notice this problem because of the failing h2 driver, but because it actually happened in a production build which used sqlite-jdbc. I've updated the tests which now include one for h2, and one for sqlite. They both fail. I still use ORMlite 5.1 and now also use sqlite-jdbc driver '3.32.3.2' .
Please, pull or clone again the JUnit test project and see for yourself.
I can also confirm though, using 1.2.128 h2 driver, the h2 test passes.

@j256
Copy link
Owner

j256 commented Dec 5, 2020

You are going to insist what? That ORMLite should modify the order of the results returned by these databases? That would be a serious bug . This is not an ORMLite issue. ORMLite is translating the query into SQL appropriate for H2 and Sqlite. Your problem is with the underlying database not returning the results in the order you expect. If you did not use ORMLite but you did the same queries to H2 and Sqlite directly, you should see the same order. That make sense?

That said, I've added support for NULLS FIRST and LAST in order by. See: 4174d74

I can push a new release now if that would be helpful.

@gogos-venge
Copy link
Author

Thank you for looking into this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants