Add maxResultBuffer property #1431

sawkoj · 2020-09-17T11:16:58Z

Implementation of new property - maxResultBuffer. Enables max bytes read during reading result set.
Enable declare in two styles:
• as size of bytes (i.e. 100, 150M, 300K, 400G);
• as percent of max heap memory (i.e. 10p, 15pct, 20percent).

It also validates if defined size isn't bigger than 90% of max heap memory. If so, it limits max bytes read to 90% of memory heap. If maxResultBuffer property isn't provided, driver work isn't changed.

…fer with Tests

…ssage, changed order of MaxResultBufferPropertyDescription Exception,

…ace in ISQLServerDataSource

… use of StringUtils in MaxResultBufferParser, replaced substring() with endsWith() and charAt(), extracted errorMessage to variable, fixed format of multipliers, added few test cases to MaxResultBufferParserTest, updated setting maxResultBuffer to activeConnectionProperties in SQLServerConnection

…urce, corrected formatting in SQLServerConnection, extracted switch expression to method, reformatted code

ghost · 2020-09-17T11:17:13Z

All CLA requirements met.

rene-ye · 2020-10-01T17:46:21Z

Hi @sawkoj, can you give some points regarding why this feature is necessary? The JDBC Statement APIs provide similar "memory" control options in setFetchSize and setMaxFieldSize. Although this doesn't directly allow you to control memory, the intentions are the same. Can you give a use case where your proposed feature would be significantly better than the existing JDBC apis?

peterbae · 2020-10-01T19:08:41Z

Thanks for the contribution @sawkoj, sorry for the delay in response. I see that this change allows the user to set a limit to how much data can be read from a resultset, but in what situation will this change be useful? So far, users haven't asked for such feature because they can choose to stream the data instead of storing the entire resultset in memory, which I think is more flexible than the driver throwing an error if the amount of data read goes over the limit specified in this buffer.

sawkoj · 2020-10-02T13:42:34Z

Hi @rene-ye @peterbae , a purpose of the added property is to protect user from Out Of Memory Exception. Properties for “memory” control, which you have mentioned, won’t be enough in some cases. The most common case, I can think of, is when we have 2 queries which produce different result sets. Let’s call them A and B. A is large, contains thousands of rows, B’s size doesn’t matter. Upon executing A’s query, only first packet of A is kept in memory. We don’t iterate or do anything with A. Next, we call B query, to get B result set. In this moment, all remaining packets of A’s result set are being read to memory with first packet of B. Here OOM can occur and setting setFetchSize or setMaxFieldSize can prevent that.

Setting these limits can be very inconvenient though. Setting fetchSize changes amount of packets that need to be send to read whole resultSet. It could be a way of controlling of memory usage, but this approach is very error prone (you have to calculate the number of bytes for single row from a given query). If any column in query/table changes, you have to recompute number of rows to fetch if you want to handle memory using setFetchSize. The similar issue applies for setting maxFieldSize. You depend memory control on size of a single column, on change of query/table, you have to take that in consideration in code.

By using maxResultBuffer you just put amount of memory that you want to be used and don’t worry about having OOM and stopping whole application.

ebrandsberg · 2020-10-02T14:33:45Z

Adding to this. This work is being done as part of a larger project at Heimdall Data, which provides a proxy for a variety of database types. We are working to bring consistent controls over memory usage across multiple driver types. This feature has already been included in the Postgres JDBC driver (pgjdbc/pgjdbc#1657), as well as a fix submitted to the Oracle MySQL jdbc driver (https://bugs.mysql.com/bug.php?id=100415), so our next target was the SQL Server driver. To completely understand the need, don't think about one connection, but when you have thousands of connections active, and you need to be able to ensure there is enough memory for all of them. @sawkoj explained one very specific case, but there are other cases where (I believe) adaptive fetching doesn't work, in particular if retrieving large result sets from tables with clustered columnstore indexes. Cursors are not allowed in this case, and can trigger oom problems.

rene-ye · 2020-10-29T17:06:43Z

Hi @sawkoj, I've been testing the PR and there is currently some undesirable behavior with the new feature. There are currently some scenarios where if the maxResultBuffer property was set to "-1", the driver will throw an error on connect. New properties should be introduced with a default value where it won't impact the rest of the driver when either not specified or off. The error message also seems to need to work, currently I'm getting:

com.microsoft.sqlserver.jdbc.SQLServerException: Invalid syntax: [Ljava.lang.Object;@628190ec in maxResultBuffer parameter at com.microsoft.sqlserver.jdbc.MaxResultBufferParser.throwNewInvalidMaxResultBufferParameterException(MaxResultBufferParser.java:120)

The above error is easily reproducible if you run our unit tests provided in the repo, if you have any trouble setting it up, we can help with that. Some of the tests require special setup, and it's ok if you can't run them on your end, but the CIs should pass.

rene-ye · 2020-10-29T17:20:06Z

src/main/java/com/microsoft/sqlserver/jdbc/MaxResultBufferParser.java

+        long number = -1;
+
+        // check for null values and empty String "", if so return -1
+        if (StringUtils.isEmpty(input)) {


Temporary solution would be to change this line to

if (StringUtils.isEmpty(input) || "-1".equalsIgnoreCase(input)) {

But we should ask whether it makes sense to allow users to even input something like -1 at all. -1 is also used as a default value by XAConnection at times.

rene-ye · 2020-10-29T17:41:05Z

src/test/java/com/microsoft/sqlserver/testframework/AbstractTest.java

@@ -355,6 +355,8 @@ protected static ISQLServerDataSource updateDataSource(String connectionString,
                            break;
                        case Constants.SEND_TEMPORAL_DATATYPES_AS_STRING_FOR_BULK_COPY:
                            ds.setSendTemporalDataTypesAsStringForBulkCopy(Boolean.parseBoolean(value));


This needs a break at the end of the case statement or it'll overflow into the next one.

rene-ye · 2020-10-29T17:41:18Z

src/test/java/com/microsoft/sqlserver/testframework/AbstractTest.java

@@ -355,6 +355,8 @@ protected static ISQLServerDataSource updateDataSource(String connectionString,
                            break;
                        case Constants.SEND_TEMPORAL_DATATYPES_AS_STRING_FOR_BULK_COPY:
                            ds.setSendTemporalDataTypesAsStringForBulkCopy(Boolean.parseBoolean(value));
+                        case Constants.MAX_RESULT_BUFFER:
+                            ds.setMaxResultBuffer(value);


Add a break to the end of the this so the above bug doesn't happen again.

…e as expected

…Parser, added check for negative values supplied as maxResultBuffer parameter, they have their own exception, changed default value of MaxResultBuffer to "-1", addressed bad format of InvalidSyntaxException, added corresponding tests to MaxResultBufferParserTest.java

sawkoj · 2020-11-04T12:27:49Z

Hi @rene-ye, thank you for your feedback. According to your suggestions, I changed the default value of maxResultBuffer property to "-1", so from now it should be parsed correctly whether value for the property is specified or not. Also, I fixed the error message (I was unnecessarily packaging it as Array). I refactored the parsing logic, so from now it's gonna throw an error upon providing negative values, tests for negative values had been added as well. Of course I didn't forget to add missing break statements in AbstractTest.java.

ulvii · 2020-11-06T22:28:19Z

src/main/java/com/microsoft/sqlserver/jdbc/ICounter.java

+/**
+ * Interface for MaxResultBufferCounter
+ */
+public interface ICounter {


Please make this interface package-private.

ulvii · 2020-11-06T22:37:05Z

src/main/java/com/microsoft/sqlserver/jdbc/IOBuffer.java

+    //Counter reference, so maxResultBuffer property can by acknowledged
+    private ICounter counter;
+
+    public ICounter getCounter() {


Package-private please.

ulvii · 2020-11-06T22:41:22Z

src/main/java/com/microsoft/sqlserver/jdbc/IOBuffer.java

+        return counter;
+    }
+
+    public void createCounter(ICounter previousCounter, Properties activeConnectionProperties) {


package-private

ulvii · 2020-11-06T23:05:35Z

src/test/java/com/microsoft/sqlserver/jdbc/MaxResultBufferTest.java

+    }
+
+    private static void setMaxResultBuffer(String maxResultBuffer) {
+        connectionString = TestUtils.addOrOverrideProperty(connectionString, "maxResultBuffer", maxResultBuffer);


Because connectionString is defined in AbstractTest and is static, this line will change the connection string for all the tests. Please use a local connection string .

ulvii · 2020-11-06T23:29:53Z

src/test/java/com/microsoft/sqlserver/jdbc/MaxResultBufferTest.java

@@ -0,0 +1,280 @@
+/*


Please add more test for different possible values of the connection property, both valid and invalid cases.

…Counter and createCounter methods in IOBuffer.java

…al copy of connectionString (named localConnectionString), so it won't affect connectionString for other tests

…dateMaxResultBuffer method in MaxResultBufferParser.java accordingly

…ated comments, added more test cases

sawkoj · 2020-11-10T15:04:23Z

Hi @ulvii, thank you for the feedback. According to your suggestions, I changed access modifiers in ICounter interface and IOBuffer class methods getCounter(), createCounter() as well. In MaxResultBufferTest I added an another private static field, which is used as local connection String. The original connection string is unaffected. Also I added more test cases to MaxResultBufferParserTest (valid and invalid ones). MaxResultBufferTest class was refactored too, I parameterized existing tests and I added different MaxResultBuffer property values to illustrate the change of behavior.

peterbae · 2020-11-23T22:52:39Z

The PR looks good to me. Once the other team members have reviewed & approved, we will re-run the CI tests and merge the PR.

ulvii · 2020-11-24T21:02:29Z

src/main/java/com/microsoft/sqlserver/jdbc/MaxResultBufferCounter.java

+
+    private void checkForMaxResultBufferOverflow(long number) throws SQLServerException {
+        if (number > maxResultBuffer) {
+            logger.log(Level.WARNING, "MaxResultBuffer exceeded: {0}. Property was set to {1}.",


Please change to :

if (logger.isLoggable(Level.SEVERE)) { logger.log(Level.SEVERE, "MaxResultBuffer exceeded: {0}. Property was set to {1}.", new Object[] {number, maxResultBuffer}); }

ulvii · 2020-11-24T21:04:09Z

src/main/java/com/microsoft/sqlserver/jdbc/MaxResultBufferParser.java

+            try {
+                number = Long.parseLong(input);
+            } catch (NumberFormatException e) {
+                logger.log(Level.INFO, ERROR_MESSAGE, new Object[] {input});


if (logger.isLoggable(Level.SEVERE)) { logger.log(Level.SEVERE, ERROR_MESSAGE, new Object[] {input}); }

ulvii · 2020-11-24T21:04:54Z

src/main/java/com/microsoft/sqlserver/jdbc/MaxResultBufferParser.java

+                    try {
+                        number = Long.parseLong(numberString);
+                    } catch (NumberFormatException e) {
+                        logger.log(Level.INFO, ERROR_MESSAGE, new Object[] {input});


if (logger.isLoggable(Level.SEVERE)) { logger.log(Level.SEVERE, ERROR_MESSAGE, new Object[] {input}); }

ulvii · 2020-11-24T21:05:13Z

src/main/java/com/microsoft/sqlserver/jdbc/MaxResultBufferParser.java

+            try {
+                number = Long.parseLong(numberString);
+            } catch (NumberFormatException e) {
+                logger.log(Level.INFO, ERROR_MESSAGE, new Object[] {input});


if (logger.isLoggable(Level.SEVERE)) { logger.log(Level.SEVERE, ERROR_MESSAGE, new Object[] {input}); }

ulvii · 2020-11-24T21:05:31Z

src/main/java/com/microsoft/sqlserver/jdbc/MaxResultBufferParser.java

+                multiplier = 1_000_000_000_000L;
+                break;
+            default:
+                logger.log(Level.INFO, ERROR_MESSAGE, new Object[] {input});


if (logger.isLoggable(Level.SEVERE)) { logger.log(Level.SEVERE, ERROR_MESSAGE, new Object[] {input}); }

ulvii · 2020-11-24T21:14:37Z

src/main/java/com/microsoft/sqlserver/jdbc/SQLServerResource.java

-            {"R_TokenRequireUrl", "Token credentials require a URL using the HTTPS protocol scheme."},};
-};
+            {"R_TokenRequireUrl", "Token credentials require a URL using the HTTPS protocol scheme."},
+            {"R_maxResultBufferPropertyDescription",


R_maxResultBufferPropertyDescription should describe what maxResultBuffer connection property does, please check R_useFmtOnlyPropertyDescription for reference. I'd suggest to create another String resource for MaxResultBuffer property exceeded: {0}. MaxResultBuffer was set to: {1}. and use R_maxResultBufferPropertyDescription for the actual description of the connection property.

…rParser, added description to R_maxResultBufferPropertyDescription in SQLServerResource, added another string resource (R_maxResultBufferPropertyExceeded) in SQLServerResource

sawkoj · 2020-11-25T11:41:28Z

Hi @peterbae and @ulvii, thanks for feedback. Loggers were adjusted in MaxResultBufferParser and in MaxResultBufferCounter. I created additional resource: R_maxResultBufferPropertyExceeded when MaxResultBuffer is exceeded. Description of R_maxResultBufferPropertyDescription is updated too.

sawkoj added 9 commits September 17, 2020 09:26

Added MaxResultBuffer property and added methods to handle it in driver

af25295

Added Counter Interface with basic implementation, added MaxResultBuf…

363ed65

…fer with Tests

Added correct Project's Formatting and loggers

ca8223a

Fixed bug with MaxResultBuffer parsing, Tweak corresponding Test class

e417451

Implemented Counter's reset and increase methods

33f1791

ResultSet tests added, added required Test Constant and Test Error Me…

e9db643

…ssage, changed order of MaxResultBufferPropertyDescription Exception,

Fixed few misspellings in MaxResultBufferTest and reverted deleted sp…

460a829

…ace in ISQLServerDataSource

Added error message for maxResultBufferInvalidSyntax in SQLServerReso…

290f031

…urce, corrected formatting in SQLServerConnection, extracted switch expression to method, reformatted code

sawkoj changed the title ~~Max result buffer~~ Add maxResultBuffer property Sep 17, 2020

ulvii added this to the 9.1.0 milestone Sep 28, 2020

ulvii removed this from the 9.1.0 milestone Oct 13, 2020

Merge branch 'dev' into maxResultBuffer

306f5ad

rene-ye reviewed Oct 29, 2020

View reviewed changes

sawkoj added 2 commits November 4, 2020 12:29

Added break statements to the AbstractTest.java, so cases would behav…

5329a2b

…e as expected

ulvii reviewed Nov 6, 2020

View reviewed changes

sawkoj added 4 commits November 9, 2020 14:08

Changed modifier public to package-private of ICounter interface, get…

3a86682

…Counter and createCounter methods in IOBuffer.java

Replaced use of connectionString in MaxResultBufferTest.java with loc…

0efd5dc

…al copy of connectionString (named localConnectionString), so it won't affect connectionString for other tests

Added more test cases to MaxResultBufferParserTest.java, updated vali…

b4ff616

…dateMaxResultBuffer method in MaxResultBufferParser.java accordingly

Refactored tests in MaxResultBufferTest.java, parameterized them, upd…

989d362

…ated comments, added more test cases

Merge branch 'dev' into maxResultBuffer

8f2336d

peterbae previously approved these changes Nov 23, 2020

View reviewed changes

ulvii reviewed Nov 24, 2020

View reviewed changes

Changed logger levels in MaxResultBufferCounter and in MaxResultBuffe…

2e5a03e

…rParser, added description to R_maxResultBufferPropertyDescription in SQLServerResource, added another string resource (R_maxResultBufferPropertyExceeded) in SQLServerResource

sawkoj dismissed peterbae’s stale review via 2e5a03e November 25, 2020 11:15

ulvii approved these changes Nov 26, 2020

View reviewed changes

ulvii added this to the 9.1.1 milestone Nov 27, 2020

lilgreenbird approved these changes Nov 27, 2020

View reviewed changes

rene-ye approved these changes Nov 27, 2020

View reviewed changes

peterbae approved these changes Nov 27, 2020

View reviewed changes

ulvii merged commit d990974 into microsoft:dev Nov 27, 2020

lilgreenbird added the Public API Changes in Public API label Dec 12, 2020

artyomdmitriev mentioned this pull request Oct 18, 2022

[QUESTION] ExceededMaxResultBufferException on a big transaction with maxResultBuffer property applied #1934

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add maxResultBuffer property #1431

Add maxResultBuffer property #1431

sawkoj commented Sep 17, 2020

ghost commented Sep 17, 2020 •

edited by ghost

Loading

rene-ye commented Oct 1, 2020 •

edited

Loading

peterbae commented Oct 1, 2020

sawkoj commented Oct 2, 2020

ebrandsberg commented Oct 2, 2020 •

edited

Loading

rene-ye commented Oct 29, 2020 •

edited

Loading

rene-ye Oct 29, 2020 •

edited

Loading

rene-ye Oct 29, 2020 •

edited

Loading

rene-ye Oct 29, 2020 •

edited

Loading

sawkoj commented Nov 4, 2020

ulvii Nov 6, 2020 •

edited

Loading

ulvii Nov 6, 2020

ulvii Nov 6, 2020

ulvii Nov 6, 2020

ulvii Nov 6, 2020

sawkoj commented Nov 10, 2020

peterbae commented Nov 23, 2020

ulvii Nov 24, 2020

ulvii Nov 24, 2020

ulvii Nov 24, 2020

ulvii Nov 24, 2020

ulvii Nov 24, 2020

ulvii Nov 24, 2020

sawkoj commented Nov 25, 2020

Add maxResultBuffer property #1431

Add maxResultBuffer property #1431

Conversation

sawkoj commented Sep 17, 2020

ghost commented Sep 17, 2020 • edited by ghost Loading

rene-ye commented Oct 1, 2020 • edited Loading

peterbae commented Oct 1, 2020

sawkoj commented Oct 2, 2020

ebrandsberg commented Oct 2, 2020 • edited Loading

rene-ye commented Oct 29, 2020 • edited Loading

rene-ye Oct 29, 2020 • edited Loading

Choose a reason for hiding this comment

rene-ye Oct 29, 2020 • edited Loading

Choose a reason for hiding this comment

rene-ye Oct 29, 2020 • edited Loading

Choose a reason for hiding this comment

sawkoj commented Nov 4, 2020

ulvii Nov 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sawkoj commented Nov 10, 2020

peterbae commented Nov 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sawkoj commented Nov 25, 2020

ghost commented Sep 17, 2020 •

edited by ghost

Loading

rene-ye commented Oct 1, 2020 •

edited

Loading

ebrandsberg commented Oct 2, 2020 •

edited

Loading

rene-ye commented Oct 29, 2020 •

edited

Loading

rene-ye Oct 29, 2020 •

edited

Loading

rene-ye Oct 29, 2020 •

edited

Loading

rene-ye Oct 29, 2020 •

edited

Loading

ulvii Nov 6, 2020 •

edited

Loading