MSSQLS TPCH Add Flag to Partition Orders and Lineitems #647
Conversation
Added a flag that allows the Order and Lineitem tables to be partitioned. Each partition spans 1 week. The parition flag is set to false by default and must be enabled. Column Store and Row store are allowed to be partitioned. When paritioning is enabled. Certain primary and foreign key definitions defined in HammerDB are no longer valid. HammerDB will still create those keys/constraints when the Orders and Lineitem tables are not partitioned.` For columnstore indices, the is an initial degredation of performance, however if the user calls Alter Index ... Reoganize or leaves the server running after the initial load for approximately 48 hours, the performance of power runs improves by 2x when compared to column store indices with no partitioning.
There was a problem hiding this comment.
Have tested multiple combinations of build and test and all work apart from the following scenario where the refresh function is enabled in the driver script when the partitioning option has been selected. In this scenario both types of build error with a constraint doesn't exist error. This is when in the refresh function the following is called to allow the refresh function to insert into lineitem first before orders.
$odbc evaldirect "ALTER TABLE LINEITEM NOCHECK CONSTRAINT LINEITEM_ORDER_FK"
The error is the same as follows:
partition orders and lineitem with no clustered columnstore
Error in Virtual User 1: Error: [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Constraint 'LINEITEM_ORDER_FK' does not exist.
[Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Could not enable or disable the constraint. See previous errors.
(executing the statement)
partition orders and lineitem with clustered columnstore
Error in Virtual User 1: Error: [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Constraint 'LINEITEM_ORDER_FK' does not exist.
[Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Could not enable or disable the constraint. See previous errors.
(executing the statement)
The fix could be to check if the constraint exists before disabling or re-enabling - or to add a catch statement and ignore failure if the error message from the catch says that the constraint doesn't exist.
When the refresh was being used in a run, a constraint that no longer existed was being altered at several stages. Update refresh logic to bypass altering this index when it does not exist.
|
Updated PR to handle the issue found. |
There was a problem hiding this comment.
This all works and OK to accept the changes. Just to note in testing on a row store of 10WH the old sales refresh when running a power test (set VU to 1 and enable refresh function) completed in 185 secs with partitioning enabled and no foreign key compared to 0.967 for the default schema. The column store was also taking much longer on old sales refresh. Therefore this is something to be noted in the documentation when partitioning is used that it is known that the refresh function takes much longer.
Added a flag that allows the Order and Lineitem tables to be partitioned. Each partition spans 1 week. The parition flag is set to false by default and must be enabled.
Column Store and Row store are allowed to be partitioned.
When paritioning is enabled. Certain primary and foreign key definitions defined in HammerDB are no longer valid. HammerDB will still create those keys/constraints when the Orders and Lineitem tables are not partitioned.`
For columnstore indices, the is an initial degredation of performance, however if the user calls Alter Index ... Reoganize or leaves the server running after the initial load for approximately 48 hours, the performance of power runs improves by 2x when compared to column store indices with no partitioning.