Modify End Use by Subcategory tables in SQL so they can be queried

[jmarrec]

Pull request overview

• Fixes Difficult to query End Uses By Subcategory tables in SQLite #7481
• There are two tables in TabularDataWithStrings that show end uses (either energy or demand) by subcategories. They are currently impossible to properly query using SQL

This Pull Request will properly fill the rowHead (forward fill the values of End Uses basically) before ouputing to SQL but after the HTML file is already written so we don't mess with this.

---

Before:

If you look at the HTML result, you see that rowHead is empty for display purposes:

This means the SQL one looks very similar, with empty "RowName" cells:

---

After: All RowNames are filled

It's still not terribly straightforward to get a specific entry for a subcategory end use, but it's doable for sure (and quite fast). I personally like this better than combining both End Use and End Use Subcat in the same RowName, or worse adding a new column (which would create way too many "NULLs").

SELECT * FROM TabularDataWithStrings
  WHERE TableName = "End Uses By Subcategory"
  AND ReportName = "AnnualBuildingUtilityPerformanceSummary"
  AND RowName = "Interior Lighting"
  AND (TabularDataIndex - (SELECT TabularDataIndex FROM TabularDataWithStrings
							  WHERE TableName = "End Uses By Subcategory"
							  AND ReportName = "AnnualBuildingUtilityPerformanceSummary"
							  AND ColumnName = "Subcategory"
							  AND RowName = "Interior Lighting"
							  AND Value = "GeneralLights"))
	  % (SELECT COUNT(Value) FROM TabularDataWithStrings
							  WHERE TableName = "End Uses By Subcategory"
							  AND ReportName = "AnnualBuildingUtilityPerformanceSummary"
							  AND ColumnName = "Subcategory") = 0

NOTE: ENHANCEMENTS MUST FOLLOW A SUBMISSION PROCESS INCLUDING A FEATURE PROPOSAL AND DESIGN DOCUMENT PRIOR TO SUBMITTING CODE

Pull Request Author

Add to this list or remove from it as applicable. This is a simple templated set of guidelines.

• Title of PR should be user-synopsis style (clearly understandable in a standalone changelog context)
• Label the PR with at least one of: Defect, Refactoring, NewFeature, Performance, and/or DoNoPublish
• Pull requests that impact EnergyPlus code must also include unit tests to cover enhancement or defect repair
• Author should provide a "walkthrough" of relevant code changes using a GitHub code review comment process
• If any diffs are expected, author must demonstrate they are justified using plots and descriptions
• If changes fix a defect, the fix should be demonstrated in plots and descriptions
• If any defect files are updated to a more recent version, upload new versions here or on DevSupport
• If IDD requires transition, transition source, rules, ExpandObjects, and IDFs must be updated, and add IDDChange label
• If structural output changes, add to output rules file and add OutputChange label
• If adding/removing any LaTeX docs or figures, update that document's CMakeLists file dependencies

Reviewer

This will not be exhaustively relevant to every PR.

• Perform a Code Review on GitHub
• If branch is behind develop, merge develop and build locally to check for side effects of the merge
• If defect, verify by running develop branch and reproducing defect, then running PR and reproducing fix
• If feature, test running new feature, try creative ways to break it
• CI status: all green or justified
• Check that performance is not impacted (CI Linux results include performance check)
• Run Unit Test(s) locally
• Check any new function arguments for performance impacts
• Verify IDF naming conventions and styles, memos and notes and defaults
• If new idf included, locally check the err file and other outputs

[jmarrec]

Check that the "RowName" isn't empty anymore

[jmarrec]

This query works in one step to query all entries for a specific end use subcategory.

[jmarrec]

This is the addition here, query for ONE fuel. And Return only "Value" not "*"

[shorowit]

@jmarrec Thank you! I'll be able to get rid of my super complicated, hacky code...

[nrel-bot-3]

@jmarrec @lgentile it has been 28 days since this pull request was last updated.

[Myoldmopar]

@shorowit have you tested this to confirm it fixes your issue? @kbenne I'd normally contact DMac with things like this to give the OS team a heads up. Can you advise who should be notified now?

[shorowit]

I have not tested it, only looked at @jmarrec's screenshots.

That said, now that I get fresh eyes on this... the PR is an incremental improvement that makes it slightly easier to query, but the query is still pretty ugly. It's fine to have this drop in, but I don't feel that the issue has been substantially addressed. The query to obtain this kind of information should not have to be so complicated.

Compare this PR's query:

SELECT * FROM TabularDataWithStrings
  WHERE TableName = "End Uses By Subcategory"
  AND ReportName = "AnnualBuildingUtilityPerformanceSummary"
  AND RowName = "Interior Lighting"
  AND (TabularDataIndex - (SELECT TabularDataIndex FROM TabularDataWithStrings
                              WHERE TableName = "End Uses By Subcategory"
                              AND ReportName = "AnnualBuildingUtilityPerformanceSummary"
                              AND ColumnName = "Subcategory"
                              AND RowName = "Interior Lighting"
                              AND Value = "GeneralLights"))
      % (SELECT COUNT(Value) FROM TabularDataWithStrings
                              WHERE TableName = "End Uses By Subcategory"
                              AND ReportName = "AnnualBuildingUtilityPerformanceSummary"
                              AND ColumnName = "Subcategory")

with the two queries I was envisioning as possibilities:

SELECT * FROM TabularDataWithStrings
  WHERE TableName = "End Uses By Subcategory"
  AND ReportName = "AnnualBuildingUtilityPerformanceSummary"
  AND RowName = "Interior Lighting"
  AND SubRowName = "GeneralLights"

or

SELECT * FROM TabularDataWithStrings
  WHERE TableName = "End Uses By Subcategory"
  AND ReportName = "AnnualBuildingUtilityPerformanceSummary"
  AND RowName = "Interior Lighting:GeneralLights"

[jmarrec]

Extracting the relevant comment from my original PR message:

It's still not terribly straightforward to get a specific entry for a subcategory end use, but it's doable for sure (and quite fast). I personally like this better than combining both End Use and End Use Subcat in the same RowName, or worse adding a new column (which would create way too many "NULLs").

The TabularDataWithStrings is a SQL heresy (note that I do understand why it is like that): it's a View that's aiming to be generic enough to store all kinds of data, and bends the rules of database design. This kind of problem wouldn't occur if this data was in its own SQL TABLE.

[mjwitte]

@jmarrec Would it be useful to duplicate the end-use information in a second well-designed table? Not necessarily as part of this PR.

[shorowit]

Personally, I don’t see why adding a new End Use SubCat column that is often null is so bad. It seems like the most generic and least obtrusive change. And it would be non-breaking.

Adding a separate table would be fine, but seems silly to query end use data one place and end use by subcategory data somewhere else.

My two cents.

[jmarrec]

I don't like adding 10,000 NULLS for what's really a corner case.

Ok, would this make you happier?

I deleted the "ColumnName==Subcategory" (should I leave it?), then for each RowName is "EndUse:Subcategory"

That'd make your query work:

[shorowit]

Not sure how others would feel, but I would be happy with this solution, yes.

[jmarrec]

@shorowit see updated solution. check the markdown, I included example queries and table outputs too

[shorowit]

I built this locally and confirmed that the results look correct and the data is now easy to query. @jmarrec, thanks so much for making this change.

[Myoldmopar]

This is really great. Verified functional improvement from @shorowit, unit test coverage, and documented output changes in the markdown. Thanks @jmarrec; merging this in.