Feature upgrade to Spark 3.2.1 #111

ronanstokes-db · 2022-10-04T02:19:04Z

Proposed changes

Describe the big picture of your changes here to communicate to the maintainers.
If it fixes a bug or resolves a feature request, please provide a link to that issue.

Types of changes

What types of changes does your code introduce to dbx?
Put an x in the boxes that apply

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Change to tutorials, tests or examples
Non code change (readme, images or other non-code assets)
Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask. We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

Lint and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)
Any dependent changes have been merged and published in downstream modules
Submission does not reduce code coverage numbers
Submission does not increase alerts or messages from LGTM

Further comments

Based supported version on Databricks runtime 9.1 LTS or later. May impact users of earlier
runtime versions but instructions are included for using older version of the Databricks Data Generator in a notebook

ronanstokes-db · 2022-10-04T02:19:45Z

ready for review

codecov · 2022-10-04T02:32:27Z

Codecov Report

Merging #111 (e8eb833) into master (109707e) will increase coverage by 2.78%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #111      +/-   ##
==========================================
+ Coverage   84.11%   86.90%   +2.78%     
==========================================
  Files          21       21              
  Lines        2134     2161      +27     
  Branches      365      367       +2     
==========================================
+ Hits         1795     1878      +83     
+ Misses        242      183      -59     
- Partials       97      100       +3

Impacted Files	Coverage Δ
dbldatagen/_version.py	`100.00% <100.00%> (ø)`
dbldatagen/data_generator.py	`83.23% <100.00%> (+0.54%)`	⬆️
dbldatagen/datagen_constants.py	`100.00% <100.00%> (ø)`
dbldatagen/text_generators.py	`79.77% <0.00%> (+21.37%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

ronanstokes-db · 2022-10-04T06:28:43Z

I also needed to add new code coverage tests as the code coverage threshold was failing - the code coverage tools now seem to discount code coverage for code in _init.py or else code coverage threshold has been raised.

ronanstokes-db · 2022-10-05T08:52:21Z

Do not commit directly as other fixes need to be commited first

alexott

most things are primarily about README, etc.

CHANGELOG.md

CONTRIBUTING.md

README.md

dbldatagen/__init__.py

…atabrickslabs/dbldatagen into feature-upgrade-to-spark-3_1_2 merged upstream changes

nfx

please resolve git conflicts, otherwise lgtm

lgtm-com · 2022-11-23T00:50:16Z

This pull request introduces 1 alert and fixes 2 when merging d3ebcf2 into 109707e - view on LGTM.com

new alerts:

1 for Syntax error

fixed alerts:

2 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

lgtm-com · 2022-11-23T01:05:17Z

This pull request introduces 1 alert when merging 85478f8 into 109707e - view on LGTM.com

new alerts:

1 for Unreachable code

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

ronanstokes-db

conflicts resolved

ronanstokes-db · 2022-11-23T01:28:51Z

please resolve git conflicts, otherwise lgtm

All conflicts have been resolved

lgtm-com · 2022-11-23T01:34:54Z

This pull request introduces 1 alert when merging 3b04db4 into 109707e - view on LGTM.com

new alerts:

1 for Unreachable code

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

ronanstokes-db

requested changes made

nfx · 2022-11-24T13:01:27Z

dbldatagen/data_generator.py

+        self.sparkSession = sparkSession
+        if sparkSession is None:
+            raise DataGenError("""Spark session not initialized
+


Added line #L171 was not covered by tests

and this statement is unreachable :) lgtm.com is soo good at this.

and this statement is unreachable :) lgtm.com is soo good at this.

except the fix for this was checked in 20 hours earlier

for some reason lgtm is taking 1 day to update results - perhaps due to transition to new code scanning ?

nfx

Lgtm

ronanstokes-db added 3 commits October 3, 2022 17:51

work in progress

dbc2ad5

work in progress

163ed73

work in progress

70af189

ronanstokes-db added 7 commits October 3, 2022 19:35

changed spaces for PEP compatibility - minor no functionality change

7a3f604

changed library dependencies for build

a23017d

additional code coverage for version checks

03e751d

additional code coverage for version checks

1499900

Removed lines of code badge as lines of code server is down

3f192e8

additional test coverage for text generators

b8e53b8

spark version check

18b05be

reverted unchanged_files

53148f2

ronanstokes-db self-assigned this Oct 4, 2022

ronanstokes-db added documentation Improvements or additions to documentation code quality dependencies Pull requests that update a dependency file labels Oct 4, 2022

ronanstokes-db linked an issue Oct 4, 2022 that may be closed by this pull request

Add support for Spark 3.1.2 #91

Closed

ronanstokes-db requested review from alexott, GeekSheikh, nfx, Tagar and pohlposition October 4, 2022 08:51

ronanstokes-db marked this pull request as draft October 5, 2022 08:53

alexott reviewed Oct 5, 2022

View reviewed changes

ronanstokes-db and others added 3 commits October 5, 2022 15:31

Merge branch 'master' into feature-upgrade-to-spark-3_1_2

f40f26c

updates in response to reviews

31fb56f

Merge branch 'feature-upgrade-to-spark-3_1_2' of https://github.com/d…

ab61f2b

…atabrickslabs/dbldatagen into feature-upgrade-to-spark-3_1_2 merged upstream changes

ronanstokes-db and others added 2 commits October 7, 2022 07:47

converted version tests to pytest

3447200

Merge branch 'master' into feature-upgrade-to-spark-3_1_2

47b903d

ronanstokes-db changed the title ~~Feature upgrade to spark 3 1 2~~ Feature upgrade to Spark 3.2.1 Oct 21, 2022

nfx requested changes Nov 21, 2022

View reviewed changes

ronanstokes-db marked this pull request as ready for review November 23, 2022 00:28

merged changes from master

d3ebcf2

updated version due to change in baseline Python requirements

85478f8

updated version due to change in baseline Python requirements

3b04db4

ronanstokes-db commented Nov 23, 2022

View reviewed changes

ronanstokes-db requested a review from nfx November 23, 2022 01:28

ronanstokes-db commented Nov 23, 2022

View reviewed changes

ronanstokes-db enabled auto-merge (squash) November 23, 2022 18:29

ronanstokes-db added 2 commits November 23, 2022 11:16

updated version due to change in baseline Python requirements

44939aa

updated version due to change in baseline Python requirements

d7cdfb8

nfx requested changes Nov 24, 2022

View reviewed changes

updated version due to change in baseline Python requirements

e8eb833

nfx approved these changes Nov 28, 2022

View reviewed changes

ronanstokes-db merged commit 981a5a4 into master Nov 28, 2022

nfx deleted the feature-upgrade-to-spark-3_1_2 branch November 28, 2022 08:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature upgrade to Spark 3.2.1 #111

Feature upgrade to Spark 3.2.1 #111

ronanstokes-db commented Oct 4, 2022 •

edited

Loading

ronanstokes-db commented Oct 4, 2022 •

edited

Loading

codecov bot commented Oct 4, 2022 •

edited

Loading

ronanstokes-db commented Oct 4, 2022

ronanstokes-db commented Oct 5, 2022

alexott left a comment

nfx left a comment

lgtm-com bot commented Nov 23, 2022

lgtm-com bot commented Nov 23, 2022

ronanstokes-db left a comment

ronanstokes-db commented Nov 23, 2022

lgtm-com bot commented Nov 23, 2022

ronanstokes-db left a comment

nfx Nov 24, 2022

ronanstokes-db Nov 24, 2022

ronanstokes-db Nov 24, 2022

nfx left a comment

Feature upgrade to Spark 3.2.1 #111

Feature upgrade to Spark 3.2.1 #111

Conversation

ronanstokes-db commented Oct 4, 2022 • edited Loading

Proposed changes

Types of changes

Checklist

Further comments

ronanstokes-db commented Oct 4, 2022 • edited Loading

codecov bot commented Oct 4, 2022 • edited Loading

Codecov Report

ronanstokes-db commented Oct 4, 2022

ronanstokes-db commented Oct 5, 2022

alexott left a comment

Choose a reason for hiding this comment

nfx left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Nov 23, 2022

lgtm-com bot commented Nov 23, 2022

ronanstokes-db left a comment

Choose a reason for hiding this comment

ronanstokes-db commented Nov 23, 2022

lgtm-com bot commented Nov 23, 2022

ronanstokes-db left a comment

Choose a reason for hiding this comment

nfx Nov 24, 2022

Choose a reason for hiding this comment

ronanstokes-db Nov 24, 2022

Choose a reason for hiding this comment

ronanstokes-db Nov 24, 2022

Choose a reason for hiding this comment

nfx left a comment

Choose a reason for hiding this comment

ronanstokes-db commented Oct 4, 2022 •

edited

Loading

ronanstokes-db commented Oct 4, 2022 •

edited

Loading

codecov bot commented Oct 4, 2022 •

edited

Loading