Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-44783][SQL][TESTS] Checks arrays as named and positional parameters #42470

Closed
wants to merge 2 commits into from
Closed

[SPARK-44783][SQL][TESTS] Checks arrays as named and positional parameters #42470

wants to merge 2 commits into from

Conversation

MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Aug 12, 2023

What changes were proposed in this pull request?

In the PR, I propose to add new test which checks arrays as named and positional parameters.

Why are the changes needed?

To improve test coverage.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

By running the modified test suite:

$ build/sbt "test:testOnly *ParametersSuite"

@github-actions github-actions bot added the SQL label Aug 12, 2023
@MaxGekk MaxGekk changed the title [WIP][SPARK-44783][SQL][TESTS] Checks arrays as named and positional parameters [SPARK-44783][SQL][TESTS] Checks arrays as named and positional parameters Aug 12, 2023
@MaxGekk MaxGekk marked this pull request as ready for review August 12, 2023 15:23
@yaooqinn
Copy link
Member

thanks, merged to master

@yaooqinn yaooqinn closed this in 0fc85a2 Aug 14, 2023
@yaooqinn yaooqinn mentioned this pull request Aug 14, 2023
Closed
HyukjinKwon
HyukjinKwon reviewed Aug 14, 2023
View reviewed changes
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM2

yaooqinn added a commit that referenced this pull request Aug 15, 2023
@yaooqinn
[SPARK-44802][INFRA] Token based ASF JIRA authentication
c36d54a 
### What changes were proposed in this pull request?

This PR add a env JIRA_ACCESS_TOKEN for merge script to enable token auth

### Why are the changes needed?

Tokens are more secure and easily revoked or expired.

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

1. locally tested

```python
>>> JIRA_ACCESS_TOKEN is not None
True
>>> asf_jira.issue("SPARK-44801")
<JIRA Issue: key='SPARK-44801', id='13547185'>
>>> if JIRA_ACCESS_TOKEN is not None:
...     asf_jira = jira.client.JIRA(jira_server, token_auth=JIRA_ACCESS_TOKEN)
... else:
...     asf_jira = jira.client.JIRA(
...         jira_server, basic_auth=(JIRA_USERNAME, JIRA_PASSWORD)
...     )
...
>>> asf_jira.issue("SPARK-44801")
<JIRA Issue: key='SPARK-44801', id='13547185'>
```

2. merged #42470 with this

Closes #42484 from yaooqinn/SPARK-44802.

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
valentinp17 pushed a commit to valentinp17/spark that referenced this pull request Aug 24, 2023
@MaxGekk
[SPARK-44783][SQL][TESTS] Checks arrays as named and positional param…
702c1a7 
…eters

### What changes were proposed in this pull request?
In the PR, I propose to add new test which checks arrays as named and positional parameters.

### Why are the changes needed?
To improve test coverage.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
By running the modified test suite:
```
$ build/sbt "test:testOnly *ParametersSuite"
```

Closes apache#42470 from MaxGekk/sql-parameterized-by-array.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
valentinp17 pushed a commit to valentinp17/spark that referenced this pull request Aug 24, 2023
@yaooqinn
[SPARK-44802][INFRA] Token based ASF JIRA authentication
5db7895 
### What changes were proposed in this pull request?

This PR add a env JIRA_ACCESS_TOKEN for merge script to enable token auth

### Why are the changes needed?

Tokens are more secure and easily revoked or expired.

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

1. locally tested

```python
>>> JIRA_ACCESS_TOKEN is not None
True
>>> asf_jira.issue("SPARK-44801")
<JIRA Issue: key='SPARK-44801', id='13547185'>
>>> if JIRA_ACCESS_TOKEN is not None:
...     asf_jira = jira.client.JIRA(jira_server, token_auth=JIRA_ACCESS_TOKEN)
... else:
...     asf_jira = jira.client.JIRA(
...         jira_server, basic_auth=(JIRA_USERNAME, JIRA_PASSWORD)
...     )
...
>>> asf_jira.issue("SPARK-44801")
<JIRA Issue: key='SPARK-44801', id='13547185'>
```

2. merged apache#42470 with this

Closes apache#42484 from yaooqinn/SPARK-44802.

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
@MaxGekk MaxGekk mentioned this pull request Sep 19, 2023
Closed
MaxGekk added a commit that referenced this pull request Sep 20, 2023
@MaxGekk
[SPARK-45224][PYTHON] Add examples w/ map and array as parameters of …
c89221b 
…`sql()`

### What changes were proposed in this pull request?
In the PR, I propose to add a few more examples for the `sql()` method in PySpark API with array and map parameters.

### Why are the changes needed?
To inform users about recent changes introduced by #42752 and #42470, and check the changes work actually.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
By running new examples:
```
$ python/run-tests --parallelism=1 --testnames 'pyspark.sql.session SparkSession.sql'
```

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #42996 from MaxGekk/map-sql-parameterized-python-connect.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
ragnarok56 pushed a commit to ragnarok56/spark that referenced this pull request Mar 2, 2024
@MaxGekk @ragnarok56
[SPARK-44783][SQL][TESTS] Checks arrays as named and positional param…
b32bd3f 
…eters

### What changes were proposed in this pull request?
In the PR, I propose to add new test which checks arrays as named and positional parameters.

### Why are the changes needed?
To improve test coverage.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
By running the modified test suite:
```
$ build/sbt "test:testOnly *ParametersSuite"
```

Closes apache#42470 from MaxGekk/sql-parameterized-by-array.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
ragnarok56 pushed a commit to ragnarok56/spark that referenced this pull request Mar 2, 2024
@yaooqinn @ragnarok56
[SPARK-44802][INFRA] Token based ASF JIRA authentication
f7b31c5 
### What changes were proposed in this pull request?

This PR add a env JIRA_ACCESS_TOKEN for merge script to enable token auth

### Why are the changes needed?

Tokens are more secure and easily revoked or expired.

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

1. locally tested

```python
>>> JIRA_ACCESS_TOKEN is not None
True
>>> asf_jira.issue("SPARK-44801")
<JIRA Issue: key='SPARK-44801', id='13547185'>
>>> if JIRA_ACCESS_TOKEN is not None:
...     asf_jira = jira.client.JIRA(jira_server, token_auth=JIRA_ACCESS_TOKEN)
... else:
...     asf_jira = jira.client.JIRA(
...         jira_server, basic_auth=(JIRA_USERNAME, JIRA_PASSWORD)
...     )
...
>>> asf_jira.issue("SPARK-44801")
<JIRA Issue: key='SPARK-44801', id='13547185'>
```

2. merged apache#42470 with this

Closes apache#42484 from yaooqinn/SPARK-44802.

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon left review comments

@cloud-fan cloud-fan cloud-fan approved these changes

@yaooqinn yaooqinn yaooqinn approved these changes

Assignees
No one assigned
Labels
SQL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@MaxGekk @yaooqinn @cloud-fan @HyukjinKwon