Modifying astroquery/sdss package init file to use most current SDSS DR (as well as associated documentation) #2478

jsobeck · 2022-08-02T20:09:36Z

Hi, this PR closes #2365. The __init__.py file in the sdss package as well as some of the related documentation (in sdss.rst) has been slightly modified to reflect and use the latest SDSS data release (DR17). Some package tests have been run. Please let me know if I need to do anything further. Thank you.

pep8speaks · 2022-08-02T20:09:39Z

Hello @jsobeck! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-09-11 04:20:09 UTC

codecov · 2022-08-02T20:22:58Z

Codecov Report

Merging #2478 (5f60c09) into main (5c1eefc) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2478   +/-   ##
=======================================
  Coverage   63.08%   63.08%           
=======================================
  Files         133      133           
  Lines       17313    17313           
=======================================
  Hits        10922    10922           
  Misses       6391     6391

Impacted Files	Coverage Δ
astroquery/sdss/core.py	`92.06% <ø> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

docs/sdss/sdss.rst

eerovaher · 2022-08-02T20:45:59Z

docs/sdss/sdss.rst

+contains new optical and infrared spectra from both Apache Point
+Observatory and Las Campanas Observatory. Previously released
+integral-field datacubes and maps, stellar library spectra, as well as
+images, are also included in DR17.  Users may select alternate DR's.


How can a different data release be selected? Documentation is not helpful if it just says that something can be done, but neither describes nor provides examples how.

Sorry, first timer. In the core routine of the sdss package, the various query procedures allow for the data release parameter to be specified (with the default DR set initially). This has never been documented properly with examples (though seems straightforward for user specification). The easiest thing would be to remove the phrase now and then, take some time to generate an example. Sound alright?

The fact that changing the data release is not documented is a shortcoming that should be fixed, but I would tend to agree that it would be beyond the scope of a pull request that simply updates the default. But perhaps it would be better to remove the entire section because only removing the last sentence might leave the wrong impression that DR17 is the only option.

I chatted about this with @weaverba137. We believe that it is important to note what the default data release is and what it contains. This omission by astroquery is what lead to the initial issue request as users were confused as to why DR14 was employed by astroquery as opposed to the most recent DR from SDSS (DR17). The language can be massaged accordingly, but users do have some inherent responsibility to understand something about the database/dataset they are querying. If a user is employing SDSS data, there is a high probability that they understand that it has issued multiple DR's over the years.

I think the language @jsobeck is suggesting should remain, because it opens the door to astroquery eventually getting the functionality to access these other SDSS data sets.

More comments below.

eerovaher · 2022-08-02T21:10:21Z

Updating the default data release needs an entry in the change log

I'll give you a few suggestions for any future pull requests you might open.

Write the pull associated issue number in the description of the pull request using one of these phrases so that the issue can be automatically linked to the pull request. Writing the issue number in the title is not helpful.

It would be much better if the pull request title would describe what its purpose is, not how it is implemented. In this case using the title of your first commit as the pull request title would have been much more useful.

Markdown interprets double underscores as bold font specifiers, which is why your pull request description contains init.py instead of __init__.py. You should use backticks to format code or filenames.

jsobeck · 2022-08-02T21:38:57Z

Got it, @eerovaher. Thanks for information. I was moving too quickly and should have been more careful/deliberate. Though I looked at the change log and knew it had to be modified, it was not clear to me that I should be the one to change it (as opposed to a "reviewer"; even after reading through, e.g., CONTRIBUTING.rst).

eerovaher · 2022-08-02T21:56:27Z

The related issue must be mentioned in the pull request description, i.e. the opening post, not the title. You can see that GitHub has not linked the issue because you have used the correct expression in the wrong place.

Though I looked at the change log and knew it had to be modified, it was not clear to me that I should be the one to change it (as opposed to a "reviewer"; even after reading through, e.g., CONTRIBUTING.rst).

Make sure to mention this over at #2462

bsipocz · 2022-08-02T22:15:03Z

There is one remote data test that is failing and has to be fixed. Also, please squash the commits down to one. And I agree with @eerovaher that the docs should contain code blocks to be useful, mentioning what the default is doesn't really bring much to the table without showing how to change it after all the default is not hidden, but listed in the signature and is set by a standard astropy config system. I agree that it can be confusing to users, but then the confusion should be fixed by showing of how to use a different release, with an example.

weaverba137 · 2022-08-02T22:23:16Z

I suspect that the broken remote data test may be fixed in #2477.

I also agree that an example with the data release set explicitly would be useful, and we can use such an example to highlight real changes that have happened to the SDSS data sets over the years. Two examples:

Compare redshifts of a small set of objects over data releases.
The photometric calibration was redone in DR13. This is not necessarily widely known, so one could compare fluxes/magnitudes for a small number of photometric objects between say, DR12 (before recalibration) and DR17.

bsipocz · 2022-08-02T22:31:26Z

The test is broken due to the change in the default dr, so definitely belongs to this PR. OTOH, I'm OK with you cherry-picking and squashing down the commits from this PR to yours, so Jennifer gets a commit credit but we're not juggling two PRs at the same time for the same module.

weaverba137 · 2022-08-02T23:03:45Z

What about merging #2477 first, then rebasing this PR? I think that may be easier in the long run.

jsobeck · 2022-08-02T23:22:43Z

Agree to the suggestion from @weaverba137 to merge #2477 first (though my input is not the most crucial here).

weaverba137 · 2022-08-02T23:32:05Z

@bsipocz, could you say which specific remote data test is failing? I'd like to verify that it is in fact fixed or at least not also broken in #2477.

bsipocz · 2022-08-03T18:15:43Z

__________________________________________ TestSDSSRemote.test_large_crossid ___________________________________________

self = <astroquery.sdss.tests.test_sdss_remote.TestSDSSRemote object at 0x14eb85490>
large_results = <SkyCoord (ICRS): (ra, dec) in deg
    [(334.26483  ,  3.0572122 ), (334.35421  ,  2.9292896 ),
     (334.35896  ,  3....  ( 22.726164 , 20.667794  ), ( 22.655787 , 20.861832  ),
     ( 22.714435 , 20.469003  ), ( 22.713696 , 20.939534  )]>

    def test_large_crossid(self, large_results):
        # Regression test for #589
    
        results = sdss.SDSS.query_crossid(large_results)
>       assert len(results) == 894
E       assert 845 == 894
E        +  where 845 = len(<Table length=845>\n obj_id        objID               ra        ...       obj_id1        type \n bytes7        int64   ...557872910172 ... 1237679541360984660 GALAXY\nobj_999 1237679476942701227 22.7136962885291 ... 1237679476942701227 GALAXY)

astroquery/sdss/tests/test_sdss_remote.py:205: AssertionError

weaverba137 · 2022-08-03T18:20:28Z

Thank you, I will take a look at the details there.

weaverba137 · 2022-08-03T18:50:46Z

I've been able to reproduce this. The issue is that the input data large_results is not necessarily deterministic even within a given data release, and certainly not deterministic from one data release to another.

Ultimately, what is test_large_crossid testing? If it is simply to ensure that the queries can handle a ~ 1000-item data set, there should be a better way to generate the input data deterministically.

bsipocz

Changing the default needs to be mentioned in the changelog. And the test is still failing.

So, I just go ahead, rebased this on top of the recently merged other PR, to be extra sure, and add the missing pieces for a quick merge.

bsipocz · 2022-09-11T04:37:03Z

Thank you @jsobeck!

weaverba137 · 2022-09-13T17:15:16Z

@bsipocz, I guess the resolution to the failed test was just to mark DR17 as default?

I'd still like to know the original purpose of test_large_crossid, if that information still exists?

bsipocz · 2022-09-13T18:03:50Z

Yes, the "fix" was to update to the result of DR17.

As for your second question, it was a regression reported in #589

bsipocz · 2022-09-13T18:05:04Z

(so I suppose we could change the test to not assert on the actual number, but that it runs and returns results, but I don't think it's a big deal to leave it as is)

weaverba137 · 2022-09-13T18:14:55Z

OK, thanx. I'll go back and read that.

eerovaher reviewed Aug 2, 2022

View reviewed changes

docs/sdss/sdss.rst Outdated Show resolved Hide resolved

eerovaher reviewed Aug 2, 2022

View reviewed changes

jsobeck changed the title ~~PR for Issue #2365: Modifying astroquery/sdss package init file and associated documentation~~ Closes #2365: Modifying astroquery/sdss package init file and associated documentation Aug 2, 2022

bsipocz added the sdss label Aug 2, 2022

jsobeck changed the title ~~Closes #2365: Modifying astroquery/sdss package init file and associated documentation~~ Modifying astroquery/sdss package init file to use most current SDSS DR (as well as associated documentation) #2365 Aug 2, 2022

jsobeck changed the title ~~Modifying astroquery/sdss package init file to use most current SDSS DR (as well as associated documentation) #2365~~ Modifying astroquery/sdss package init file to use most current SDSS DR (as well as associated documentation) Aug 2, 2022

bsipocz added this to the v0.4.7 milestone Sep 11, 2022

jsobeck added 2 commits September 10, 2022 21:13

SDSS module: set default data release to DR17

7b2c0eb

Slight change to sdss-associated documentation

32c7543

bsipocz approved these changes Sep 11, 2022

View reviewed changes

Minor updates for DR17

5f60c09

bsipocz force-pushed the main branch from 81a70e0 to 5f60c09 Compare September 11, 2022 04:20

bsipocz merged commit 8c3e869 into astropy:main Sep 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modifying astroquery/sdss package init file to use most current SDSS DR (as well as associated documentation) #2478

Modifying astroquery/sdss package init file to use most current SDSS DR (as well as associated documentation) #2478

jsobeck commented Aug 2, 2022 •

edited

Loading

pep8speaks commented Aug 2, 2022 •

edited

Loading

codecov bot commented Aug 2, 2022 •

edited

Loading

eerovaher Aug 2, 2022 •

edited

Loading

jsobeck Aug 2, 2022

eerovaher Aug 2, 2022

jsobeck Aug 2, 2022 •

edited

Loading

weaverba137 Aug 2, 2022

eerovaher commented Aug 2, 2022

jsobeck commented Aug 2, 2022 •

edited

Loading

eerovaher commented Aug 2, 2022

bsipocz commented Aug 2, 2022

weaverba137 commented Aug 2, 2022

bsipocz commented Aug 2, 2022

weaverba137 commented Aug 2, 2022

jsobeck commented Aug 2, 2022

weaverba137 commented Aug 2, 2022

bsipocz commented Aug 3, 2022

weaverba137 commented Aug 3, 2022

weaverba137 commented Aug 3, 2022

bsipocz left a comment

bsipocz commented Sep 11, 2022

weaverba137 commented Sep 13, 2022

bsipocz commented Sep 13, 2022

bsipocz commented Sep 13, 2022 •

edited

Loading

weaverba137 commented Sep 13, 2022

Modifying astroquery/sdss package init file to use most current SDSS DR (as well as associated documentation) #2478

Modifying astroquery/sdss package init file to use most current SDSS DR (as well as associated documentation) #2478

Conversation

jsobeck commented Aug 2, 2022 • edited Loading

pep8speaks commented Aug 2, 2022 • edited Loading

Comment last updated at 2022-09-11 04:20:09 UTC

codecov bot commented Aug 2, 2022 • edited Loading

Codecov Report

eerovaher Aug 2, 2022 • edited Loading

Choose a reason for hiding this comment

jsobeck Aug 2, 2022

Choose a reason for hiding this comment

eerovaher Aug 2, 2022

Choose a reason for hiding this comment

jsobeck Aug 2, 2022 • edited Loading

Choose a reason for hiding this comment

weaverba137 Aug 2, 2022

Choose a reason for hiding this comment

eerovaher commented Aug 2, 2022

jsobeck commented Aug 2, 2022 • edited Loading

eerovaher commented Aug 2, 2022

bsipocz commented Aug 2, 2022

weaverba137 commented Aug 2, 2022

bsipocz commented Aug 2, 2022

weaverba137 commented Aug 2, 2022

jsobeck commented Aug 2, 2022

weaverba137 commented Aug 2, 2022

bsipocz commented Aug 3, 2022

weaverba137 commented Aug 3, 2022

weaverba137 commented Aug 3, 2022

bsipocz left a comment

Choose a reason for hiding this comment

bsipocz commented Sep 11, 2022

weaverba137 commented Sep 13, 2022

bsipocz commented Sep 13, 2022

bsipocz commented Sep 13, 2022 • edited Loading

weaverba137 commented Sep 13, 2022

jsobeck commented Aug 2, 2022 •

edited

Loading

pep8speaks commented Aug 2, 2022 •

edited

Loading

codecov bot commented Aug 2, 2022 •

edited

Loading

eerovaher Aug 2, 2022 •

edited

Loading

jsobeck Aug 2, 2022 •

edited

Loading

jsobeck commented Aug 2, 2022 •

edited

Loading

bsipocz commented Sep 13, 2022 •

edited

Loading