Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2020.4 #342

Merged
merged 6 commits into from
Apr 24, 2020
Merged

2020.4 #342

merged 6 commits into from
Apr 24, 2020

Conversation

javierggt
Copy link
Contributor

@javierggt javierggt commented Apr 20, 2020

The 2020.4 release includes:

  • the approved-last-week-at-LR updates to the ACIS checkers and adds the new ACIS bep fep checkers to the ska3-flight package set.
  • updates to proseco, sparkles, and chandra_aca in support of the ACA limit increase
  • the updates to xija (new ODE solver) already promoted in the Matlab 2020_005 environment
  • the recent major enhancements to xija's gui_fit

These substantive updates and a few others are also called out in bold in the full list of changes below.

New Packages

aca_view 0.3.0

bep_pcb_check 3.0.0
fep1_actel_check 3.0.0
fep1_actel_check 3.0.0

Changes

acis_thermal_check: 2.9.2 -> 3.0.0 (2.9.2 -> 2.9.3 -> 3.0.0)

  • PR 26: Setting up automated builds
  • PR 25: acis_thermal_check 3.0

acisfp_check: 2.7.2 -> 3.0.0 (2.7.2 -> 2.7.3 -> 3.0.0)

  • PR 19: Setting up automated builds
  • PR 21: update acisfp_check for acis_thermal_check 3.0

backstop_history: 1.1.2 -> 1.1.3 (1.1.2 -> 1.1.3)

  • PR 8: Setting up automated builds

dea_check: 2.3.2 -> 3.0.0 (2.3.2 -> 2.3.3 -> 3.0.0)

  • PR 21: Setting up automated builds
  • PR 22: update dea_check for acis_thermal_check 3.0

dpa_check: 2.5.2 -> 3.0.0 (2.5.2 -> 2.5.3 -> 3.0.0)

  • PR 20: Setting up automated builds
  • PR 21: update dpa_check for acis_thermal_check 3.0

psmc_check: 1.3.1 -> 3.0.0 (1.3.1 -> 1.3.2 -> 3.0.0)

  • PR 19: Setting up automated builds
  • PR 20: update psmc_check for acis_thermal_check 3.0

acdc: 4.4.1 -> 4.5.0 (4.4.1 -> 4.5.0)

  • PR 46: Setting up automated builds
  • PR 48: Write out updated sparkles pkl
  • PR 47: Fiddle with the scaling for the warm pix plot

agasc: 4.8.0 -> 4.8.1 (4.8.0 -> 4.8.1)

  • PR 41: Setting up automated builds
  • PR 42: Make tests pass on Windows

annie: 0.9.1 -> 0.10.1 (0.9.1 -> 0.10.0 -> 0.10.1)

  • PR 98: Setting up automated builds
  • PR 97: End simulation at large att offset
  • PR 100: Minor test fix to pass on windows

chandra_aca: 4.28.1 -> 4.29.1 (4.28.1 -> 4.29.0 -> 4.29.1)

  • PR 91: Setting up automated builds
  • PR 92: Use repository_dispatch to trigger build
  • PR 88: Add acquisition probability model 2020-02 and make it default (supports new ACA limit)
  • PR 93: change to fix release workflow
  • PR 96: Make test_get_aimpoint work without being on VPN

mica: 4.20.0 -> 4.21.0 (4.20.0 -> 4.21.0)

  • PR 218: Minor PY3 updates and DS10.8.3 update for mica vv processing (These mica updates will allow us to move its cron task from an old Python 2 to new Python 3 job, and also support the minor changes to dy, dz, dtheta asol data due to the new AXISCORRECT changes in DS10.8.3)
  • PR 219: Mag stats update
  • PR 214: Remove dark model
  • PR 215: Setting up automated builds
  • PR 173: Refactor stats into update and get_stats modules

parse_cm: 3.5.1 -> 3.5.2 (3.5.1 -> 3.5.2)

  • PR 20: Setting up automated builds
  • PR 21: Fix tests and code to pass on Windows

proseco: 4.7.2 -> 4.8.0 (4.7.2 -> 4.8.0)

  • PR 315: Apply temperature scaling to guide faint mag limit (supports new ACA limit)
  • PR 316: Update penalty limit to -8.1C (to support -7.1C planning limit). (supports new ACA limit)
  • PR 297: Allow optimizing acq halfw for included star
  • PR 320: Update tests for 2020-02 acq model update
  • PR 319: Setting up automated builds
  • PR 321: Fix optimizing halfw for case of many include_ids
  • PR 322: Add a test of calc_p_safe for two identical catalogs

pyyaks: 4.4.1 -> 4.4.2 (4.4.1 -> 4.4.2)

  • PR 10: Setting up automated builds
  • PR 11: Fix tests to pass on Windows

quaternion: 3.5.1 -> 3.5.2 (3.5.1 -> 3.5.2)

  • PR 28: Properly unpickle quaternions from versions earlier than 3.5

sparkles: 4.4.0 -> 4.6.0 (4.4.0 -> 4.5.0 -> 4.6.0)

xija: 4.16.0 -> 4.18.2 (4.16.0 -> 4.17.0 -> 4.17.1 -> 4.18.0 -> 4.18.1 -> 4.18.2)

  • PR 76: Setting up automated builds
  • PR 77: enable triggering conda build using repository dispatch
  • PR 79: change to fix release workflow
  • PR 75: New ODE solver
  • PR 80: Rename dev/core.pyx to alt_core/core.pyx
  • PR 82: Bugfix to build on Windows from mbaski
  • PR 69: gui_fit upgrades
  • PR 83: [bugfix] Wrong line is updated in gui_fit
  • PR 84: Fix bug in looking for bad times

Testing status

We note two test failures that we consider non-issues for this release:

  • Ska.engarchive: The Ska.engarchive failure is a test failure in the sync tests. The test now does not pass in current HEAD ska3/flight (previous installed version) and the 2020.4 release does not touch Ska.engarchive, so this is not a regression. Additionally, the HEAD ska3/flight code is not actually run to make the sync files. We have an open action to investigate the test failure but consider it a non-issue for this release.
  • Ska.quatutil: The module passes unit tests when run in the test environment (the point of the integration testing). We have a minor testing defect when run from the automated integration testing of ska_testr and consider it a non-issue for this release
**************************************************************
***      Package                  Script            Status ***
*** ------------------ ---------------------------- ------ ***
***   Chandra.Maneuver                 test_unit.py   pass ***
***   Chandra.Maneuver           post_check_logs.py   pass ***
***       Chandra.Time                 test_unit.py   pass ***
***       Chandra.Time           post_check_logs.py   pass ***
*** Chandra.cmd_states                 test_unit.py   pass ***
*** Chandra.cmd_states           post_check_logs.py   pass ***
***         Quaternion                 test_unit.py   pass ***
***         Quaternion           post_check_logs.py   pass ***
***            Ska.DBI                 test_unit.py   pass ***
***            Ska.DBI           post_check_logs.py   pass ***
***          Ska.Numpy                 test_unit.py   pass ***
***          Ska.Numpy           post_check_logs.py   pass ***
***        Ska.ParseCM             test_unit_git.sh   pass ***
***        Ska.ParseCM           post_check_logs.py   pass ***
***          Ska.Shell                 test_unit.py   pass ***
***          Ska.Shell           post_check_logs.py   pass ***
***          Ska.Table             test_unit_git.sh   ---- ***
***          Ska.Table           post_check_logs.py   ---- ***
***     Ska.engarchive    test_make_archive_long.sh   ---- ***
***     Ska.engarchive                 test_unit.py   FAIL ***
***     Ska.engarchive           post_check_logs.py   FAIL ***
***            Ska.ftp                 test_unit.py   pass ***
***            Ska.ftp           post_check_logs.py   pass ***
***       Ska.quatutil             test_unit_git.sh   pass ***
***       Ska.quatutil           post_check_logs.py   FAIL ***
***            Ska.tdb                 test_unit.py   pass ***
***            Ska.tdb           post_check_logs.py   pass ***
***          acis_taco                 test_unit.py   pass ***
***          acis_taco           post_check_logs.py   pass ***
***       acisfp_check              test_regress.sh   pass ***
***       acisfp_check         test_regress_head.sh   pass ***
***       acisfp_check           post_check_logs.py   pass ***
***       acisfp_check              post_regress.py   pass ***
***       acisfp_check         post_regress_head.py   pass ***
***              agasc                 test_unit.py   pass ***
***              agasc           post_check_logs.py   pass ***
***              annie                 test_unit.py   pass ***
***              annie           post_check_logs.py   pass ***
***        chandra_aca                 test_unit.py   pass ***
***        chandra_aca           post_check_logs.py   pass ***
***            cxotime                 test_unit.py   pass ***
***            cxotime           post_check_logs.py   pass ***
***          dea_check              test_regress.sh   pass ***
***          dea_check         test_regress_head.sh   pass ***
***          dea_check           post_check_logs.py   pass ***
***          dea_check              post_regress.py   pass ***
***          dea_check         post_regress_head.py   pass ***
***          dpa_check              test_regress.sh   pass ***
***          dpa_check         test_regress_head.sh   pass ***
***          dpa_check           post_check_logs.py   pass ***
***          dpa_check              post_regress.py   pass ***
***          dpa_check         post_regress_head.py   pass ***
***               kadi         test_regress_long.sh   ---- ***
***               kadi                 test_unit.py   pass ***
***               kadi           post_check_logs.py   pass ***
***               kadi         post_regress_long.py   ---- ***
***              maude                 test_unit.py   pass ***
***              maude           post_check_logs.py   pass ***
***               mica                 test_unit.py   pass ***
***               mica           post_check_logs.py   pass ***
***   package_manifest              test_regress.sh   pass ***
***   package_manifest           post_check_logs.py   pass ***
***   package_manifest              post_regress.py   pass ***
***           parse_cm                 test_unit.py   pass ***
***           parse_cm           post_check_logs.py   pass ***
***            proseco                 test_unit.py   pass ***
***            proseco           post_check_logs.py   pass ***
***         psmc_check              test_regress.sh   pass ***
***         psmc_check         test_regress_head.sh   pass ***
***         psmc_check           post_check_logs.py   pass ***
***         psmc_check              post_regress.py   pass ***
***         psmc_check         post_regress_head.py   pass ***
***             pyyaks                 test_unit.py   pass ***
***             pyyaks           post_check_logs.py   pass ***
***           sparkles                 test_unit.py   pass ***
***           sparkles           post_check_logs.py   pass ***
***          starcheck              test_regress.sh   pass ***
***          starcheck           post_check_logs.py   pass ***
***          starcheck              post_regress.py   pass ***
***          timelines test_unit_git_sybase_long.sh   ---- ***
***          timelines           post_check_logs.py   ---- ***
***               xija                 test_unit.py   pass ***
***               xija           post_check_logs.py   pass ***
**************************************************************

Checklist

  • Document all changes
  • Build all packages
    • noarch
    • linux 64
    • mac 64
  • Packages copied to test channel
  • test status documented
  • get approval
  • Individual packages moved to main conda channel
  • Update production systems:
    • announce to aca@cfa
    • ska3-flight package moved to main conda channel
    • update aca@HEAD
    • run testr@HEAD
    • update sot@cheru
    • update sot@chimchim
    • announce to chandracode and mpweekly

Fixes Issues

Fixes #325
Fixes #318
Fixes #331
Fixes #336
Fixes #335
Fixes #334
Fixes #337
Fixes #324
Fixes #319
Fixes #316
Fixes #327
Fixes #328
Fixes #330
Fixes #339
Fixes #313

jeanconn and others added 3 commits April 20, 2020 14:29
Update agasc to 4.8.1
Update acisfp_check to 3.0.0
Update acis_thermal_check to 3.0.0
Update backstop_history to 1.1.3
Update chandra_aca to 4.29.1
Update dea_check to 3.0.0
Update dpa_check to 3.0.0
Update parse_cm to 3.5.2
Update proseco to 4.8.0
Update psmc_check to 3.0.0
Update pyyaks to 4.4.2
Update Quaternion to 3.5.2
Update sparkles to 4.6.0
Update xija to 4.17.1

Updated and an empty message aborts the commit.
@jeanconn
Copy link
Contributor

Looks good. I made a new releaes for mica that should be good to go.

@jeanconn
Copy link
Contributor

So PR got expanded from "just the ones we really need" to "everything except the namespace packages". That adds a file like parse_cm and pyyaks. I'm good with that and seems straightforward and appropriate.

@javierggt
Copy link
Contributor Author

Well, I'm fine either way. I can remove the updates to pyyaks, parse_cm, agasc and backstop_history, which are not relevant or windows-related. Perhaps also annie.

@jeanconn
Copy link
Contributor

Also fine either way. The content for briefing is tiny for each.

pkg_defs/ska3-flight/meta.yaml Outdated Show resolved Hide resolved
@jeanconn
Copy link
Contributor

jeanconn commented Apr 20, 2020

Also, what's the magic again to get the PR list* that you're now using? I assume it isn't quite automated for this kind PR but not sure if that changed (and it is automated) or if it is a process in the skare3_tools or wiki.

*awesome

@javierggt
Copy link
Contributor Author

Also, what's the magic again to get the PR list* that you're now using? I assume it isn't quite automated for this kind PR but not sure if that changed (and it is automated) or if it is a process in the skare3_tools or wiki.

Right now the list comes from a script I have not checked in. It is based on a script in skare3_tools/scripts/skare3_update_summary. My local version can compare between two sets of versions.

The list of issues that say "Fixes #..." is another script that is also not checked in. It's a little function that takes the milestone and prints out all skare3 issues labeled "Package update" for that milestone.

@@ -1,6 +1,6 @@
package:
name: ska3-flight
version: 2020.02.12
version: 2020.04.24
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forget if there was some agreement or reasoning otherwise, but I thought this should just be 2020.4. That makes the version here match the release tag and the milestone.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In other words, starting in 2020 all the version tags will be YYYY.N where N just increases by 1 for each release. This can include skipped releases. That would include the others like ska3-template or ska3-core.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My disagreement was that when these were candidates I lost my ability to make packages with unique names as we went through updates. For example, in this PRI already would have incremented to 04.25 after the mica fix because knowing me I'd have built 04.24 without mica and would have that actual packagehanging around to confuse me (or my auto package building wouldn't have actually rebuilt ska3-flight after adding mica because 04.24 already existed).

I don't really care if we move forward with some of @javier's ideas for SCM of the meta.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind either way. I also thought that it would be 2020.4, but in 2020.3 I saw Jean was still following the date format and that got me thinking that releases can be 2020.n until the actual final tag, and then they get their final "date" name. But that was just me thinking, I agree.

So... no problem in changing it to 2020.4

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have certainly experienced the pain of "over-caching" by conda and getting confused. So we might consider using a standard versioning convention like 2020.4rc<N> until the last one. I can't tell if the pain of that is greater than the pain of remembering to clear caches or force-build.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, the Python standard https://www.python.org/dev/peps/pep-0440/#pre-releases does not include a dash before the rc.

Copy link
Contributor

@jeanconn jeanconn Apr 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had wondered if it would be less work in the end to just manage ska3-flight and ska3-core as two new very small git repos but I think @javierggt has better ideas.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also talked about maybe just combining ska3-core into ska3-flight but I have enjoyed having noarch updates possible for ska3-flight (though that could all be moot with real multiplatform auto builds).

Copy link
Contributor Author

@javierggt javierggt Apr 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I have in mind at this time is the following:

  • One releases one meta-package at a time, so ska3-core would have to be released before ska3-flight. Each ska3-flight corresponds to one and only one ska3-core. There is no issue.
  • When you make a release (or just a tag) an action is triggered. This action reads meta.yaml and overwrites the version of the package to create a conda package which is uploaded to a repository. This is the same as with other packages, except the name of the package to build comes from the tag (not the repository name).
  • this package eventually winds its way to the test conda channel (instead of masters).
  • when a new ska3-flight package appears in the test conda channel, that triggers a testr run.
  • eventually the results from this run make it to the dashboard (steps omitted here).
  • Once we agree on a final yaml config, we tag/release again, without the rc#. The process repeats, and the package ends up wherever we want it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure if we can promote ska3-core updates independent of ska3-flight, but we could probably release in that order? Not sure where to capture this discussion but should probably resolve the conversation for this line of the meta.yaml :-)

@jeanconn
Copy link
Contributor

I'll change my review to approved when #321 gets merged.

@javierggt javierggt closed this Apr 24, 2020
@javierggt javierggt reopened this Apr 24, 2020
@javierggt javierggt merged commit 68d9db9 into master Apr 24, 2020
@javierggt javierggt deleted the 2020.4 branch May 6, 2020 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment