Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2022.2 #774

Merged
merged 5 commits into from
Mar 10, 2022
Merged

2022.2 #774

merged 5 commits into from
Mar 10, 2022

Conversation

javierggt
Copy link
Contributor

@javierggt javierggt commented Jan 27, 2022

ska3-flight 2022.2

This PR includes an update of ska3-flight, ska3-core and ska3-perl meta-packages. Most of the changes to ska3-flight packages were needed to comply with small changes in the interface of ska3-core packages. The following are the ska3-flight changes that are not related to core package changes:

  • proseco, sparkles and starcheck now include checks for overlapping guide star tracks.

Interface Impacts:

There are no interface changes in Ska code, but this is a comprehensive update of external packages. Interface changes in external packages can cause behavior changes, although we have not identified many of them in integration and functional testing.

The issues we have identified were the appearance of warnings stemming from the following:

  • Since Numpy 1.20.0, numpy.object and other numpy types are deprecated (https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations)
  • Matplotlib's date2num function changed behavior (default args, units) in multiple version releases.
  • Astropy changes ecsv from 0.9 to 1.0. ECSV files are currently saved without applying formatting to the values. (this can be relevant as some regression tests use ECSV+diff as basis for test result comparisons).

Testing:

The latest release candidates are installed in /proj/sot/ska3/test on HEAD, and all release candidates are available for testing from the usual channels:

conda create -n ska3-flight-2022.2rc6 --override-channels \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/test \
  ska3-flight==2022.2rc6

Functional tests

  • acaview
  • xija_gui_fit
  • esaview

Review

All operations critical or impacting PR's are independently and carefully reviewed. For other PR's the level of detail for review is calibrated to operations criticality. Some PR's that are confined to aspect-team-specific processing may have little to no independent review.

Deployment

ska3-flight 2022.2 will be promoted to flight conda channel and installed on HEAD and GRETA Linux after the FSDS Jira ticket is approved.

Code changes

These are the changes in the linux-64 platform. Changes in osx-64 or win-64 differ.

ska3-flight changes (2022.1 -> 2022.2rc6)

Updated Packages

  • acis_taco: 4.2.0 -> 4.2.1 (4.2.0 -> 4.2.1)
    • PR 21 (javierggt): remove use of deprecated np.int
  • annie: 0.11.0 -> 0.11.1 (0.11.0 -> 0.11.1)
    • PR 109 (javierggt): remove use of deprecated np.float and np.int
  • chandra_aca: 4.34.0 -> 4.34.1 (4.34.0 -> 4.34.1)
    • PR 120 (javierggt): remove use of deprecated np.bool and np.int
  • kadi: 5.8.0 -> 5.8.1 (5.8.0 -> 5.8.1)
    • PR 214 (javierggt): remove use of deprecated np.str
    • PR 215 (javierggt): fix warning
  • proseco: 5.4.0 -> 5.5.0 (5.4.0 -> 5.5.0)
    • PR 370 (jeanconn): Exclude guide star if it overlaps with brighter/better guide star
    • PR 371 (javierggt): Prevent matplotlib warning
  • ska.engarchive: 4.54.0 -> 4.54.1 (4.54.0 -> 4.54.1)
    • PR 225 (javierggt): remove use of deprecated np.bool
  • ska.matplotlib: 3.13.0 -> 3.14.1 (3.13.0 -> 3.13.1 -> 3.14.0 -> 3.14.1)
    • PR 20 (javierggt): remove use of deprecated np.float
    • PR 21 (javierggt): Prevent matplotlib warning
    • PR 24 (taldcroft): Fix plot_cxctime for compatibility with matplotlib 3.5
  • ska.numpy: 3.9.0 -> 3.9.1 (3.9.0 -> 3.9.1)
    • PR 14 (javierggt): remove use of deprecated np.int
  • ska3-core: 2021.8 -> 2022.2rc6
  • sparkles: 4.13.1 -> 4.14.0 (4.13.1 -> 4.14.0)
    • PR 166 (jeanconn): Add a check for two tracked things that are within 12 pixels.
  • starcheck: 13.14.0 -> 13.15.1 (13.14.0 -> 13.15.0 -> 13.15.1)
    • PR 379 (jeanconn): Add missing newline to guide count warn
    • PR 380 (jeanconn): Add track overlap check
    • PR 381 (javierggt): prevent matplotlib warning
    • PR 382 (jeanconn): Set linestyle via fmt in plot_two
  • testr: 4.6.0 -> 4.7.0 (4.6.0 -> 4.7.0)
    • PR 38 (taldcroft): Add has_internet function to test_helpers
    • PR 37 (javierggt): Ignore deprecation warning that comes from external packages
  • xija: 4.24.0 -> 4.24.1 (4.24.0 -> 4.24.1)
    • PR 115 (javierggt): remove use of deprecated np.float and np.int

ska3-core changes (2021.8 -> 2022.2rc6)

New Packages

  • _openmp_mutex: 4.5
  • alsa-lib: 1.2.3
  • anyio: 2.2.0
  • appdirs: 1.4.4
  • argon2-cffi: 20.1.0
  • astroquery: 0.4.5
  • async_generator: 1.10
  • backports: 1.0
  • backports.zoneinfo: 0.2.1
  • black: 19.10b0
  • bottleneck: 1.3.2
  • brotli: 1.0.9
  • c-ares: 1.18.1
  • charset-normalizer: 2.0.4
  • click: 8.0.3
  • colorama: 0.4.4
  • coverage: 6.2
  • debugpy: 1.5.1
  • et_xmlfile: 1.1.0
  • font-ttf-dejavu-sans-mono: 2.37
  • font-ttf-inconsolata: 2.001
  • font-ttf-source-code-pro: 2.030
  • font-ttf-ubuntu: 0.83
  • fonts-conda-forge: 1
  • fonttools: 4.28.5
  • gettext: 0.21.0
  • giflib: 5.2.1
  • glib-tools: 2.68.4
  • html5lib: 1.1
  • hypothesis: 6.29.3
  • jbig: 2.1
  • jeepney: 0.7.1
  • jupyter_server: 1.4.1
  • jupyterlab_pygments: 0.1.2
  • jupyterlab_widgets: 1.0.0
  • kaleido-core: 0.2.1
  • keyring: 23.4.0
  • lerc: 2.2.1
  • libblas: 3.9.0
  • libcblas: 3.9.0
  • libclang: 11.1.0
  • libdeflate: 1.7
  • libev: 4.33
  • libevent: 2.1.10
  • libgfortran4: 7.5.0
  • libglib: 2.68.4
  • liblapack: 3.9.0
  • libllvm11: 11.1.0
  • libmamba: 0.19.1
  • libmambapy: 0.19.1
  • libnghttp2: 1.46.0
  • libogg: 1.3.5
  • libopenblas: 0.3.12
  • libopus: 1.3.1
  • libpq: 13.3
  • libuv: 1.40.0
  • libvorbis: 1.3.7
  • libwebp: 1.2.1
  • libwebp-base: 1.2.1
  • libxkbcommon: 1.0.3
  • llvm-openmp: 12.0.1
  • mathjax: 2.7.7
  • matplotlib-inline: 0.1.2
  • munkres: 1.1.4
  • mypy_extensions: 0.4.3
  • mysql-common: 8.0.25
  • mysql-libs: 8.0.25
  • nbclassic: 0.2.6
  • nbclient: 0.5.3
  • nest-asyncio: 1.5.1
  • nspr: 4.32
  • nss: 3.69
  • openblas: 0.3.12
  • openpyxl: 3.0.9
  • pathspec: 0.7.0
  • platformdirs: 2.4.0
  • ply: 3.11
  • psycopg2: 2.9.2
  • pybind11-abi: 4
  • pyqt-impl: 5.12.3
  • pyqt5-sip: 4.19.18
  • pyqtchart: 5.12
  • pyqtwebengine: 5.12.1
  • pytest-arraydiff: 0.3
  • pytest-astropy: 0.9.0
  • pytest-astropy-header: 0.1.2
  • pytest-cov: 3.0.0
  • pytest-doctestplus: 0.11.2
  • pytest-filter-subpackage: 0.1.1
  • pytest-mock: 3.6.1
  • pytest-openfiles: 0.5.0
  • pytest-remotedata: 0.3.3
  • python-mimeparse: 1.6.0
  • pyvo: 1.2.1
  • regex: 2021.11.2
  • regions: 0.5
  • reproc: 14.2.3
  • reproc-cpp: 14.2.3
  • secretstorage: 3.3.1
  • sniffio: 1.2.0
  • sortedcontainers: 2.4.0
  • tenacity: 8.0.1
  • tomli: 1.2.2
  • typed-ast: 1.4.3
  • typing-extensions: 3.10.0.2
  • tzdata: 2021e
  • unicodedata2: 14.0.0
  • yaml-cpp: 0.6.3

Removed Packages

  • intel-openmp
  • libllvm9
  • mkl
  • mkl-service
  • mkl_fft
  • mkl_random
  • more-itertools
  • pandoc
  • pbr
  • retrying
  • sip
  • snappy

Updated Packages

  • asgiref: 3.2.10 -> 3.4.1
  • astroid: 2.4.2 -> 2.9.0
  • astropy: 4.2.1 -> 5.0
  • astropy-healpix: 0.5 -> 0.6
  • attrs: 19.3.0 -> 21.4.0
  • autopep8: 1.5.3 -> 1.6.0
  • babel: 2.8.0 -> 2.9.1
  • bcrypt: 3.1.7 -> 3.2.0
  • beautifulsoup4: 4.9.1 -> 4.10.0
  • blas: 1.0 -> 1.1
  • bleach: 3.3.0 -> 4.1.0
  • blosc: 1.19.0 -> 1.21.0
  • bokeh: 2.1.1 -> 2.4.2
  • ca-certificates: 2021.1.19 -> 2021.10.26
  • certifi: 2020.12.5 -> 2021.10.8
  • cffi: 1.14.0 -> 1.14.6
  • chardet: 3.0.4 -> 4.0.0
  • conda: 4.8.3 -> 4.11.0
  • conda-build: 3.18.11 -> 3.21.7
  • conda-package-handling: 1.6.1 -> 1.7.3
  • cryptography: 2.9.2 -> 36.0.0
  • cycler: 0.10.0 -> 0.11.0
  • cython: 0.29.21 -> 0.29.25
  • dbus: 1.13.16 -> 1.13.18
  • decorator: 4.4.2 -> 5.1.0
  • defusedxml: 0.6.0 -> 0.7.1
  • django: 3.0.3 -> 3.1.7
  • docutils: 0.16 -> 0.17.1
  • docxtpl: 0.10.0 -> 0.11.5
  • expat: 2.2.9 -> 2.4.3
  • filelock: 3.0.12 -> 3.4.2
  • flake8: 3.8.3 -> 3.9.2
  • fontconfig: 2.13.0 -> 2.13.1
  • freetype: 2.10.2 -> 2.11.0
  • gitdb: 4.0.5 -> 4.0.7
  • gitpython: 3.1.3 -> 3.1.18
  • glib: 2.65.0 -> 2.68.4
  • gst-plugins-base: 1.14.0 -> 1.18.5
  • gstreamer: 1.14.0 -> 1.18.5
  • h5py: 2.10.0 -> 3.6.0
  • hdf5: 1.10.4 -> 1.10.6
  • icu: 58.2 -> 68.2
  • idna: 2.10 -> 3.3
  • imagesize: 1.2.0 -> 1.3.0
  • importlib-metadata: 1.7.0 -> 4.10.1
  • importlib_metadata: 1.7.0 -> 4.10.1
  • ipykernel: 5.3.3 -> 6.4.1
  • ipython: 7.16.1 -> 7.29.0
  • ipywidgets: 7.5.1 -> 7.6.5
  • isort: 4.3.21 -> 5.9.3
  • jedi: 0.17.1 -> 0.18.0
  • jinja2: 2.11.2 -> 2.11.3
  • jira: 2.0.0 -> 3.0.1
  • joblib: 0.16.0 -> 1.1.0
  • jpeg: 9b -> 9d
  • jplephem: 2.14 -> 2.17
  • json5: 0.9.5 -> 0.9.6
  • jupyter_client: 6.1.6 -> 7.1.0
  • jupyter_console: 6.1.0 -> 6.4.0
  • jupyter_core: 4.6.3 -> 4.9.1
  • jupyterlab: 2.1.5 -> 3.2.1
  • jupyterlab_server: 1.2.0 -> 2.10.2
  • kiwisolver: 1.2.0 -> 1.3.2
  • krb5: 1.18.2 -> 1.19.2
  • lazy-object-proxy: 1.4.3 -> 1.6.0
  • lcms2: 2.11 -> 2.12
  • ld_impl_linux-64: 2.33.1 -> 2.35.1
  • libarchive: 3.4.2 -> 3.5.2
  • libcurl: 7.71.1 -> 7.80.0
  • libedit: 3.1.20191231 -> 3.1.20210910
  • libgcc-ng: 9.1.0 -> 11.2.0
  • libgfortran-ng: 7.3.0 -> 7.5.0
  • libiconv: 1.15 -> 1.16
  • liblief: 0.10.1 -> 0.11.5
  • libsolv: 0.7.16 -> 0.7.19
  • libstdcxx-ng: 9.1.0 -> 11.2.0
  • libtiff: 4.1.0 -> 4.3.0
  • libxml2: 2.9.10 -> 2.9.12
  • libxslt: 1.1.34 -> 1.1.33
  • line_profiler: 2.1.2 -> 3.3.1
  • llvmlite: 0.33.0 -> 0.37.0
  • lxml: 4.5.2 -> 4.5.1
  • lz4-c: 1.9.2 -> 1.9.3
  • mamba: 0.5.1 -> 0.19.1
  • markupsafe: 1.1.1 -> 2.0.1
  • matplotlib: 3.2.2 -> 3.5.1
  • matplotlib-base: 3.2.2 -> 3.5.1
  • mock: 4.0.2 -> 4.0.3
  • mpld3: 0.3 -> 0.5.7
  • nb_conda_kernels: 2.2.3 -> 2.3.1
  • nbconvert: 5.6.1 -> 6.3.0
  • nbformat: 5.0.7 -> 5.1.3
  • ncurses: 6.2 -> 6.3
  • networkx: 2.4 -> 2.6.3
  • nodejs: 10.13.0 -> 16.13.1
  • notebook: 6.0.3 -> 6.4.6
  • numba: 0.50.1 -> 0.54.1
  • numexpr: 2.7.1 -> 2.8.1
  • numpy: 1.18.5 -> 1.20.3
  • numpy-base: 1.18.5 -> 1.20.3
  • openssl: 1.1.1i -> 1.1.1m
  • packaging: 20.4 -> 21.3
  • pandas: 1.0.5 -> 1.3.5
  • pandocfilters: 1.4.2 -> 1.4.3
  • paramiko: 2.7.1 -> 2.8.1
  • parso: 0.7.0 -> 0.8.3
  • patchelf: 0.11 -> 0.14.3
  • pcre: 8.44 -> 8.45
  • pillow: 7.2.0 -> 8.4.0
  • pip: 20.1.1 -> 21.2.4
  • pkginfo: 1.5.0.1 -> 1.8.2
  • plotly: 4.14.3 -> 5.5.0
  • pluggy: 0.13.1 -> 1.0.0
  • prometheus_client: 0.8.0 -> 0.12.0
  • prompt-toolkit: 3.0.5 -> 3.0.20
  • prompt_toolkit: 3.0.5 -> 3.0.20
  • psutil: 5.7.0 -> 5.9.0
  • ptyprocess: 0.6.0 -> 0.7.0
  • py: 1.9.0 -> 1.10.0
  • py-lief: 0.10.1 -> 0.11.5
  • pycodestyle: 2.6.0 -> 2.7.0
  • pycparser: 2.20 -> 2.21
  • pyerfa: 1.7.1.1 -> 2.0.0.1
  • pyflakes: 2.2.0 -> 2.3.1
  • pygments: 2.6.1 -> 2.10.0
  • pylint: 2.5.3 -> 2.12.2
  • pyopenssl: 19.1.0 -> 21.0.0
  • pyparsing: 2.4.7 -> 3.0.6
  • pyqt: 5.9.2 -> 5.12.3
  • pyrsistent: 0.16.0 -> 0.18.0
  • pytest: 6.2.0 -> 6.2.5
  • python: 3.8.3 -> 3.8.12
  • python-dateutil: 2.8.1 -> 2.8.2
  • python-docx: 0.8.10 -> 0.8.11
  • python-kaleido: 0.0.3 -> 0.2.1
  • python-libarchive-c: 2.9 -> 3.2
  • pytz: 2020.1 -> 2021.3
  • pyyaml: 5.3.1 -> 6.0
  • pyzmq: 19.0.1 -> 22.3.0
  • qt: 5.9.7 -> 5.12.9
  • qtconsole: 4.7.5 -> 5.1.1
  • qtpy: 1.9.0 -> 1.10.0
  • readline: 8.0 -> 8.1.2
  • requests: 2.24.0 -> 2.27.1
  • ripgrep: 11.0.2 -> 13.0.0
  • rope: 0.17.0 -> 0.21.1
  • ruamel_yaml: 0.15.87 -> 0.15.100
  • scikit-learn: 0.23.1 -> 1.0.2
  • scipy: 1.5.0 -> 1.7.3
  • send2trash: 1.5.0 -> 1.8.0
  • setuptools: 49.2.0 -> 58.0.4
  • setuptools-scm: 4.1.2 -> 6.3.2
  • setuptools_scm: 4.1.2 -> 6.3.2
  • smmap: 3.0.2 -> 4.0.0
  • snowballstemmer: 2.0.0 -> 2.2.0
  • soupsieve: 2.0.1 -> 2.3.1
  • sphinx: 3.1.2 -> 4.2.0
  • sphinx-argparse: 0.2.5 -> 0.3.1
  • sphinxcontrib-htmlhelp: 1.0.3 -> 2.0.0
  • sphinxcontrib-serializinghtml: 1.1.4 -> 1.1.5
  • sqlite: 3.32.3 -> 3.37.0
  • sqlparse: 0.3.1 -> 0.4.1
  • tbb: 2020.0 -> 2021.5.0
  • terminado: 0.8.3 -> 0.9.4
  • testpath: 0.4.4 -> 0.5.0
  • threadpoolctl: 2.1.0 -> 2.2.0
  • tk: 8.6.10 -> 8.6.11
  • toml: 0.10.1 -> 0.10.2
  • tornado: 6.0.4 -> 6.1
  • tqdm: 4.47.0 -> 4.62.3
  • traitlets: 4.3.3 -> 5.1.1
  • typing_extensions: 3.7.4.2 -> 3.10.0.2
  • urllib3: 1.25.9 -> 1.26.7
  • wheel: 0.34.2 -> 0.37.1
  • wrapt: 1.11.2 -> 1.13.3
  • zeromq: 4.3.2 -> 4.3.4
  • zipp: 3.1.0 -> 3.7.0
  • zstd: 1.4.5 -> 1.5.0

ska3-core-latest changes (2021.4 -> 2022.2rc6)

New Packages

  • astroquery
  • backports
  • backports.zoneinfo
  • openpyxl
  • ply
  • regions

ska3-flight-latest changes (2021.2 -> 2022.2rc6)

Removed Packages

  • acisfp_check
  • bep_pcb_check
  • dea_check
  • dpa_check
  • fep1_actel_check
  • fep1_mong_check
  • psmc_check

ska3-perl changes (2020.14 -> 2022.2rc6)

New Packages

  • fonts-anaconda: 1
  • fonts-conda-ecosystem: 1
  • intel-openmp: 2021.4.0
  • mkl: 2021.4.0
  • mkl_fft: 1.3.1

Removed Packages

  • krb5
  • libcurl
  • libssh2

Updated Packages

  • xorg-libx11: 1.6.9 -> 1.6.12
  • xorg-libxft: 2.3.2 -> 2.3.4

Related Issues

Fixes #686
Fixes #756
Fixes #757
Fixes #758
Fixes #759
Fixes #760
Fixes #761
Fixes #762
Fixes #768
Fixes #769
Fixes #770
Fixes #771
Fixes #773
Fixes #778
Fixes #780
Fixes #781
Fixes #782
Fixes #783
Fixes #784

@javierggt javierggt force-pushed the 2022.2 branch 2 times, most recently from 3170aae to 2bb7512 Compare January 28, 2022 20:24
@taldcroft
Copy link
Member

The instructions for creating the environment don't work, where the first error I see is that proseco-5.5.0 is not available.
After some playing around I found this would resolve everything, but it probably isn't right.

mamba create -n ska3-flight-2022.2rc3 --override-channels \
 -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/next  \
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/test \
-c https://icxc.cfa.harvard.edu/aspect/ska3-conda/masters \ 
  ska3-flight==2022.2rc3 ska3-core=2022.2rc3

@javierggt
Copy link
Contributor Author

The latest packages (proseco, ska.matplotlib, sparkles, starcheck, testr) had not been promoted from masters yet. Fixed.

@taldcroft
Copy link
Member

Confirmed it works now thanks. In the boilerplate create command (conda create -n ska3-flight-2022.2rc# ...) in the description can you replace conda with mamba (in general)? That will make would-be testers happier.

@taldcroft
Copy link
Member

And sorry, not sure where to comment on the template, but "ska3-flight 2022.2 will be promoted to flight conda channel and installed on HEAD and GRETA Linux after this week's loads are approved." should be more like "ska3-flight 2022.2 will be promoted to flight conda channel and installed on HEAD and GRETA Linux at a time outside of the load review cycle."

Since FSDS is not tied to load review, all we do now is not promote between when loads are released and when they are approved.

@javierggt
Copy link
Contributor Author

in the description can you replace conda with mamba (in general)?

That will work only if they have mamba installed already, which usually means you have a prior version of ska3-flight. I can change it, because it would work for us three. Just something to keep in mind.

@javierggt
Copy link
Contributor Author

And sorry, not sure where to comment on the template

an issue. Also the template for release issues requires update.

@taldcroft
Copy link
Member

That will work only if they have mamba installed already

Ah right. Never mind.

@jeanconn
Copy link
Contributor

jeanconn commented Feb 3, 2022

Though maybe we should go ahead and make a "clean install" a two step procedure that installs mamba first and then mamba installs the rest.

@taldcroft taldcroft mentioned this pull request Feb 4, 2022
2 tasks
@javierggt javierggt force-pushed the 2022.2 branch 2 times, most recently from 9eaf75a to 70abd2d Compare February 8, 2022 15:41
@javierggt javierggt merged commit 111822f into master Mar 10, 2022
@javierggt javierggt deleted the 2022.2 branch April 21, 2022 13:58
@javierggt javierggt mentioned this pull request May 18, 2022
@javierggt javierggt mentioned this pull request Aug 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment