Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RLS: 1.3 #40169

Closed
simonjayhawkins opened this issue Mar 2, 2021 · 97 comments
Closed

RLS: 1.3 #40169

simonjayhawkins opened this issue Mar 2, 2021 · 97 comments
Labels
Release
Milestone
1.3

Comments

@simonjayhawkins
Copy link
Member

Tracking issue for the 1.3 release.

https://github.com/pandas-dev/pandas/milestone/80

Currently scheduled for May 31, 2021

List of open regressions: https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3ARegression
@simonjayhawkins simonjayhawkins added this to the 1.3 milestone Mar 2, 2021
@simonjayhawkins
Copy link
Member Author

Currently scheduled for May 31, 2021

so we should be releasing the 1.3.0rc0 shortly (late next week or week after?), with the release of 1.3.0 a couple of weeks later

if we do 1.3.0 in mid/late June then from https://numpy.org/neps/nep-0029-deprecation_policy.html

On Dec 26, 2021 drop support for Python 3.7 (initially released on Jun 27, 2018)

The next release can be scheduled for 6 months, and we can drop 3.7 support in the next release.

v1.2.0 was released on Dec 26, 2020, so these dates seem consistent.

when v1.3.0rc0 is released we branch straight away so that we can continue to work on master during the rc period (this is how it was done for v1.2.0rc0)

so immediately after v1.3.0rc0 we can drop support for python 3.7 on master.

once v1.3.0rc0 is released, we will only backport regression fixes, maybe some bugfixes, maybe extra work on new enhancements and maybe also changes to experimental features.

when v1.3.0rc0 is released, we also tag master (v2.0.0.dev0 and add a whatsnew)

so immediately after v1.3.0rc0 we can start deprecating the deprecations introduced in 1.x (may need discussion regarding deprecations introduced late in the 1.x series)

@jreback
Copy link
Contributor

jreback commented May 26, 2021

can u open an issue about dropping 3.7

@simonjayhawkins
Copy link
Member Author

about doing it, or discuss whether to do it?

@jreback
Copy link
Contributor

jreback commented May 26, 2021

yeah to do it

@jreback
Copy link
Contributor

jreback commented Jun 4, 2021

@simonjayhawkins lets do the RC soon - waiting for the remaining pr labeled for 1.3 is not worth it

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jun 4, 2021 •
edited
Loading

hmm, we should get #39908 in. I think definitely worth it.

v1.2.0 was released on Dec 26, 2020 and the next release would ideally be after Dec 26, 2021 so we can drop Python 3.7

so a release around Jun 26, 2021, with a rc 2 weeks before around 12th gives us at least a few days to sort out the loose ends.

anytime next week would be consistent with these dates.

(and get 1.2.5 done first)

@simonjayhawkins
Copy link
Member Author

once v1.3.0rc0 is released, we will only backport regression fixes, maybe some bugfixes, maybe extra work on new enhancements and maybe also changes to experimental features.

I am planning to continue with some PyArrow string array changes after the rc (add arithimetic ops, min and max, more documentation and carry on with some perf improvements of the str accessor methods)

this list of open PRs milestone 1.3 https://github.com/pandas-dev/pandas/pulls?q=is%3Aopen+is%3Apr+milestone%3A1.3 contains some doc changes, which can also be done during the rc period.

#41718 is now ready and the last of the deprecations in #41485 with just a doc update outstanding #41485 (comment) that can be done during the rc.

#41654, #41586, #39908 and #35417 have behavioral changes to core functionality that if not merged before the rc. should probably wait to the next release.

We should be on course to release 1.3.0rc0 this week assuming we can finish up the 4 PRs mentioned above.

@lithomas1
Copy link
Member

@simonjayhawkins Will there be a 1.4 release? Looks like there is a milestone created on Github for it.

@jreback
Copy link
Contributor

jreback commented Jun 9, 2021

i created 1.4; it's possible we make this 2.0 at some point - not determined yet

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jun 10, 2021 •
edited
Loading

rc pre-release checklist

I think #41888 is the only open PR that is now outstanding, the others can be moved of the milestone or merged during rc

@simonjayhawkins
Copy link
Member Author

will combine these three into checking for "Blocker for rc" label, so use that label for anything that should be done for the rc otherwise will release.

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jun 11, 2021 •
edited
Loading

rc pre-release checklist

@jreback
Copy link
Contributor

jreback commented Jun 11, 2021

let's move all issues off of 1.3 except for the specifically mentioned ones. If there are no PRs then we are not waiting.

@simonjayhawkins
Copy link
Member Author

I'm going though now, removing some. We need visibility of the regressions (mainly performance) for 1.3. (the regression label alone is not enough since we have unresolved regressions from earlier releases)

@simonjayhawkins
Copy link
Member Author

reduced to 12 open issues. https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.3

includes some regressions (behavior and performance) on master and a new bug on master.

There are a few that can maybe be closed (awaiting responses or will look in more detail if no response).

If regressions are not fixed during rc they get milestoned 1.3.1.

There are no blockers for the actual release (have removed the blocker labels that were on some)

@simonjayhawkins
Copy link
Member Author

the overnight macOS py_3.7_64 failed on test_rename_inplace. restarted.

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jun 12, 2021 •
edited
Loading

  • ⬜ wheels building - final check

actually, I don't think this is needed in general for a rc. If the wheels fail to build and we need to make changes, we then create a new tag, for example v1.3.0rc1. (which we don't do for other releases)

only #41888 merged since nightly build

@simonjayhawkins
Copy link
Member Author

the overnight macOS py_3.7_64 failed on test_rename_inplace. restarted.

passed on restart. assume flakey.

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jun 12, 2021 •
edited
Loading

The release readiness scripts are failing on the conda test. would need to replicate locally as ci logs no help. Have not done a dummy run for this release so I will start the release (locally) and see if fixable along the way. (I'm assuming for now it's just the release scripts that need updating)

@simonjayhawkins
Copy link
Member Author

@simonjayhawkins
Copy link
Member Author

xref #41295 (comment). I'm going to do a few manual tags instead of using the scripts.

@simonjayhawkins
Copy link
Member Author

xref #41295 (comment). I'm going to do a few manual tags instead of using the scripts.

I'm not sure how clean my enviroment is. so will start again

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jun 12, 2021 •
edited
Loading

@pandas-dev/pandas-core starting release now. please do not merge anything to master

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jun 12, 2021 •
edited
Loading

@pandas-dev/pandas-core starting release now. please do not merge anything to master

can now resume merges to master for 1.4. do not merge anything with addition to 1.3 release notes unless being backported. (i.e regressions, new bugs on master, doc etc)

@simonjayhawkins
Copy link
Member Author

I like to have the wheels downloaded before doing the github release to reduce the lag between the any notifications from Github and the wheels being available.

image

i'm struggling with internet connectivity at the moment on 2 different networks. There's the G7 down the road so that could potentially be the reason. but with these upload speeds i'm gonna struggle getting the wheels uploaded to PyPI

The wheels have also doubled in size which is not helping either #38852 (comment)

will delay the github release while @fangchenli looks at the wheel size issue #38852 (comment)

if my connectivity is not improved by then (or I can't find another ISP with a better connectivity) then may need someone to upload the wheels to PyPI (the wheels are in the staging area)

@MarcoGorelli
Copy link
Member

if my connectivity is not improved by then (or I can't find another ISP with a better connectivity) then may need someone to upload the wheels to PyPI (the wheels are in the staging area)

I have until about 5:30 BST to do this if you want, do you have instructions?

@simonjayhawkins
Copy link
Member Author

I would need to get the dist uploaded somewhere and we need do the Github release before uploading to PyPI

on O2 i'm getting 2Mbps which is not great, but better than BT and Three. My release machine is headless. Will see if I can tether it and still access to O2 phone.

@phofl
Copy link
Member

phofl commented Jun 12, 2021

I could help as well if needed. If you need help during the evening, you can ping me

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jun 25, 2021 •
edited
Loading

I expect we will be releasing 1.3 sometime next week.

current status

https://github.com/pandas-dev/pandas/milestone/80

1 open PR #42068

4 open regressions #41997, #41999, #41998 and #42092 reported since rc

other open issues are mainly performance regressions.

@MarcoGorelli
Copy link
Member

@simonjayhawkins you'd suggest on #41283 to include that in 1.3 - is that still the case?

@simonjayhawkins
Copy link
Member Author

yep. i'll add the milestone.

@rhshadrach
Copy link
Member

#42286 should be an easy fix, I plan to get a PR up tonight.

@simonjayhawkins
Copy link
Member Author

current status

There are several open performance issues milestoned 1.3 but non are labelled blockers.

here are some rough timings to help put these into perspective (the 2 commits compared may require more samples and we should be comparing with 1.2.5)

(pandas-dev) simon@T3630:~/pandas/asv_bench (master)$ asv compare v1.3.0.dev0 1.3.x --only-changed --sort ratio|head -n 20
       before           after         ratio
     [0f587028]       [f34376d1]
     <v1.3.0.dev0^0>       <1.3.x>  
+        540±30μs       41.5±0.5ms    76.86  frame_methods.Shift.time_shift(1)
+     4.80±0.08ms          180±1ms    37.54  index_object.IntervalIndexMethod.time_intersection(100000)
+      4.99±0.1ms          101±1ms    20.18  index_object.IntervalIndexMethod.time_intersection_one_duplicate(100000)
+     2.48±0.01ms       27.4±0.4ms    11.03  indexing.InsertColumns.time_assign_list_like_with_setitem
+         187±4μs      1.78±0.01ms     9.56  index_object.IntervalIndexMethod.time_intersection_one_duplicate(1000)
+     2.22±0.05μs       16.3±0.3μs     7.37  period.Indexing.time_get_loc
+     4.78±0.07ms       30.8±0.5ms     6.46  frame_methods.MaskBool.time_frame_mask_bools
+      89.2±0.4μs          459±7μs     5.14  inference.ToTimedelta.time_convert_int
+         283±3μs      1.29±0.02ms     4.54  stat_ops.FrameOps.time_op('prod', 'int', 1)
+      83.2±0.5ms          378±3ms     4.54  frame_ctor.FromDicts.time_nested_dict_int64
+      4.20±0.1μs       18.1±0.2μs     4.31  indexing.NonNumericSeriesIndexing.time_getitem_scalar('period', 'non_monotonic')
+         265±4μs      1.09±0.01ms     4.13  stat_ops.FrameOps.time_op('sum', 'int', 1)
+         518±4μs      1.84±0.01ms     3.55  stat_ops.FrameOps.time_op('mean', 'int', 1)
+         632±6μs      2.08±0.05ms     3.29  index_object.IntervalIndexMethod.time_intersection_both_duplicate(1000)
+        622±10μs      1.95±0.02ms     3.13  rolling.EWMMethods.time_ewm('Series', 1000, 'float', 'mean')
+         627±8μs      1.94±0.02ms     3.09  rolling.EWMMethods.time_ewm('Series', 10, 'float', 'mean')
+     1.44±0.06ms      4.29±0.07ms     2.97  stat_ops.FrameOps.time_op('var', 'int', 1)

There is an open PR to address some of these, #42293 so may have a better picture later today.

we have 1 issue labelled as a blocker, #41815 although we could maybe remove the blocker label since #42006 fixed the import issue.

behviour regressions, #42286 has an open PR to fix, #42292

1.3.1, https://github.com/pandas-dev/pandas/milestone/87 is scheduled for 3 weeks, July 25, 2021 so if no other blockers, any objections with moving the issues/PRs on 1.3.0 to 1.3.1 tomorrow and releasing either tomorrow/Thursday?

@jreback
Copy link
Contributor

jreback commented Jun 29, 2021

no objection

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jun 29, 2021 •
edited
Loading

pre-release checklist for v1.3.0

✅ wheels building MacPython/pandas-wheels#135
✅ release readiness passing https://github.com/simonjayhawkins/pandas-release/actions?query=workflow%3A%22Tag+Release%22
✅ release notes checked https://pandas.pydata.org/pandas-docs/dev/whatsnew/v1.3.0.html
✅ ci checks https://github.com/pandas-dev/pandas/commits/1.3.x
✅ no manual backports outstanding https://github.com/pandas-dev/pandas/pulls?q=is%3Apr+label%3A%22Still+Needs+Manual+Backport%22+is%3Aclosed
✅ no open issues/PRs (except this one) https://github.com/pandas-dev/pandas/milestone/80

@simonjayhawkins
Copy link
Member Author

pre-release checks passed, closed off milestone

@simonjayhawkins
Copy link
Member Author

simonjayhawkins commented Jul 2, 2021 •
edited
Loading

starting release now.

@simonjayhawkins
Copy link
Member Author 

HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Project size too large. Limit for project 'pandas' total size is 10 GB. See https://pypi.org/help/#project-size-limit

@jreback do we need to delete some old versions?

@jreback
Copy link
Contributor

jreback commented Jul 2, 2021

wow that's crazy

@jreback
Copy link
Contributor

jreback commented Jul 2, 2021

yeah start at the oldest and just delete wheels (leave source)

@jreback
Copy link
Contributor

jreback commented Jul 2, 2021

alt reach out and see if we can get it bumped ? but may take too long to delay

@simonjayhawkins
Copy link
Member Author

I don't have manage permissions for PyPI

@simonjayhawkins
Copy link
Member Author

yeah start at the oldest and just delete wheels (leave source)

or perhaps could remove the older pre-releases instead.

alt reach out and see if we can get it bumped ? but may take too long to delay

should I do this? If you have elevated permissions it may be better if you do this.

@rhshadrach
Copy link
Member

what about deleting pre-releases first if necessary?

@jreback
Copy link
Contributor

jreback commented Jul 2, 2021

good point
let me do

@jreback
Copy link
Contributor

jreback commented Jul 2, 2021

of course my pypi access is not working, so waiting them to reset the 2FA.

@jreback
Copy link
Contributor

jreback commented Jul 2, 2021

maybe @jorisvandenbossche can help out in the interim.

@adamjstewart
Copy link
Contributor

Source for 1.3.0 isn't available on PyPI, but it sounds like you're aware of that and working on fixing the problem!

@jreback jreback mentioned this issue Jul 2, 2021
Closed
3 tasks
@simonjayhawkins
Copy link
Member Author

[x] available on conda-forge https://anaconda.org/conda-forge/pandas/files

except pypy. will open issue.

@jorisvandenbossche
Copy link
Member

I am also only a "maintainer" of the pandas package on PyPI, and not an "owner". So Jeff, if you can recover your login that would be best. Or ask Wes.

alt reach out and see if we can get it bumped ? but may take too long to delay

should I do this? If you have elevated permissions it may be better if you do this.

Yes, we should do that. If you could open an issue for pandas requesting a larger storage limit, that would be great

@simonjayhawkins
Copy link
Member Author

Yes, we should do that. If you could open an issue for pandas requesting a larger storage limit, that would be great

from https://pypi.org/help/#project-size-limit

first remove any unnecessary releases or individual files to lower your overall project size.

we have lots of binaries for older release, pre-releases and old windows installers

If that is not possible, we can sometimes increase your limit

The above cleaning should free up plenty.

@jreback
Copy link
Contributor

jreback commented Jul 2, 2021

seems my 2FA is off for some reason, very odd, i filed an issue, but who knows how long this takes

@jreback
Copy link
Contributor

jreback commented Jul 2, 2021

xref pypi/support#1192

@auvipy
Copy link

auvipy commented Jul 3, 2021

removing old pre-releases would be good move IMHO

@jreback
Copy link
Contributor

jreback commented Jul 4, 2021

@simonjayhawkins old releases have been deleted - go for uploading 1.3

@simonjayhawkins
Copy link
Member Author

done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Release
Projects
None yet
Milestone
1.3
Development

No branches or pull requests