Pandas Integration

[fzhu2e]

This is to add some ideas to the milestone with the same title. Essentially the question @CommonClimate asked:"What pandas capabilities would you find most useful in Pyleoclim, which weren’t doable or too tedious before?"

I have the below ideas in my mind:
1.1. One capability that pandas has, I’ve been thinking for a while, is the operator overloading to support convenient basic calculations between two series, such as +, -, *, /. One example is that we want to remove the mean of a series.
1.2. With operator overloading, we may also create a convenient way to create a MultipleSeries from multiple Series objects like ms = ts1 & ts2 & ts3 and ms2 = ms1 - ts.
2. Another capability I find very useful and has already been implemented in cfr is the annualize(months=...) method, which is based on xarray but essentially a delicate usage of a couple of pandas methods. It enables seasonalization of a monthly series with arbitrary seasons by specifying months, e.g., JJA average w/ months=[6, 7, 8], and DJF w/ months=[12, 1, 2].

Regarding point 1.2, we need to think about how to deal with the metadata when combining multiple Series objects. Note that this issue is not tied to this new approach of MulitpleSeries creation; it exists with the old approach (using the MultipleSeries.series_list attribute) as well.

Welcome to share your thoughts on this topic.

[alexkjames]

A couple of thoughts below, building on Feng's post:

1. Something I find myself doing often when testing new methods with synthetic data is replacing subsections of Series objects with other Series objects. That is, replacing the time and values of a subsection of a Series with the time and values of another Series (typically to introduce some sort of unusual behavior into one section of a time series). I'm not sure if this is much easier to do in pandas, but it is something I've been doing a lot lately. It would also make the matter of building composite records much more straightforward, which could also be a plus for some people. I suppose my ask here is being able to more flexibly and easily combine series objects with one another (indexing by time while doing so).
2. Another functionality that would be useful would be the ability to interpolate over nan values. As it stands Pyleoclim ignores nans when performing interpolation because it's built on scipy's interp1d. It would be nice if we could specify that we'd like nans to be filled in during the interpolation step. Currently they can be dropped and then filled in via pyleoclim interpolation, but this tends to introduce uncertainty in the time axis that wasn't present before because time points need to be dropped and then recreated by the interpolation method (which doesn't always do precisely what you'd like it to).

I'm not intimately familiar with all of pandas capabilities, so it's entirely possible that pandas doesn't make some or all of these things easier, just throwing them out there!

[CommonClimate]

Thanks @alexkjames . For 1, are you thinking of something of the form ts1 & ts2 where the values/times are joined together (in chronological order, of course)? Marco would tell you to think about how metadata from both series would need to be combined, then. If they are the same, no problem, but otherwise the combination would need rules. What would those be?

Re: 2, this is definitely on our end; I suggest you open an issue about it so we think of an appropriate response. For many tasks if it simply easier to drop NaNs, but I agree that other applications need to original data. What rules would you put in place for that?