MovingWindow: Create Interface for requesting data and handling the data copy behaviour

[matthias-wende-frequenz]

When asking the MovingWindow for a subwindow we might return a slice of the buffer, that means a reference to the underlying numpy based ringbuffer.
There are cases, e.g. when dealing with small window sizes and high sample rates, where the data might get updated from the data stream before the requested subwindow has been processed.

As a solution we want to tell the MovingWindow to return a copy of the data instead of a reference. In order to do that we need to add an interface, such that users can control copy behavior.

We might implement a method for the MovingWindow as follows.

window(
	self,
	start: datetime,
	end: datetime,
	fill_gaps: Optional[Quantity] = None,
	force_copy: Bool = False
)

This method should return a sorted numpy array that holds the resampled data from start to end and fills the gaps with the Value that is provided by fill_gaps.

---

Summary of the (ongoing) work related to this and what I see essential for v1.0.

Essential for this issue:

• For v1.0:

  • Treat NaN as gap in ring buffer #641
  • Add capacity, oldest and newest timestamp to moving window #598
  • Fixes on copy behavior in ring buffer window method #638
  • Support int indices and slice index behavior in ring buffer and moving window #668
  • Fix single element access for moving window #672
  • Fix getitem magic in moving window #676 (If not ready for v1.0 getitem magic should be removed before the release)

• Post v1.0?

  • Add option to impute missing values in window method for moving window #669.
  • Inconsistent gap treatment in ring buffer #646

Related to moving window:

• For v1.0
  • Replace len magic with count_valid method in ring buffer and moving window #664
  • Remove setitem magic from ring buffer #670
• Post v1.0?
  • Expose gaps in moving window #558

[matthias-wende-frequenz]

FYI @cwasicki @thomas-nicolai-frequenz

[thomas-nicolai-frequenz]

I don't understand why we could not design it in a way where the user passes his numpy array by reference the moving window is writing to. This way we could avoid the need for the fill_gaps parameter. In that case the numpy array could be initialised with whatever values the someone wants.

[cwasicki]

To me passing in an array feels less like typical behavior of other functions in numpy or pandas.

fill_gaps: Optional[Quantity] = None,

Since we should not have gaps in the moving window but only NaNs or uninitialized in the beginning, I would name this differently.

start: datetime,
end: datetime,

That should also work with index, not only datetime.

[matthias-wende-frequenz]

I don't understand why we could not design it in a way where the user passes his numpy array by reference the moving window is writing to. This way we could avoid the need for the fill_gaps parameter. In that case the numpy array could be initialised with whatever values the someone wants.

This is about the gaps that are coming from the resampler, e.g. due to faulty hardware.

Passing the np array is also planned but an independent issue (#530).

[thomas-nicolai-frequenz]

This is about the gaps that are coming from the resampler, e.g. due to faulty hardware.

Yes but in this case we would copy the NaN's that are coming from the resampler. This has to be treated differently I think but not 100% sure.

[matthias-wende-frequenz]

Yes but in this case we would copy the NaN's that are coming from the resampler. This has to be treated differently I think but not 100% sure.

You are right, we shouldn't copy NaN's. If fill_gaps is set to None and there are gaps then we should raise an exception.

[llucax]

Blocked because we need to fix some bugs in the ring buffer for this to be useful.

[cwasicki]

Blocked because we need to fix some bugs in the ring buffer for this to be useful.

@llucax Which bugs are those?

[cwasicki]

@llucax Which bugs are those?

Found it: #578

[llucax]

OK, if you need any further info about this, ask @matthias-wende-frequenz

[thomas-nicolai-frequenz]

@leandro-lucarella-frequenz but this bug is fixed and merged. Can we unblock the issue?

[llucax]

Done.

[cwasicki]

@llucax As discussed summary of the (ongoing) work related to this and what I see essential for v1.0.

Essential for this issue:

• For v1.0:

  • Treat NaN as gap in ring buffer #641
  • Add capacity, oldest and newest timestamp to moving window #598
  • Fixes on copy behavior in ring buffer window method #638
  • Support int indices and slice index behavior in ring buffer and moving window #668
  • Fix single element access for moving window #672
  • Fix getitem magic in moving window #676 (If not ready for v1.0 getitem magic should be removed before the release)

• Post v1.0?

  • Add option to impute missing values in window method for moving window #669.
  • Inconsistent gap treatment in ring buffer #646

Related to moving window:

• For v1.0
  • Replace len magic with count_valid method in ring buffer and moving window #664
  • Remove setitem magic from ring buffer #670
• Post v1.0?
  • Expose gaps in moving window #558

[cwasicki]

Apart from a very special bug (#646) all associated work has been merged.