Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: .dt.time() was panicking for datetimes prior to unix epoch #13812

Merged
merged 3 commits into from
Jan 20, 2024
Merged

fix: .dt.time() was panicking for datetimes prior to unix epoch #13812

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c569b87
fix: .dt.time() was panicking for datetimes prior to unix epoch
MarcoGorelli Jan 18, 2024
245a87f
SimpleImpl
mcrumiller Jan 19, 2024
f7419ad
Merge pull request #8 from mcrumiller/man-on-the-mooner
MarcoGorelli Jan 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 33 additions & 15 deletions crates/polars-core/src/chunked_array/logical/datetime.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,21 +81,39 @@ impl LogicalType for DatetimeChunked {
},
#[cfg(feature = "dtype-time")]
(Datetime(tu, _), Time) => match tu {
TimeUnit::Nanoseconds => Ok((self.0.as_ref() % NS_IN_DAY)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the issue with the current implementation is that self.0.as_ref() % NS_IN_DAY might return a negative value

.cast(&Int64)
.unwrap()
.into_time()
.into_series()),
TimeUnit::Microseconds => Ok((self.0.as_ref() % US_IN_DAY * 1_000i64)
.cast(&Int64)
.unwrap()
.into_time()
.into_series()),
TimeUnit::Milliseconds => Ok((self.0.as_ref() % MS_IN_DAY * 1_000_000i64)
.cast(&Int64)
.unwrap()
.into_time()
.into_series()),
TimeUnit::Nanoseconds => {
let ca = self.0.apply_values(|v| {
Copy link
Collaborator Author

@MarcoGorelli MarcoGorelli Jan 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I timed this separately, and apply_values is faster here than apply

Copy link
Contributor

@mcrumiller mcrumiller Jan 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit more concise (I'm always looking to reduce code length) and possibly faster since it removes the if branch (although maybe a premature optimization). Also I think we're already at i64 so we can remove the extra .cast(&Int64).unwrap().

            #[cfg(feature = "dtype-time")]
            (Datetime(tu, _), Time) => Ok({
                let (modder, multiplier) = match tu {
                    TimeUnit::Nanoseconds => (NS_IN_DAY, 1i64),
                    TimeUnit::Microseconds => (US_IN_DAY, 1_000i64),
                    TimeUnit::Milliseconds => (MS_IN_DAY, 1_000_000i64),
                };
                self.0
                    .apply_values(|v| (v % modder * multiplier) + (NS_IN_DAY * (v < 0) as i64))
                    .into_time()
                    .into_series()
            }),

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thinks this is a nice improvement. @mcrumiller 👍

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup - @mcrumiller fancy opening a PR to my PR, so you're included in the commit?

Copy link
Contributor

@mcrumiller mcrumiller Jan 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure (although I don't care about the cred)--need to figure out how to check out a branch on another fork first.

Edit: I used git remote add <Marco's fork url> followed by a checkout of his branch. Not sure if that's the best way to do it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good learning experience then :) but yes, that's how I'd have done it

alternatively,

git checkout -b pr/marcogorelli/man-on-the-moon
git fetch marcogorelli
git reset --hard marcogorelli/man-on-the-moon
<do your changes>
git push -U marcogorelli HEAD:man-on-the-moon

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got that working, now trying to figure out how to do create the PR, as your fork doesn't seem to show up in git's list:

image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edit: w00t

let v = v % NS_IN_DAY;
if v < 0 {
v + NS_IN_DAY
} else {
v
}
});
Ok(ca.cast(&Int64).unwrap().into_time().into_series())
},
TimeUnit::Microseconds => {
let ca = self.0.apply_values(|v| {
let v = v % US_IN_DAY * 1_000i64;
if v < 0 {
v + NS_IN_DAY
} else {
v
}
});
Ok(ca.cast(&Int64).unwrap().into_time().into_series())
},
TimeUnit::Milliseconds => {
let ca = self.0.apply_values(|v| {
let v = v % MS_IN_DAY * 1_000_000i64;
if v < 0 {
v + NS_IN_DAY
} else {
v
}
});
Ok(ca.cast(&Int64).unwrap().into_time().into_series())
},
},
_ => self.0.cast(dtype),
}
Expand Down
14 changes: 12 additions & 2 deletions py-polars/tests/unit/namespaces/test_datetime.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,18 @@ def test_local_datetime_sortedness(time_zone: str | None, expected: bool) -> Non
def test_local_time_sortedness(time_zone: str | None) -> None:
ser = (pl.Series([datetime(2022, 1, 1, 23)]).dt.replace_time_zone(time_zone)).sort()
result = ser.dt.time()
assert result.flags["SORTED_ASC"] is False
assert result.flags["SORTED_DESC"] is False
assert result.flags["SORTED_ASC"]
assert not result.flags["SORTED_DESC"]
Comment on lines -147 to +148
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one should've been sorted to begin with, but that's already being dealt with in #13808



@pytest.mark.parametrize("time_unit", ["ms", "us", "ns"])
def test_local_time_before_epoch(time_unit: TimeUnit) -> None:
ser = pl.Series([datetime(1969, 7, 21, 2, 56, 2, 123000)]).dt.cast_time_unit(
time_unit
)
result = ser.dt.time().item()
expected = time(2, 56, 2, 123000)
assert result == expected


@pytest.mark.parametrize(
Expand Down