Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Project Refactoration #23

Closed
wants to merge 12 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 24 additions & 51 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,64 +2,37 @@ name: 'Test'

on: [push, pull_request, workflow_dispatch]


jobs:
test:
runs-on: ubuntu-18.04
runs-on: ${{ matrix.os }}
timeout-minutes: 10
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ['3.8', '3.9', '3.10']

steps:
- name: Check out code
uses: actions/checkout@v2

- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: "3.9"

- name: Cache pip
uses: actions/cache@v2
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-

- name: Install dependencies
run:
python -m pip install -r tests/requirements.txt
- uses: actions/checkout@v2

- name: Run pyright tests
uses: gramster/pyright-action@main
with:
project: pyrighttestconfig.json
warn-partial: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}

- name: Run pytest
run:
pytest tests/pandas

- name: Run mypy
run:
mypy tests/pandas typings/pandas

- name: Build wheel and install and remove typings
run: |
python setup.py build bdist_wheel
find ./dist/*.whl | xargs pip install
rm -rf typings
pip install pyright

# - name: run pyright command line
# run:
# pyright -p pyrightdistconfig.json --dependencies
- name: Upgrade pip
run: |
python -m pip install --upgrade pip
- name: Install Poetry and project dependencies
run: |
pip install poetry
poetry install
- name: Run Pyright
run: poetry run pyright

# - name: Run pyright against dist
# uses: gramster/pyright-action@main
# with:
# project: pyrightdistconfig.json
# extra-args: --dependencies
# warn-partial: true
- name: Run Pytest
run: poetry run pytest

- name: Run mypy against dist
run:
mypy tests/pandas
- name: Run MyPy
run: poetry run mypy pandas-stubs tests
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One issue that I have run into with respect to using the more traditional method of a build using setup.py is that the wheel that is built, when installed, isn't passing the tests. So if you look at https://github.com/pandas-dev/pandas-stubs/blob/main/.github/workflows/test.yml , you will see this in there:

     - name: Build wheel and install and remove typings
        run: |
          python setup.py build bdist_wheel
          find ./dist/*.whl | xargs pip install
          rm -rf typings
          pip install pyright
      
      # - name: run pyright command line
      #   run:
      #     pyright -p pyrightdistconfig.json --dependencies

      # - name: Run pyright against dist
      #   uses: gramster/pyright-action@main
      #   with:
      #     project: pyrightdistconfig.json
      #     extra-args: --dependencies
      #     warn-partial: true

      - name: Run mypy against dist
        run:
          mypy tests/pandas

The problem right now is that pyright is failing with this, and there is a long discussion at microsoft/pyright#3540 , there seems to be an issue with the presence of the py.typed file that is interpreted differently on MacOS versus Windows and Linux.

In any case, adding the tests here to build the wheel, install it, and then test pyright and mypy against the installed wheel is something that should be done.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a lot of trouble figuring out the "py.typed" with "partial\n" string it was causing the problem, at least in my environment. not just in wheel installed case. So for a while, I just delete the file to show the poetry idea. I have the three Local OS here, I'm gonna test and see if I can help to fix the bug in the official pyright repository. To add the test with the wheel is not gonna be hard when the pyright is gonna work properly.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it turns out that if I remove py.typed, then everything works locally and with the installed wheel. That version is now in main. So remove py.typed from your version and it should be OK

1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,4 @@ dmypy.json

# Pyre type checker
.pyre/
/poetry.lock
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not important for now: It could be helpful to add poetry.lock to the repository (poetry install will take less time and it improves reproducibility) but every now and then someone needs to update poetry.lock (is there a github bot for this?).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm i see, probably a good ideia let poetry.lock available. But what do you mean with github bot, I didn't get the ideia.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If poetry.lock is added to the repository, it will eventually be out-of-date (for example not using the latest numpy). Instead of having to create a PR that updates poetry.lock, it would be handy if this process could be automated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh indeed , looks like a headache hahha. I think is better to keep that way, It could cause problems in CI/CD, too.

10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,14 @@ This is the home for pandas typing stubs supported by the pandas core team. The

To contribute changes to the stubs, you must include an appropriate test. See `pandas-stubs/tests` for examples.

## Build instructions
### Documentation

Use `python setup.py build bdist_wheel` to build the wheel. NOTE: `setuptools 62.3.2` is required!
- Enter in docs to see:
- 1 - How to setup the enviroment
- 2 - How to test the project
- 3 - How to follow the code style
- 4 - Security stuffs
- 5 - How to publish
Comment on lines +14 to +18
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you create links from here to the docs you created?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohh yeah, it is easier to see.


## Evolution

Expand All @@ -23,3 +28,4 @@ As both projects move forward, this page will track what the differences are (if
We are indebted to Microsoft and that project for the initial set of public type stubs. We are also grateful for the original pandas-stubs project at <https://github.com/VirtusLab/pandas-stubs> that created the framework for testing the stubs.

Last update to README: 6/4/2022: 11:35 EDT

7 changes: 7 additions & 0 deletions docs/1 - setup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
## Setup Environment

- Make sure you have python >= 3.8 installed
- Install poetry if you still don't have: pip install poetry
- Install the project dependencies with: poetry install -vvv
- Run all tests to make sure the project is ok: poetry run all_tests

3 changes: 3 additions & 0 deletions docs/2 - tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Test

- These tests originally came from https://github.com/VirtusLab/pandas-stubs.
5 changes: 5 additions & 0 deletions docs/3 - style.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
## Code style

- It's important to follow the code style from the project:
- poetry run black pandas-stubs tests scripts
- poetry run isort pandas-stubs tests scripts
3 changes: 3 additions & 0 deletions docs/4 - security.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Security

-
3 changes: 3 additions & 0 deletions docs/5 - publish.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Publish

You know ... just type "poetry publish pandas-stubs"
23 changes: 11 additions & 12 deletions typings/pandas/__init__.pyi → pandas-stubs/__init__.pyi
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import pandas.testing as testing

from ._config import (
describe_option as describe_option,
get_option as get_option,
Expand All @@ -7,6 +9,7 @@ from ._config import (
set_option as set_option,
)
from .core.api import (
NA as NA,
BooleanDtype as BooleanDtype,
Categorical as Categorical,
CategoricalDtype as CategoricalDtype,
Expand All @@ -19,18 +22,17 @@ from .core.api import (
Grouper as Grouper,
Index as Index,
IndexSlice as IndexSlice,
Int8Dtype as Int8Dtype,
Int16Dtype as Int16Dtype,
Int32Dtype as Int32Dtype,
Int64Dtype as Int64Dtype,
Int64Index as Int64Index,
Int8Dtype as Int8Dtype,
Interval as Interval,
IntervalDtype as IntervalDtype,
IntervalIndex as IntervalIndex,
MultiIndex as MultiIndex,
NA as NA,
NaT as NaT,
NamedAgg as NamedAgg,
NaT as NaT,
Period as Period,
PeriodDtype as PeriodDtype,
PeriodIndex as PeriodIndex,
Expand All @@ -40,11 +42,11 @@ from .core.api import (
Timedelta as Timedelta,
TimedeltaIndex as TimedeltaIndex,
Timestamp as Timestamp,
UInt8Dtype as UInt8Dtype,
UInt16Dtype as UInt16Dtype,
UInt32Dtype as UInt32Dtype,
UInt64Dtype as UInt64Dtype,
UInt64Index as UInt64Index,
UInt8Dtype as UInt8Dtype,
array as array,
bdate_range as bdate_range,
date_range as date_range,
Expand All @@ -61,10 +63,7 @@ from .core.api import (
unique as unique,
value_counts as value_counts,
)
from .core.tools import to_datetime as to_datetime, to_timedelta as to_timedelta
from .core.arrays.sparse import SparseDtype as SparseDtype
from .tseries import offsets as offsets
from .tseries.api import infer_freq as infer_freq
from .core.computation.api import eval as eval
from .core.reshape.api import (
concat as concat,
Expand All @@ -81,8 +80,7 @@ from .core.reshape.api import (
qcut as qcut,
wide_to_long as wide_to_long,
)
from .util._print_versions import show_versions as show_versions
from .io.json import json_normalize as json_normalize
from .core.tools import to_datetime as to_datetime, to_timedelta as to_timedelta
from .io.api import (
ExcelFile as ExcelFile,
ExcelWriter as ExcelWriter,
Expand All @@ -108,9 +106,10 @@ from .io.api import (
read_table as read_table,
to_pickle as to_pickle,
)

from .io.json import json_normalize as json_normalize
from .tseries import offsets as offsets
from .tseries.api import infer_freq as infer_freq
from .util._print_versions import show_versions as show_versions
from .util._tester import test as test

import pandas.testing as testing

__version__: str
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ from .config import (
option_context as option_context,
options as options,
reset_option as reset_option,
set_option as set_option)

set_option as set_option,
)
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
from contextlib import ContextDecorator
from typing import Any, Union, overload, Literal
from typing import Any, Literal, Union, overload

def get_option(pat: str) -> Any: ...
def set_option(pat: str, val: object) -> None: ...
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
from .interval import Interval as Interval
from .tslibs import (
NaT as NaT,
NaTType as NaTType,
Expand All @@ -7,4 +8,3 @@ from .tslibs import (
Timestamp as Timestamp,
iNaT as iNaT,
)
from .interval import Interval as Interval
Original file line number Diff line number Diff line change
@@ -1,22 +1,9 @@
from __future__ import annotations

from typing import (
Any,
Generic,
TypeVar,
Union,
overload,
)
from typing import Any, Generic, TypeVar, Union, overload

import numpy as np

from pandas._typing import npt

from pandas._typing import (
IntervalClosedType,
Timedelta,
Timestamp,
)
from pandas._typing import IntervalClosedType, Timedelta, Timestamp, npt

VALID_CLOSED: frozenset[str]

Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
from __future__ import annotations

from typing import Union

class NAType:
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,10 @@ __all__ = [
"OutOfBoundsDatetime",
]

from np_datetime import OutOfBoundsDatetime as OutOfBoundsDatetime

from .nattype import NaT, NaTType, iNaT, nat_strings
from .offsets import BaseOffset, Tick
from .period import Period
from .timestamps import Timestamp
from .timedeltas import Timedelta
from .nattype import (
NaT,
NaTType,
iNaT,
nat_strings,
)
from .offsets import BaseOffset, Tick
from np_datetime import OutOfBoundsDatetime as OutOfBoundsDatetime
from .timestamps import Timestamp
Original file line number Diff line number Diff line change
@@ -1,15 +1,7 @@
from datetime import (
datetime,
timedelta,
tzinfo as _tzinfo,
)
from typing import (
Any,
Union,
)
from datetime import datetime, timedelta, tzinfo as _tzinfo
from typing import Any, Union

import numpy as np

from pandas._libs.tslibs.period import Period

NaT: NaTType
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
from __future__ import annotations

from datetime import (
datetime,
timedelta,
)
from datetime import datetime, timedelta
from typing import (
TYPE_CHECKING,
Any,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
from __future__ import annotations

from datetime import datetime
from typing import Any

Expand Down
Original file line number Diff line number Diff line change
@@ -1,18 +1,8 @@
from datetime import timedelta
from typing import (
ClassVar,
Literal,
Type,
TypeVar,
overload,
)
from typing import ClassVar, Literal, Type, TypeVar, overload

import numpy as np

from pandas._libs.tslibs import (
NaTType,
Tick,
)
from pandas._libs.tslibs import NaTType, Tick
from pandas._typing import npt

# This should be kept consistent with the keys in the dict timedelta_abbrevs
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,10 @@ from datetime import (
tzinfo as _tzinfo,
)
from time import struct_time
from typing import (
ClassVar,
TypeVar,
overload,
)
from typing import ClassVar, TypeVar, overload

import numpy as np

from pandas._libs.tslibs import (
BaseOffset,
NaTType,
Period,
Tick,
Timedelta,
)
from pandas._libs.tslibs import BaseOffset, NaTType, Period, Tick, Timedelta

_DatetimeT = TypeVar("_DatetimeT", bound=datetime)

Expand Down
Loading