Poetry dependency manager #2704

mikejhuang · 2023-07-21T03:05:00Z

Use Poetry's dependency resolver to solve dependency cross compatible versioning.

Steps

Use poetry to setup pyproject.toml file with python = ">=3.8,<3.12"
Create a copy of requirements.txt without explicit versions at poetry_py38-311.txt
poetry add `cat poetry_py38-311.txt`
poetry does not resolve python version requirements and this needs to be adjusted manually
xarray, scipy, and numpy versions adjusted to meet python version requirements
poetry add `cat poetry_py38-311.txt` completed successfully and created solution of valid dependency ranges in pyproject.toml
All dependencies are installed, poetry.lock file stores the actual installed versions for reproducibility
Run tests by creating virtualenv with 3.8, 3.9, 3.10, 3.11
For failing tests, either fix code or revert package version
Final explicitly defined requirements

Fixes for compatibility

pandas 1.1.5 -> 2.0.3
- Int64Index deprecated DEPR: Remove Int64Index, UInt64Index, Float64Index and Index._is_backward_compat_public_numeric_index pandas-dev/pandas#42717
- pd.Series.str.replace must specify regex=True if regex is used.
- pd.Series.reindex with a MultiIndex can no longer take in single integer Index as an input
- check_less_precise is a deprecated parameter in pd.testing.assert_series_equal. This is replaced with r_tol and a_tol
numpy 1.23.5
- np.bool, np.float, np.int are deprecated. Use 'bool', 'float', or 'int' instead.

Reverting versions instead of fixing

hdmf=<3.4.7
- hdmf==3.7.0: ValueError: io has already been set for this container (name=root, type=<class 'pynwb.file.NWBFile'>)
- hdmf=>3.5.0 ValueError: Invalid dataset identifier (invalid dataset identifier)
pandas<2.0.0
- padas==2.0.3: VisualBehaviorProjectCache.get_ophys_session_table( index_column="ophys_experiment_id").index is type 'object', expected int64
  All the values in this index are int64 so I'm not sure why it was loaded as an object type.

mikejhuang · 2023-07-24T23:28:27Z

Not going to lint this one since the changes made are all single-line edits that did not change the lint.

Modify Stimfile to add additional properties. Modify BS UUID from stimulus file method. Add timezone to behavior metdata table cloud api. Add datetime conversion. Change age_in_days to pandas.Int64 type. Fix imaging_plane_group typing. Input dummy time stamps for r and RMSE values. Enforce typing on image_plane_group_count. Add new Unittests Update existing unittests Add optional to stim file return. Add new NWBWiter subclass. Modify ophys nwb creation script. Add metadata table schema file. Add update function for targeted_imaging_depth Add new _update_session functino to nwb writer. Remove old writers and tests. Move base behavior schema

Add parser unittests. Move add_experience_level from vbn_2022

Add VBO project codes and stimulus_block maps Add project code to stimulus file processing. Add gray screen block at start of stimulus. Update write_nwb schemas for dir output.

Add dummy trials object to testing. Update presentations.pkl file. Update docstrings. update test pickle data.

Update tested python versions to 3.8 - 3.10

This reverts commit d637703.

Modify output path Add full output dict. Properly use output schema. Add behavior_session and ophys_experiment prefix.

LEFT JOIN on container due to potential missing container

Add link to issue with histogram.

Add _enforce typing function to enforce int types. remove fillna from image_plane_count Rename add_experince_level. Generalize experience_level calculation for edge cases. Add experience_level to all tables. Rename targeted_areas -> num_targeted_structures Fix typo in num_depths_per_area Enforce metadata table column order

Add column ordering and integer type fixing. Generalize ordering and typing across metadata and other tables. Change null values to -99 and use int type. Better enforce typing for calculated values in stimulus_processing. Update stimulus_block_name to properly handle training sessions. Fix bug in trials_id being added for grey screen stimulus Add blank stimulus to start of stimulus presentations where needed. Don't add trials_id if stimulus_block isn't there.

Add calculation to templates Modify templates to take/return list. Make movie loading optional. Move metadata_parsers Add index type to output dataframe of StimulusTemplates. Change Templates input to dictionary. Add default image and movie template keys. Add raise for illformed template input. Add/update docs. Update unittests. Add unittest for templates exception,

Debug dropping first omitted row. Debug for unittest and data without omitted.

Unpeg scikit-learn in test_requirements.txt

Remove scikit-learn from testing.

…cture in list

Debug dropping first omitted row. Debug for unittest and data without omitted.

Unpeg scikit-learn in test_requirements.txt

Remove scikit-learn from testing.

… on str.replace

mikejhuang force-pushed the poetry_dependency_manager branch 2 times, most recently from 29af88a to fca9216 Compare July 24, 2023 22:59

rly mentioned this pull request Jul 24, 2023

[Bug]: Container.__read_io not found if ContainerSubclass.__init__ raises error hdmf-dev/hdmf#913

Closed

3 tasks

mikejhuang requested review from aamster and morriscb July 25, 2023 00:15

mikejhuang changed the base branch from rc/2.15.2 to master August 2, 2023 06:24

morriscb and others added 23 commits August 1, 2023 23:58

Add session_type and project code parsing.

cd50b38

Add parser unittests. Move add_experience_level from vbn_2022

isort+black

64c23f4

Add stimulus_block_name column to stimulus_presentations.

17f3d0e

Add VBO project codes and stimulus_block maps Add project code to stimulus file processing. Add gray screen block at start of stimulus. Update write_nwb schemas for dir output.

Add trials_id creation to BehaviorSession.from_lims

c3e7928

Add dummy trials object to testing. Update presentations.pkl file. Update docstrings. update test pickle data.

isort+black

e9f1a8d

Changes to support python 3.10

1c19b40

Update tested python versions to 3.8 - 3.10

update setup

db70ea2

remove upper bound on matplotlib

d0ceb73

Revert "allow manually running nightly"

ce177cf

This reverts commit d637703.

set seed for df.sample

2896364

Update notebooks

5b777cd

Clean up NWB creation code.

7f13c53

Modify output path Add full output dict. Properly use output schema. Add behavior_session and ophys_experiment prefix.

isort+black

d46c6be

read session_type from db

4407881

LEFT JOIN on container due to potential missing container

remove logic that read session_type and daq from vbn code path

8392556

read date of acquisition from lims

550cef0

remove unused import

9501ffc

lint

f82959c

ignore column order in test

c5b2ae4

revert left join on ophys containers

9fd61ce

abstracts out logic to merge columns

54db7f6

fix bug due to duplicated project_code logic

5827255

Ahad-Allen and others added 28 commits August 2, 2023 00:01

Updating images to use PIL Image

c30f325

Add negative running speed check.

d198cfc

Add link to issue with histogram.

Add new information to templates data.

c6570bd

load probe lfp metadata without needing to load lfp

d161f41

Add removal of omitted stimulus to presentations.

5175183

Debug dropping first omitted row. Debug for unittest and data without omitted.

Unpeg statsmodels

1e59556

Unpeg scikit-learn in test_requirements.txt

Check for non-zero df length.

d6d254e

Remove scikit-learn from testing.

isort+black

c86dabf

set structure_id to primary_injection_structure instead of first stru…

e3ef243

…cture in list

Add removal of omitted stimulus to presentations.

2de5752

Debug dropping first omitted row. Debug for unittest and data without omitted.

Unpeg statsmodels

3c13e21

Unpeg scikit-learn in test_requirements.txt

Check for non-zero df length.

90c0e8d

Remove scikit-learn from testing.

isort+black

c0db510

lint

ed2c98a

lint

ed71f8e

poetry dependency setup for py3.8 to 3.10

fd12319

fix deprecated np.bool

3675589

fix Int64Index deprecation and explicit requirement for regex boolean…

2e1a63f

… on str.replace

fix pandas reindex change

83d771f

change check_less_precise to rtol in assert_series_equal

b04777b

fix attempt to reindex a MultiIndex

c3a72ab

fix deprecated np.float and np.int

b59b39b

downgrade pandas=1.53, hdmf=3.4.7, pynwb=2.3.3

45949e5

add python 3.11 to github actions

8aa90c9

add Python 3.11 in setup.py

3fefc44

mikejhuang force-pushed the poetry_dependency_manager branch from 407f5a5 to 3fefc44 Compare August 2, 2023 08:21

mikejhuang closed this Aug 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Poetry dependency manager #2704

Poetry dependency manager #2704

mikejhuang commented Jul 21, 2023 •

edited

Loading

mikejhuang commented Jul 24, 2023

Poetry dependency manager #2704

Poetry dependency manager #2704

Conversation

mikejhuang commented Jul 21, 2023 • edited Loading

Steps

Fixes for compatibility

Reverting versions instead of fixing

mikejhuang commented Jul 24, 2023

mikejhuang commented Jul 21, 2023 •

edited

Loading