Add implementation of Functional Neural Process #187

dereklhansen · 2021-01-22T21:15:09Z

This adds an implementation of the Functional Neural Process (FNP) to the repo.

In addition to the implementation of FNP, this PR currently adds two working examples of the FNP on synthetic data (located in 'case_studies/fnp':

A one-dimensional example which illustrates different variants of the FNP
An example on generated star-like objects which exhibit a counter-clockwise rotation

This commit moves a lot of the code that was in the onedim.py file into the main fnp.py file. This lets us write tests using the one dimensional FNP example Squashed commit of the following: commit 0db2d3e032d9e8fd4481a8311379234b57c6384f Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 16:06:36 2021 -0500 Incorporated the one dimensional example as a unit test for FNP commit cda0ce36ff39f49cc2c19ef27dba796ceaf47740 Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 15:37:44 2021 -0500 Got the RegressionFNP tested in one dimensional case commit 754c9093abae98915db50da1e81bbd1e0a3d6450 Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 14:17:46 2021 -0500 Moved onedim training and visualize to fnp.py commit 41aa5921416fcded2a55a97ef208dc0baf934fd1 Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 14:11:03 2021 -0500 Made od an argument rather than global commit 5dd7c50be4479d8c8b5bbf569d747e36c21ed75b Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 14:06:23 2021 -0500 Moved OneDimDataset into fnp.py commit 263f8397be6ecba7faba758691ae501fa6b8161c Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 14:01:56 2021 -0500 ensure that the indices match those from before commit 2c22b0d438a6fb81a76ceb0eb988ff48f24d4917 Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 13:51:59 2021 -0500 Deleted old onedim dataset code commit dc90af2fc86ab50e14d90a77e02f329456d3496e Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 13:18:31 2021 -0500 Moved onedim dataset into a class

Squashed commit of the following: commit d7695879fad3f469305df2bb0b3c09e49b33a425 Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 18:38:32 2021 -0500 Added test of star rotation commit 5ef52ce57e9dcef42ef4421033a4c41a6d7b153e Author: Derek Hansen <dereklh@umich.edu> Date: Fri Jan 22 17:26:01 2021 -0500 Moved rotate_dgp.py code into fnp.py

codecov · 2021-01-22T23:51:26Z

Codecov Report

Merging #187 (80dbbaf) into master (bd4ef0b) will increase coverage by 0.29%.
The diff coverage is 97.25%.

@@            Coverage Diff             @@
##           master     #187      +/-   ##
==========================================
+ Coverage   96.29%   96.58%   +0.29%     
==========================================
  Files          13       15       +2     
  Lines        1672     1963     +291     
==========================================
+ Hits         1610     1896     +286     
- Misses         62       67       +5

Flag	Coverage Δ
unittests	`96.58% <97.25%> (+0.29%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
bliss/utils.py	`96.77% <96.77%> (ø)`
bliss/models/fnp.py	`97.37% <97.37%> (ø)`
bliss/sleep.py	`96.28% <0.00%> (+1.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd4ef0b...02b8a23. Read the comment docs.

dereklhansen · 2021-02-04T14:52:41Z

Remove the sklearn dependency since it's only used for the StandardScaler() in the test code

bliss/models/fnp.py

jeff-regier

Nice. This is very readable code. Good to wait for reviews from @ismael-mendoza and @Runjing-Liu120 before merging since this is such a major PR (although much more manageable now that I see how it is organized)

jeff-regier · 2021-02-05T16:28:09Z

Looks good to me. @ismael-mendoza @Runjing-Liu120 if you have feedback please try to weigh in sometime today

Runjing-Liu120 · 2021-02-05T18:36:20Z

Looks good to me, too. The hard part will be figuring out how to add this to our wake.py / decoder.py files -- we'll have to think carefully about that. But this is great so far!

ismael-mendoza · 2021-02-05T18:40:44Z

Hi @jeff-regier and @dereklhansen , I'll be done with my review shortly

dereklhansen · 2021-02-05T18:48:16Z

There seems to be a problem with the CI environment that is causing the latest tests to fail. The latest commit I pushed shouldn't have caused this error.

I found a bug report on the poetry repo (issue 3650)

ismael-mendoza

Looks really good @dereklhansen, thanks for adding this code. I just had a couple of suggestions/questions.

bliss/models/fnp.py

tests/test_fnp.py

dereklhansen · 2021-02-05T19:01:04Z

I added a fix for the poetry CI issues in #210

All math functions can be taken from numpy instead FNP.encode didn't actually need to have two separate pathways for inference and prediction

dereklhansen added 11 commits January 21, 2021 14:38

Initial commit of FNP code with one dimensional example

7c59e62

Imported rotating star code

0b7f478

reformatted imported code with black

7175f90

more changes to get rotate.py running

1adc6c5

Use apngasm to assemble animated PNG of samples

df2882f

Updated default rotate.py settings to match earlier

e6fbac1

Decorated 1-dimensional example comment

6659554

bumped up error-reduction threshold for deep set FNP

2aa41db

Lowered the number of epochs in onedim test

a8a26fc

black formatting fix

d03ee9e

dereklhansen force-pushed the derek/fnp branch from e6848a7 to d03ee9e Compare January 28, 2021 15:21

dereklhansen added 16 commits January 28, 2021 10:26

Merge remote-tracking branch 'origin/master' into derek/fnp

c1a2633

Added commented out goal for the interface

791cf6c

Deleted unused skip and residual layer code

6e9ae38

Removed more unreachable code; black fixes

9d2f483

fixed indentation error

093728d

First step in removing separate PoolingFNP

4b8c9b5

deleted unneeded function

9fdd502

Removed train_separate_extrapolate option

2568990

Made make_fc_net an MLP module

577a563

Removed unused train_separate_proposal option

0a2398b

Removed train_separate_extrapolate from rotate case study

4ad55ab

Merge remote-tracking branch 'origin/master' into derek/fnp

3b882d8

start to removing mu_nu_theta

2f25b94

Refactored out all method overrides of PoolingFNP

b694e21

pulled most complexity into SetPooler class

b9b146b

Removed calc_trans_cond_y from poolers

56ce94f

dereklhansen added 3 commits February 4, 2021 09:10

delete comment

8c1f74a

Removed star rotation test

184543e

fixed linter issues

24c5b57

dereklhansen added 5 commits February 4, 2021 10:07

Remove untested code related to convnets

7fe1c0b

Move utility modules to utils.py

ece3e95

Added some comments and attributions from imported code

ed1a54a

Improve MLP by removing beginning special case

5f3994d

Removed StandardScaler and sklearn dependency

88264d1

jeff-regier reviewed Feb 4, 2021

View reviewed changes

bliss/models/fnp.py Show resolved Hide resolved

jeff-regier approved these changes Feb 4, 2021

View reviewed changes

dereklhansen added 3 commits February 4, 2021 11:44

Moved FNP test to lightning module

bf359d0

Add attention block docstrings and delete unused ISAB

59ee488

Updated Sphinx and added documentation

f3acb7c

Runjing-Liu120 approved these changes Feb 5, 2021

View reviewed changes

Fixed bug in prediction when not sampling

ef8a8ba

ismael-mendoza approved these changes Feb 5, 2021

View reviewed changes

dereklhansen added 5 commits February 5, 2021 14:14

Replace math with np; refactor FNP.encode

396962d

All math functions can be taken from numpy instead FNP.encode didn't actually need to have two separate pathways for inference and prediction

Merge remote-tracking branch 'origin/master' into derek/fnp

b1b3242

Refactored g in DepGraph to be more readable

f96ce50

remove extraneous comments

49a548f

Remove pytorch issue and use log1p

02b8a23

dereklhansen merged commit b457e23 into master Feb 5, 2021

dereklhansen deleted the derek/fnp branch February 5, 2021 20:16

dereklhansen mentioned this pull request Feb 5, 2021

Use MultiheadAttention instead of MAB #211

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add implementation of Functional Neural Process #187

Add implementation of Functional Neural Process #187

dereklhansen commented Jan 22, 2021

codecov bot commented Jan 22, 2021 •

edited

Loading

dereklhansen commented Feb 4, 2021

jeff-regier left a comment

jeff-regier commented Feb 5, 2021

Runjing-Liu120 commented Feb 5, 2021

ismael-mendoza commented Feb 5, 2021

dereklhansen commented Feb 5, 2021 •

edited

Loading

ismael-mendoza left a comment

dereklhansen commented Feb 5, 2021

Add implementation of Functional Neural Process #187

Add implementation of Functional Neural Process #187

Conversation

dereklhansen commented Jan 22, 2021

codecov bot commented Jan 22, 2021 • edited Loading

Codecov Report

dereklhansen commented Feb 4, 2021

jeff-regier left a comment

Choose a reason for hiding this comment

jeff-regier commented Feb 5, 2021

Runjing-Liu120 commented Feb 5, 2021

ismael-mendoza commented Feb 5, 2021

dereklhansen commented Feb 5, 2021 • edited Loading

ismael-mendoza left a comment

Choose a reason for hiding this comment

dereklhansen commented Feb 5, 2021

codecov bot commented Jan 22, 2021 •

edited

Loading

dereklhansen commented Feb 5, 2021 •

edited

Loading