Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support fully automatic build #18

Closed
leofang opened this issue Jan 15, 2019 · 7 comments
Closed

support fully automatic build #18

leofang opened this issue Jan 15, 2019 · 7 comments

Comments

@leofang
Copy link
Contributor

leofang commented Jan 15, 2019 •
edited
Loading

In setup.py (and/or accompanying codes) we should:

  • detect existence of GPU and test if CUDA driver & toolkit are properly installed
  • detect existence of MPI library
  • detect existence of FFTW library
  • compile the CUDA code into ~/.ptycho_gui/ (can be avoided if we use RawKernel.compile() in the backend, see Add RawKernel.compile() method cupy/cupy#1889); this would also involve moving configure.sh to the backend.
  • detect current beamline and install beamline-specific packages for accessing databroker
  • Choose the right CuPy version to install if GPU is to be used
  • other things?
@leofang
Copy link
Contributor Author

leofang commented Jan 28, 2019

Related: leofang/ptycho/issues/7

@leofang
Copy link
Contributor Author

leofang commented Jan 31, 2019 •
edited
Loading

MPI related issues would be addressed by NSLS-II/lightsource2-recipes#534 (if we're doing conda install in the control network; pip install would remain a question).

@leofang
Copy link
Contributor Author

leofang commented Mar 21, 2019

  • detect existence of GPU and test if CUDA driver & toolkit are properly installed

I think this can be easily achieved using functionalities from https://github.com/cupy/cupy/blob/master/install/build.py

@leofang
Copy link
Contributor Author

leofang commented Mar 21, 2019

Could also be useful: https://github.com/cupy/cupy/blob/86c22271d0b417e6cfd7b539e160c050752c95f9/cupy_setup_build.py#L606

@leofang
Copy link
Contributor Author

leofang commented Mar 29, 2019

New thought: keep all .cubin files in ptycho_gui/core/ptycho. The .so file generated by Cython is also kept there, so making the package self-contained rather than compiling to user's home dir may be a better idea for distributing the package in Conda.

This was referenced Mar 29, 2019
Closed
Closed
leofang added a commit that referenced this issue May 9, 2019
@leofang
add a script to automatically build cuda sources
4563900 
1. this is an ongoing effort to address #18
2. to use this, do:

import build_cuda_source
a = build_cuda_source.build()

a is a list of generated cubin files
leofang added a commit that referenced this issue May 10, 2019
@leofang
bugs fixed in build_cuda_source.py
a37ccbe 
This is part of ongoing efforts addressing #18

The key function name is changed from 'build' to 'compile':

    import build_cuda_source
    a = build_cuda_source.compile()
leofang added a commit that referenced this issue May 10, 2019
@leofang
move build_cuda_source from here to backend; compile .cubin during bu…
e725322 
…ild; bump version

This is part of ongoing efforts addressing #18. This allows the .cubin
files be generated automatically based on the installed CUDA toolkit
version.
leofang added a commit that referenced this issue May 10, 2019
@leofang
add cupy (with right version) as dependency if needed
80853c7 
This is part of ongoing effort addressing #18
@leofang
Copy link
Contributor Author

leofang commented May 11, 2019

See the progress report in the conda_build branch here: NSLS-II/lightsource2-recipes#486 (comment)

@leofang leofang mentioned this issue May 11, 2019
Merged
leofang added a commit that referenced this issue May 11, 2019
@leofang
add a script to automatically build cuda sources
55dd38f 
1. this is an ongoing effort to address #18
2. to use this, do:

import build_cuda_source
a = build_cuda_source.build()

a is a list of generated cubin files
leofang added a commit that referenced this issue May 11, 2019
@leofang
bugs fixed in build_cuda_source.py
af3e844 
This is part of ongoing efforts addressing #18

The key function name is changed from 'build' to 'compile':

    import build_cuda_source
    a = build_cuda_source.compile()
leofang added a commit that referenced this issue May 11, 2019
@leofang
move build_cuda_source from here to backend; compile .cubin during bu…
649a971 
…ild; bump version

This is part of ongoing efforts addressing #18. This allows the .cubin
files be generated automatically based on the installed CUDA toolkit
version.
leofang added a commit that referenced this issue May 11, 2019
@leofang
add cupy (with right version) as dependency if needed
83fee2d 
This is part of ongoing effort addressing #18
@leofang leofang mentioned this issue May 13, 2019
Merged
@leofang
Copy link
Contributor Author

leofang commented May 13, 2019

As of v1.0.3, the GPU issues are mostly resolved at the pip level, while the mpi and fftw issues are solved at the Conda level (see PR at NSLS-II/lightsource2-recipes/pull/486). For Conda users, therefore, this issue may be considered resolved. For pip users, it is just a matter of checking if external libraries exist. There's not much we can do at the pip level. Closing this.

@leofang leofang closed this as completed May 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@leofang