The repo is the official implemenation of Cinematic Behavior Transfer via NeRF-based Differentiable Filming.
We have VirtualFilmStudio to showcase some of our explorations in filmmaking directions.
git clone https://github.com/VirtualFilmStudio/Cinetransfer.git
We recommend you use anaconda to run our CinematicTransfer.
Please refer to SLAHMR work to install env. Notes:
- SLAHMR project have update, please use release branch.
- Use SLAHMR - Google Colab version to install is better. Conda env name will set to slahmr.
Prepare test data. Please place data according to the following framework.
|-Cinetransfer
| |-camera_optim
| |-data
| |-torch_ngp
| |-slahmr
| | |-demo
| | | |-videos
| | | | |-arc2.mp4
|...
Preprocess data by using SLAHMR.
cd {YOUR_ROOT}/Cinetransfer/slahmr/slahmr
# delete slahmr default test dataset
rm ../demo/videos/022691_mpii_test.mp4
# copy test data to slahmr demo
cp ../../data/videos/arc2.mp4 ../demo/videos/
# preprocess data by using SLAHMR
# If the server is not configured with virtual graphics, it is recommended to set 'run_vis=False'
source activate slahmr && python run_opt.py data=video data.seq=arc2 data.root={YOUR_ROOT}/Cinetransfer/slahmr/demo run_opt=True run_vis=True
After execution, an outputs folder will be generated. You can find a folder starting with arc2, which contains optimized smpl.
mkdir {YOUR_ROOT}/Cinetransfer/data/arc2
cd {YOUR_ROOT}/Cinetransfer/camera_optim
# Copy the slahmr preprocessed data to the Cinetransfer/data folder and adjust the data structure
source activate slahmr && python trans_slahmr_to_nerf.py --src_root {YOUR_ROOT}/Cinetransfer/slahmr/demo --dst_root {YOUR_ROOT}/Cinetransfer/data
# Generate mask data
source activate slahmr && python filmingnerf/preproc/gen_mask.py --data data --seq_name arc2 --data_path {YOUR_ROOT}/Cinetransfer
Copy {YOUR_ROOT}/Cinetransfer/slahmr/outputs/.../arc2-xxxx/motion_chunks/arc2_{MAX_NUM}_world_results.npz
to the {YOUR_ROOT}/Cinetransfer/data/arc2/
folder and rename it to arc2.npz
.
conda deactivate
cd {YOUR_ROOT}/Cinetransfer
git clone --recursive https://github.com/ashawkey/torch-ngp.git
mv torch-ngp torch_ngp
cd torch_ngp
conda create -n cinetrans python=3.9 -y
source activate cinetrans && pip install -r requirements.txt
source activate cinetrans && conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
# install the tcnn backbone
source activate cinetrans && pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
# install all extension modules
bash scripts/install_ext.sh
cd raymarching
source activate cinetrans && python setup.py build_ext --inplace
# install to python path (you still need the raymarching/ folder, since this only install the built extension.)
source activate cinetrans && pip install .
Notes:
- If you have any problems installing
torch-ngp
environment .You can refer to torch-ngp. - Rename
{YOUR_ROOT}/Cinetransfer/torch-ngp
to{YOUR_ROOT}/Cinetransfer/torch_ngp
.
cd {YOUR_ROOT}/Cinetransfer/camera_optim
source activate cinetrans && sudo apt-get update
source activate cinetrans && sudo apt install xvfb
source activate cinetrans && pip install einops chardet
source activate cinetrans && pip install pyrender
source activate cinetrans && pip install smplx[all]
source activate cinetrans && pip install imageio[ffmpeg] imageio[pyav]
Open {YOUR_ROOT}/Cinetransfer/torch_ngp/dnerf/utils.py
, in the file top add this
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
source activate cinetrans && xvfb-run --auto-servernum --server-num=1 python run_train_nerf.py --seq_name arc2
By using camera_ui
to optim first_frame.
cd {YOUR_ROOT}/Cinetransfer/camera_optim
source activate cinetrans && pip install gradio
source activate cinetrans && pip install plotly
# temp visual result can be found in {out_dir}/{seq_name}/cache_first_pose.png
source activate cinetrans && python run_camera_ui.py --seq_name arc2 --out_dir ../data/cam_optim_res
When open the UI, please follow those steps to check first pose:
- Step1. Adjust the slide rail on the left to control the Camera pose.
- Step2. Click the 'render' button to view the rendering results.
- Step3. Finally click the 'optim' button to optimize a single frame.
Note: You can repeat the above process until you select the most suitable camera pose.
cd {YOUR_ROOT}/Cinetransfer/camera_optim
source activate cinetrans && python run_camera_opt.py --seq_name arc2 --out_dir ../data/cam_optim_res
cd {YOUR_ROOT}/Cinetransfer/camera_optim
# smooth (optional)
source activate cinetrans && xvfb-run --auto-servernum --server-num=1 python run_camera_smooth.py --seq_name arc2 --out_dir ../data/cam_optim_res
source activate cinetrans && xvfb-run --auto-servernum --server-num=1 python run_camera_vis.py --seq_name arc2 --out_dir ../data/cam_optim_res
If you find this paper and repo useful for your research, please consider citing our paper.
@article{jiang2023cinematic,
title={Cinematic Behavior Transfer via NeRF-based Differentiable Filming},
author={Jiang, Xuekun and Rao, Anyi and Wang, Jingbo and Lin, Dahua and Dai, Bo},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2024}
}
The code is based on SLAHMR, torch-ngp, thanks to them!