3D-Text2LIVE (BU GRS CS640 Project 2022)

Zero-shot, text-driven appearance manipulation on multiple views of an object to generate 3D renderings.
Report Bug · Request Feature

Table of Contents

• Table of Contents
• About The Project
• Report and Presentation
• Models and Results
• Proposed Architecture
• Sample Results
  • Golden Hotdog
  • Ice Chair
  • Ship on Fire
• Hardware Requirements
• License
• Contributors
  • Animikh Aich
  • Himanshu Patil
  • Vedika Srivastava
• Acknowledgements

About The Project

This project is a part of the Boston University Course: GRS CS640 - Artificial Intelligence and builds on top of Text2LIVE. In particular, it involves the following three papers:

• Text2LIVE: Text-Driven Layered Image and Video Editing
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• DreamFusion: Text-to-3D using 2D Diffusion

A wide range of editing effects are now available to content creators thanks to extensive research into changing the appearance and style of objects in photographs. However, majority of the research in this field focuses on global editing rather than localized editing. To address this (Bar-Tal et al. 2022) developed an algorithm with localized editing of images using only text prompt. Given the substantial work being done on 3D objects and the widespread usage of 3D models in CAD modeling and video games, the same flexibility and range of editing effects ought to be available in 3D. Due to this, we propose 3D Text2LIVE, which gives the same degree of creative control over the appearance and style of 3D models as can be done with 2D photographs

Report and Presentation

• Project report: Report.pdf
• Project Presentation: Google Slides

Models and Results

Link to the trained models and results can be found here: Google Drive

Proposed Architecture

Sample Results

Golden Hotdog

Ice Chair

Ship on Fire

Hardware Requirements

We recommend an Nvidia GPU for Training the models. As per our experimentation the following specifications are recommended:

• Text2LIVE: Nvidia A100 (or any GPU with VRAM greater than 18 GB)
• NeRF: Nvidia Tesla V100 (or any GPU with 11 GB VRAM or Higher)
• DreamFusion3D: Nvidia Tesla V100 (or any GPU with VRAM greater than 16 GB)

License

Distributed under the GNU AGPL V3 License. See LICENSE for more information.

Contributors

Animikh Aich

• LinkedIn: animikh-aich
• Email: animikh@bu.edu
• GitHub: animikhaich
• Twitter: @AichAnimikh

Himanshu Patil

• LinkedIn: hipatil
• Email: hipatil@bu.edu
• GitHub: HiPatil

Vedika Srivastava

• LinkedIn: vedika-srivastava
• Email: vedikas@bu.edu
• GitHub: VedikaSrivastava

Acknowledgements

• Omer Bar-Tal (Text2LIVE)
• Yen-Chen Lin (NeRF)
• Kiui - Jiaxiang Tang (Stable Dreamfusion)