Skip to content

InquisitiveAS/ai2SpeckleApp_textureSearch_v0

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

71 Commits
 
 
 
 
 
 

Repository files navigation

ai2SpeckleApp:Texture Search_v0

image

Documentation

image

In the above diagram, There are three components to our "ai2SpeckleApp:matgen Web Application"

  1. A Front-end React Page that loads a Speckle Stream Viewer(3D Object Loader) via an authentication page through your Speckle account.

    • AI Image Analyzer Application GUI Window: A Button that takes a snapshot of the stream and uses AI Render Image API to render the stream and feeds it to the AI Image Analyzer Application based on whether the user accepts it.
    • A Button that calls the AI image Analyzer Application that then applies material to the elements in the Speckle Stream.
  2. An AI Image Renderer Application GUI Window This window does have a text prompt for the user to enter the Text Prompt which opens an "AI Image Rendered" window. The user has an option to either continue,re-render, or change the prompt for re-rendering. For the AI Image Rendering, when we click the AI Render or Rerender button, the button makes an API call to the free available state-of-the-art Image to Image AI Generators in the Generative AI community.

    We explored several Free Image to Image AI Renderers that offer free or paid pricing for API usage outlined below:

  1. AI Image Analyzer Application GUI Window(AI Workflow diagram)

    When the user is satisfied with 2 and proceeds with the "Continue" button, it sends the Rendered Image to the AI Image Analyzer Application GUI Window. This GUI window has a Machine Learning pipeline that generates Material as JPG/PNG which are hosted on PolyHaven so the user can download with a downloadable link Under the hood of this machine learning pipeline, there are several steps like Image Segmentation, Image Recognition from the Hugging Face dataset, and Image Similarity search to identify the materials based on labels inside the datasets and lastly post-processing the segmented labeled image from AI render Image to new materials.

    This technique expands the material database for users to select and apply later either to their BIM Applications or CAD software like Unreal,Unity3D,etc.

    All with the Power of the Speckle !!

INSTALLATION

Please follow the steps on the authentication for installing this repository

FUTURE IMPROVEMENTS

image

Credits | Teammates

Josie Harrison (Machine Learning Developer) Marwan Elzainy(Fullstack Developer) Jordana Rosa(Idea Conception) Abhishek Shinde(Researcher|Documentation)

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •