Some projects to showcase my learning, particularly focussing on fine-tuning models on biological data and image processing in general. Some might be related to experiment data for which I won't share the data for obvious reasons. Most of the projects focus on Computer Vision since I work with image data a lot, but I will try to put more projects related other domains such as NLP.
Trying out Segment Anything model from MetaAI with bounding box prompts. In this case, we give the bounding box coordinates to the model of the ROI and generate the a high-quality segmentation.
-
Automatic mask generation
-
Bounding box prompt for mask generation
Using Pytorch I implemented a Vision Transformer totally from scratch, similar to the paper An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. I closely followed Uva deep learning course explanation and how to define pytorch lightning training setup.
 |
Prediction  |
|---|
Segmentation using Segformer base model from HuggingFace on Retina blood vessel dataset with a simple gradio interface for inferencing
Dataset obtained from Kaggle - Blood Vessel Data
Segmentation of dendritic spines from 2-photon microscopy images. I used the base Segformer model and then converted it into a LoRA model using the PEFT library to reduce the number of parameters.
More details here-> ReadMe
Spine segmentation by finetuning Segment Anything model using points as prompts rather than bounding boxes.
For each of the ground truth masks, points (n=100) are sampled from the non-zero pixel positions then forward it to the prompt encoder.
I used the HuggingFace 🤗 SamModel class.
(It will be a nice experiment to see the quality of detection using bounding box prompt vs points)
Dataset - My own spine images
A simple pupil segmentation followed by centroid tracking. Segmentation is done using K-means and simple image-processing tricks. It's not SOTA, but I came with it in 4 hours for a competition.
Dataset - proprietary
 |
 |
|---|
A simple panoptic segmentation of the pupil for eye-tracking combining CLIP-Seg and SAM models. An improved version of the previous simple pupil segmentation I did for a competition.
Dataset: Private
My approach for the Kaggle competition Leash Bio to predict the binding nature between a protein and a molecule. Using simple Xgboost with molecular fingerprints as input features and also fine-tuning BERT model from HuggingFace, by tokenizing the protein and the smiles together using a [SEP] token similar to what is done for question-answering datasets. Also comes with a small gradio interface for easy testing new inputs.
I tackled an image registration problem to compute the suitable affine transforms between fixed and moving images of blood vessels using Spatial Transformers. The model is initialized with an identity transformation matrix [1, 0, 0, 0, 1, 0] and the model's job is to refine this matrix to get the suitable final transformation matrix. I used Normalized Cross Correlation (NCC) as the loss function as this is one of the standards used in unsupervised image registration and we try to minimize this function.
A simple way to remove background from images using Apple's Depth-pro model.
Simulating brainbow-like multicolour images from volumetric images of neuron reconstructions. Replication of the algorithm from Automated scalable segmentation of neurons from multispectral images, Sümbül et al. (2016)
