Process

The initial step involved searching for animal videos featuring clear movement captured by a stationary camera. These videos were then broken down into individual image frames. Using the first frame from each video, masks were created to isolate individual animals. These masks were generated using SAM (Segment Anything Model), which can create precise masks from just a single point selected within the animal's body. In cases where multiple animals were present, their individual masks were combined into a single mask.

The second part of the process, led by Anushmita and me, begins by loading the first frame and overlaying it with the corresponding mask(s) to ensure precise pixel alignment. We utilized the CoTracker model as our foundational software for point tracking. The videos, converted into PyTorch tensors, are fed into the model alongside their masks. The final output produces a colorful visualization that effectively tracks and displays the animals' movements throughout the video.