Dominick Reilly

I am a second-year PhD student at the University of North Carolina at Charlotte majoring in Computer Science. My research focuses on understanding human actions in videos, specifically daily-living actions that are observed in real-world environments. I am interested in utilizing both egocentric and third person viewpoints, as well as multiple modalities, to solve the challenges of daily-living actions. I am advised by Dr. Srijan Das.

Previously, I was a member of the Image Privacy Lab under the supervision of Dr. Liyue Fan. Our research adapted the standard notion of differential privacy to image data, with the objective of creating robust image obfuscation algorithms that offer rigorous privacy guarantees while preserving high utility.

News

Feb 2024	One paper, “Just Add π! Pose Induced Video Transformers for Understanding Activities of Daily Living”, is accepted to CVPR 2024!
Oct 2023	One paper, “Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders”, is accepted to WACV 2024!
May 2023	I was awarded The Chateaubriand Fellowship! Starting Feb 2024, I will spend 4 months at INRIA, Sophia Antipolis, France as a research intern with the STARS Team.

Selected publications

×

Just Add π! Pose Induced Video Transformers for Understanding Activities of Daily Living

Dominick Reilly, and Srijan Das

In IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference (CVPR), 2024

Abs arXiv Code

Video transformers have become the de facto standard for human action recognition, yet their exclusive reliance on the RGB modality still limits their adoption in certain domains. One such domain is Activities of Daily Living (ADL), where RGB alone is not sufficient to distinguish between visually similar actions, or actions observed from multiple viewpoints. To facilitate the adoption of video transformers for ADL, we hypothesize that the augmentation of RGB with human pose information, known for its sensitivity to fine-grained motion and multiple viewpoints, is essential. Consequently, we introduce the first Pose Induced Video Transformer: PI-ViT (or π-ViT), a novel approach that augments the RGB representations learned by video transformers with 2D and 3D pose information. The key elements of π-ViT are two plug-in modules, 2D Skeleton Induction Module and 3D Skeleton Induction Module, that are responsible for inducing 2D and 3D pose information into the RGB representations. These modules operate by performing pose-aware auxiliary tasks, a design choice that allows π-ViT to discard the modules during inference. Notably, π-ViT achieves the state-of-the-art performance on three prominent ADL datasets, encompassing both real-world and large-scale RGB-D datasets, without requiring poses or additional computational overhead at inference.
×

Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders

Srijan Das, Tanmay Jain, Dominick Reilly, Pranav Balaji, and 5 more authors

In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024

Abs arXiv Code

Vision Transformers (ViTs) have become ubiquitous in computer vision. Despite their success, ViTs lack inductive biases, which can make it difficult to train them with limited data. To address this challenge, prior studies suggest training ViTs with self-supervised learning (SSL) and fine-tuning sequentially. However, we observe that jointly optimizing ViTs for the primary task and a Self-Supervised Auxiliary Task (SSAT) is surprisingly beneficial when the amount of training data is limited. We explore the appropriate SSL tasks that can be optimized alongside the primary task, the training schemes for these tasks, and the data scale at which they can be most effective. Our findings reveal that SSAT is a powerful technique that enables ViTs to leverage the unique characteristics of both the self-supervised and primary tasks, achieving better performance than typical ViTs pre-training with SSL and fine-tuning sequentially. Our experiments, conducted on 10 datasets, demonstrate that SSAT significantly improves ViT performance while reducing carbon footprint. We also confirm the effectiveness of SSAT in the video domain for deepfake detection, showcasing its generalizability.