Yong Jae Lee

Associate Professor
Department of Computer Science
UC Davis

Office: Academic Surge 2075
Email: yongjaelee at ucdavis dot edu
Phone: (530) 752-3788

Bio / CV
/ Research Statement / Google Scholar

I am an Associate Professor in the Computer Science Department at UC Davis.  Before joining UC Davis in Fall 2014, I was a Postdoctoral Fellow in the EECS Dept at UC Berkeley working with Alyosha Efros.  I also spent a year as a Postdoctoral Fellow in the Robotics Institute at Carnegie Mellon University (8/2012-8/2013), where I had the good fortune to work with Alyosha Efros and Martial Hebert.  I obtained my Ph.D. from the University of Texas at Austin in May 2012 under the supervision of Kristen Grauman, and my B.S. from the University of Illinois at Urbana-Champaign in May 2006.  I have also worked with Larry Zitnick and Michael Cohen as a summer intern at Microsoft Research.

I will join the Computer Sciences Department at the University of Wisconsin-Madison in Fall 2021. I will be on leave for one year as an AI Visiting Faculty at Cruise till then.
Prospective PhD students: please apply to the UW-Madison Computer Sciences program if you'd like to work with me.



News


Research Interests

My research interests are in computer vision, machine learning, and computer graphics.  I am particularly interested in creating robust visual recognition systems that can understand visual data with minimal human supervision.


Teaching

ECS 174: Computer Vision (Spring 2020)
ECS 269: Visual Recognition (Fall 2019)
ECS 174: Computer Vision (Spring 2019)

ECS 269: Visual Recognition (Fall 2018)
ECS 174: Computer Vision (Spring 2018)

ECS 289G: Visual Recognition (Winter 2018)
ECS 174: Computer Vision (Spring 2017)
ECS 289G: Visual Recognition (Fall 2016)

ECS 289G: Visual Recognition (Fall 2015)
ECS 189G: Intro to Computer Vision (Spring 2015)

ECS 289H: Visual Recognition (Fall 2014)


Lab Members

Xueyan Zou (PhD student)
Utkarsh Ojha (PhD student)
Haotian Liu (PhD student)
Yuheng Li
(PhD student)
Rafael A. Rivera-Soto (MS student)


Alumni

Former PhD students
Fanyi Xiao
(2015-2020)
Research Scientist at Amazon Research; UC Davis Best Graduate Researcher in Computer Science Award 2018
Krishna Kumar Singh (
2015-2020) Research Scientist at Adobe Research; UC Davis Best Graduate Researcher in Computer Science Award (Honorable Mention) 2019
Maheen Rashid (2015-2021) Research Scientist at Univrses
 

Former MS students
Zhongzheng (Jason) Ren (2016-2018)
PhD student at UIUC

Wenjian Hu (2017-2018) Research Scientist at Facebook
Leonardo Ferrer (2017) Software Engineer at Google
Wei-Pang (Tyler) Jan (2019) Software Engineer at Amazon
Chong Zhou (2019-2020) PhD student at Nanyang Technological University
Yangming Wen (2019-2020)       

Former BS students and visitors
Antonia Creswell (2014-2015) PhD student at Imperial College London
Yi Mang (Terry) Yang (2017-2018
) Software Engineer at Amazon
Xie Zhou
(2018-2019) MS student at UC Berkeley
Daniel Bolya (2018-2019) PhD student at Georgia Tech; Chancellor's Award for Excellence in Undergraduate Research (Honorable Mention) 2019 
Aron Sarmasi (2018-2019)
MS student at UC Davis
Waiyu Lam (2019-2020)
MS student at Cornell
Qi Zhu (summer 2015, co-supervised with Ian Davidson) PhD student at UIUC
Xiuye Gu (summer 2016, 2018-2019) MS student at Stanford
Haolin Fu (summer 2016, co-supervised with Cho-Jui Hsieh) MS student at Yale
Haotian Liu (summer 2018) PhD student at UC Davis
Hao Yu (summer 2018) PhD student at Boston University
Weixin Luo (visiting research scholar, 2018-2019)


Funding

I am grateful for the support by the National Science Foundation (CAREER IIS-1751206, IIS-1748387, and IIS-1812850), Army Research Office (Young Investigator Program), Hellman Fellows Program, Intel, Adobe, Nvidia, Amazon, and Sony.


Recent talks


Real-time Instance Segmentation with the YOLACT Family
OpenMMLab Tutorial, CVPR, June 2021 (virtual)

[talk video]

Learning to Understand Visual Data with Minimal Human Supervision
University of Wisconsin - Madison, April 2020 (virtual)

[talk video]


Preprints

Audiovisual SlowFast Networks for Video Recognition
Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer
arXiv 2019
[arXiv]

Publications


NEW!  YolactEdge: Real-time Instance Segmentation on the Edge
Haotian Liu*, Rafael A. Rivera-Soto*, Fanyi Xiao, and Yong Jae Lee
IEEE International Conference on Robotics and Automation (ICRA), 2021
(*equal contribution)
[arXiv] [code] [youtube] [talk video] [Colab Notebook] [Colab Notebook (TensorRT)]

NEW!  Few-shot Image Generation via Cross-domain Correspondence
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
[project page] [arXiv] [code]

NEW!  Progressive Temporal Feature Alignment Network for Video Inpainting
Xueyan Zou , Linjie Yang, Ding Liu, and Yong Jae Lee
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
[arXiv] [code] [youtube]


NEW!  Generating Furry Cars: Disentangling Object Shape and Appearance across Multiple Domains
Utkarsh Ojha, Krishna Kumar Singh, and Yong Jae Lee
International Conference on Learning Representations (ICLR), 2021

[project page] [open review] [arXiv] [talk video]


NEW!  SinGAN-GIF: Learning a Generative Video Model from a Single GIF
Rajat Arora and Yong Jae Lee
Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2021
[project page] [pdf] [talk video]


Elastic-InfoGAN: Unsupervised Disentangled Representation Learning in Class-Imbalanced Data
Utkarsh Ojha, Krishna Kumar Singh, Cho-Jui Hsieh, and Yong Jae Lee
Neural Information Processing Systems (NeurIPS), 2020

[project page] [arXiv] [code]
YOLACT++: Better Real-time Instance Segmentation
Daniel Bolya*, Chong Zhou*, Fanyi Xiao, and Yong Jae Lee
IEEE Transactions on Pattern Analysis and Machine Intelligence (
TPAMI), 2020 (journal extension of our ICCV 2019 conference paper with improved models)
(*equal contribution)
[arXiv] [code]

Delving Deeper into Anti-aliasing in ConvNets
Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee
Proceedings of the British Machine Vision Conference (BMVC), 2020
(Oral presentation)
Best Paper Award  
[project page] [arXiv] [code] [talk video]
Password-conditioned Anonymization and Deanonymization with Face Identity Transformers
Xiuye Gu, Weixin Luo, Michael Ryoo, and Yong Jae Lee
Proceedings of the European Conference on Computer Vision (ECCV), 2020
[arXiv] [code] [demo] [1 min talk video] [10 min talk video]

Boxer: Preventing Fraud by Scanning Credit Cards
Zainul Abi Din, Hari Venugopalan, Jaime Park, Andy Li, Weisu Yin, Haohui Mai, Yong Jae Lee, Steven Liu, and Samuel T. King
Proceedings of the USENIX Security Symposium (USENIX Security), 2020
[pdf] [project page] [talk video]
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation
Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020
[arXiv] [code] [youtube] [talk video]

Don’t Judge an Object by Its Context: Learning to Overcome Contextual Bias
Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020 (Oral presentation)
[arXiv] [project page]

Instance-aware, Context-focused, and Memory-efficient Weakly-supervised Object Detection
Zhongzheng Ren, Zhiding Yu, Xiaodong Yang, Ming-Yu Liu, Yong Jae Lee, Alexander Schwing, and Jan Kautz
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020
[arXiv] [project page] [code]

Action Graphs: Weakly-supervised Action Localization with Graph Convolution Networks
Maheen Rashid, Hedvig Kjellström, and Yong Jae Lee
Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2020
[arXiv] [code]


YOLACT: Real-time Instance Segmentation
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee
Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019 (Oral presentation)
Most Innovative Award, COCO Object Detection Challenge, ICCV 2019
[arXiv] [code] [pdf] [talk video]


Identity from here, Pose from there: Self-supervised Disentanglement and Generation of Objects using Unlabeled Videos
Fanyi Xiao, Haotian Liu, and Yong Jae Lee
Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019
[pdf]


FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery
Krishna Kumar Singh*,  Utkarsh Ojha*, and Yong Jae Lee
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Oral presentation)
(*equal contribution)
[project page] [pdf] [arXiv] [code] [youtube] [talk video]


You reap what you sow: Using Videos to Generate High Precision Object Proposals for Weakly-supervised Object Detection
Krishna Kumar Singh and Yong Jae Lee
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019
[project page] [pdf] [code]


HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-scale Point Clouds
Xiuye Gu, Yijie Wang, Chongruo Wu, Yong Jae Lee, and Panqu Wang
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019
[pdf] [supp] [code]

Video Object Detection with an Aligned Spatial-Temporal Memory
Fanyi Xiao and Yong Jae Lee
Proceedings of the European Conference on Computer Vision (ECCV), 2018
[project page] [pdf] [code]

Learning to Anonymize Faces for Privacy Preserving Action Detection
Zhongzheng Ren, Yong Jae Lee, and Michael Ryoo
Proceedings of the European Conference on Computer Vision (ECCV), 2018
[project page] [pdf] [youtube]


DOCK: Detecting Objects by transferring Common-sense Knowledge
Krishna Kumar Singh, Santosh Divvala, Ali Farhadi,
and Yong Jae Lee
Proceedings of the European Conference on Computer Vision (ECCV), 2018
[project page] [pdf] [code]


A Visual Attention Grounding Neural Model for Multimodal Machine Translation
Mingyang Zhou, Runxiang Cheng,
Yong Jae Lee, and Zhou Yu
Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018 (Oral presentation)
[pdf

Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery
Zhongzheng Ren and Yong Jae Lee
Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[project page] [pdf] [code]

Who Will Share My Image? Predicting the Content Diffusion Path in Online Social Networks
Wenjian Hu, Krishna Kumar Singh*, Fanyi Xiao*, Jinyoung Han, Chen-Nee Chuah, and Yong Jae Lee
Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM), 2018
(*equal contribution)
[pdf]

Can a Machine Learn to See Horse Pain? An Interdisciplinary Approach Towards Automated Decoding of Facial Expressions of Pain in the Horse
Pia Andersen, Karina Gleerup, Jennifer Wathan, Britt Coles, Hedvig Kjellström, Sofia Broome, Yong Jae Lee, Maheen Rashid, Claudia Sonder, Erika Rosenberger, and Deborah Forster
International Conference on Methods and Techniques in Behavioral Research (Measuring Behavior), 2018

[pdf]

What Should I Annotate? An Automatic Tool for Finding Video Segments for EquiFACS Annotation
Maheen Rashid, Sofia Broome, Pia Andersen, Karina Gleerup, and Yong Jae Lee
International Conference on Methods and Techniques in Behavioral Research (Measuring Behavior), 2018

[pdf]


Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localization
Krishna Kumar Singh and Yong Jae Lee
Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017
[project page] [pdf] [supp] [code]

Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
Fanyi Xiao, Leonid Sigal, and Yong Jae Lee
Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017
[project page] [pdf]


Interspecies Knowledge Transfer for Facial Keypoint Detection
Maheen Rashid, Xiuye Gu, and Yong Jae Lee
Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017
[project page] [pdf]
[code] [data]

Identifying First-Person Camera Wearers in Third-Person Videos
Chenyou Fan, Jangwon Lee, Mingze Xu, Krishna Kumar Singh, Yong Jae Lee, David Crandall and Michael Ryoo
Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017
[pdf]

Who Moved My Cheese? Automatic Annotation of Rodent Behaviors with Convolutional Neural Networks
Zhongzheng Ren, Adriana Noronha, Annie Vogel Ciernia, and Yong Jae Lee
Proceedings of the Winter Conference on Applications of Computer Vision
(WACV), 2017
[project page] [pdf]
[code] [data]

Analyzing the Adoption and Cascading Process of OSN-Based Gifting Applications: An Empirical Study
M. Rezaur Rahman, Jinyoung Han, Yong Jae Lee, and Chen-Nee Chuah
ACM Transactions on the Web
(TWEB), 2017
[pdf]

End-to-End Localization and Ranking for Relative Attributes
Krishna Kumar Singh and Yong Jae Lee
Proceedings of the European
Conference on Computer Vision (ECCV), 2016
[project page] [pdf]
[code]

Track and Transfer: Watching Videos to Simulate Strong Human Supervision for Weakly-Supervised Object Detection
Krishna Kumar Singh, Fanyi Xiao, and Yong Jae Lee
Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2016
[project page] [pdf] [arXiv (with more results)]
[code]
Track and Segment: An Iterative Unsupervised Approach for Video Object Proposals
Fanyi Xiao and Yong Jae Lee
Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Spotlight presentation)
[project page] [pdf] [code]


Localizing and Visualizing Relative Attributes
Fanyi Xiao and Yong Jae Lee
Springer Book Chapter on Visual Attributes, 2016

[pdf]
[code]

Discovering Mid-level Visual Connections in Space and Time
Yong Jae Lee, Alexei A. Efros, and Martial Hebert
Springer Book Chapter on Visual Analysis and Geo-Localization of Large Scale Imagery, 2016

[pdf]
[code] [data]

Discovering the Spatial Extent of Relative Attributes
Fanyi Xiao and Yong Jae Lee
Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015 (Oral presentation)
[project page] [pdf] [slides] [code] [video presentation]


FlowWeb: Joint Image Set Alignment by Weaving Consistent, Pixel-wise Correspondences
Tinghui Zhou, Yong Jae Lee, Stella X. Yu, and Alexei A. Efros
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015 (
Oral presentation)
[project page] [pdf] [code]


Predicting Important Objects for Egocentric Video Summarization
Yong Jae Lee and Kristen Grauman
International Journal of Computer Vision (IJCV),
2015         
[project page] [pdf] [arXiv] [data]

Weakly-supervised Discovery of Visual Pattern Configurations
Hyun Oh Song, Yong Jae Lee, Stefanie Jegelka, and Trevor Darrell
Neural Information Processing Systems (NIPS), 2014
[pdf]


AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections
Jun-Yan Zhu, Yong Jae Lee, and Alexei A. Efros
ACM Transactions on Graphics (Proceedings of SIGGRAPH), 2014 (
Oral presentation)
[project page]
[pdf] [youtube] [See article in The New Yorker]


Style-aware Mid-level Representation for Discovering Visual Connections in Space and Time
Yong Jae Lee, Alexei A. Efros, and Martial Hebert
Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2013 (
Oral presentation)
[project page]
[pdf] [slides] [code] [data] [video presentation]


Discovering Important People and Objects for Egocentric Video Summarization
Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012
[project page]
[pdf] [supp] [extended abstract] [data]


Object-Graphs for Context-Aware Visual Category Discovery
Yong Jae Lee and Kristen Grauman
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2012
[project page]
[pdf] [code]


Key-Segments for Video Object Segmentation
Yong Jae Lee, Jaechul Kim, and Kristen Grauman
Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2011
[project page]
[pdf] [code] [data]


ShadowDraw: Real-Time User Guidance for Freehand Drawing
Yong Jae Lee, Larry Zitnick, and Michael Cohen
ACM Transactions on Graphics (Proceedings of SIGGRAPH), 2011 (
Oral presentation)
[project page]
[pdf] [slides] [video] [youtube] [data]


Face Discovery with Social Context
Yong Jae Lee and Kristen Grauman
Proceedings of the British Machine Vision Conference (BMVC), 2011         
[project page]
[pdf] [extended abstract]


Learning the Easy Things First: Self-Paced Visual Category Discovery
Yong Jae Lee and Kristen Grauman
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011
[project page]
[pdf]


Object-Graphs for Context-Aware Category Discovery
Yong Jae Lee and Kristen Grauman

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010 (
Oral presentation)
[project page]
[pdf] [supp] [slides] [code]


Collect-Cut: Segmentation with Top-Down Cues Discovered in Multi-Object Images
Yong Jae Lee and Kristen Grauman

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010
[project page]
[pdf] [supp] [data]


Foreground Focus: Unsupervised Learning from Partially Matching Images
Yong Jae Lee and Kristen Grauman

International Journal of Computer Vision (IJCV), 2009         
[project page]
[pdf]


Shape Discovery from Unlabeled Image Collections
Yong Jae Lee and Kristen Grauman

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009
[project page]
[pdf] [supp]


Foreground Focus: Finding Meaningful Features in Unlabeled Images
Yong Jae Lee
and Kristen Grauman

Proceedings of the British Machine Vision Conference (BMVC), 2008 (
Oral presentation)
[project page]
[pdf] [slides]


Ray-based Color Image Segmentation
Changhai Xu, Yong Jae Lee, and Benjamin Kuipers
Proceedings of the Canadian Conference on Computer and Robot Vision (CRV), 2008
                   
[pdf]


Theses

PhD thesis: Visual Object Category Discovery in Images and Videos
MS thesis: Foreground Focus: Finding Meaningful Features in Unlabeled Images