VideoMAE

Paper	Colab	HF Space	HF Hub

Video masked autoencoders (VideoMAE) are seen as data-efficient learners for self-supervised video pre-training (SSVP). Inspiration was drawn from the recent ImageMAE, and customized video tube masking with an extremely high ratio was proposed. Due to this simple design, video reconstruction is made a more challenging self-supervision task, leading to the extraction of more effective video representations during this pre-training process. Some hightlights of VideoMAE:

• Masked Video Modeling for Video Pre-Training
• A Simple, Efficient and Strong Baseline in SSVP
• High performance, but NO extra data required

This is a unofficial Keras reimplementation of VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training model. The official PyTorch implementation can be found here.

Model Zoo

The pre-trained and fine-tuned models are listed in MODEL_ZOO.md. Following are some hightlights.

Kinetics-400

For Kinetrics-400, VideoMAE is trained around 1600 epoch without any extra data. The following checkpoints are available in both tensorflow SavedModel and h5 format.

^{?* Official ViT-H backbone of VideoMAE has weight issue in pretrained model, details https://github.com/MCG-NJU/VideoMAE/issues/89.} ^{The FLOPs of encoder models (FT) are reported only.}

Something-Something V2

For SSv2, VideoMAE is trained around 2400 epoch without any extra data.

UCF101

For UCF101, VideoMAE is trained around 3200 epoch without any extra data.