https://MSAIL.github.io/slides/ Slides Source Themes Academic (https://sourcethemes.com/academic/)en-us https://MSAIL.github.io/slides/videobert/ Mon, 01 Jan 0001 00:00:00 +0000 https://MSAIL.github.io/slides/videobert/ <h1 id="videobert">VideoBERT</h1> <h2 id="c-sun-et-al">C. Sun et al.</h2> <h3 id="google-research">Google Research</h3> <p>Presented by: Nikhil Devraj</p> <p style="color:grey"> MSAIL </p> <hr> <h2 id="motivation">Motivation</h2> <ul> <li>Representations of video data generally capture only low-level features and not semantic data</li> </ul> <center> <img src="https://MSAIL.github.io/slides/low_level.png" alt="Low-level features" height="200"/> </center> <ul> <li> <a href="https://arxiv.org/abs/1810.04805" target="_blank" rel="noopener">BERT</a> performs really well on language modeling tasks</li> </ul> <hr> <h2 id="contributions">Contributions</h2> <ul> <li>Combined <a href="https://en.wikipedia.org/wiki/Speech_recognition" target="_blank" rel="noopener">ASR</a>, <a href="https://en.wikipedia.org/wiki/Vector_quantization" target="_blank" rel="noopener">Vector Quantization</a>, and BERT to learn high-level features over long time spans in video tasks</li> <li>A first step in the direction of learning high-level joint representations</li> </ul> <hr> <p><img src="https://MSAIL.github.io/slides/videobert_flow.png" alt="VideoBERT Flow"></p> <hr> <h1 id="background">Background</h1> <hr> <h2 id="bert">BERT</h2> <ul> <li>Pretrained language model used to generate a probability distribution of tokens</li> <li>Obtained by training model on &ldquo;masking&rdquo; task</li> </ul> <hr> <center> <img src="https://MSAIL.github.io/slides/bert.png" alt="BERT" height="600"/> </center> <hr> <h2 id="supervised-learning">Supervised Learning</h2> <ul> <li>Expensive to get labeled data</li> <li>Short term events in video data</li> </ul> <hr> <h2 id="unsupervised-learning">Unsupervised Learning</h2> <ul> <li>Learns from unlabeled data</li> <li>Normal approaches used latent variables (i.e. GAN, VAE) <ul> <li>differ from BERT</li> </ul> </li> </ul> <hr> <h2 id="self-supervised-learning">Self-supervised Learning</h2> <p><img src="https://MSAIL.github.io/slides/self-sup-lecun.png" alt="Self-supervised example"></p> <p> <a href="https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html" target="_blank" rel="noopener">More on self supervised learning</a></p> <hr> <h2 id="cross-modal-learning">Cross-Modal Learning</h2> <ul> <li>Synchronized audio and visual signals allow them to supervise each other</li> <li>Use ASR as a source of crossmodal supervision</li> </ul> <hr> <h2 id="instructional-video-datasets">Instructional Video Datasets</h2> <ul> <li>Papers used LMs to analyze these videos with manually provided data</li> <li>Datasets were too small</li> </ul> <hr> <h1 id="method">Method</h1> <hr> <h2 id="omitted-the-rest">Omitted the rest</h2> <p>You get the principles I&rsquo;m getting at though right?</p>