Meta AI Chief Yann LeCun’s Self-Supervised Idea Finally Sees Open Source Light
Meta has been one of the biggest proponents of self-supervised learning when it comes to AI. Today, Meta AI has announced I-JEPA, a self-supervised computer vision model that learns the world by predicting it, based on Yann LeCun’s vision of autonomous machine intelligence to learn and reason similar to how humans and animals do. Click here to check it out. The paper is also being presented at CVPR2023 next week. The model’s training code and checkpoints are open-sourced under a non-commercial licence. I-JEPA (Image Joint Embedding Predictive Architecture) learns by creating an internal model representing the outside world and compares abstract representation of images, instead of comparing the pixels. According to the paper, this model delivers strong performance on various computer vision tasks, and is highly efficient than other similar models. I-JEPA offers versatile applicability without requiring extensive fine-tuning. Meta AI successfully trained a visual transformer model with 632M parameters utilising 16 A100 GPUs within a span of 72 hours. This model attains state-of-the-art results for low-shot classification on ImageNet, with a mere 12 labelled examples per class. In comparison, alternative approaches often consume two to 10 times the GPU-hours and yield inferior error rates when trained with an equivalent dataset size. By learning from representations instead of pixels, the model is able to avoid biases and issues that occur due to invariance-based pre-training. This also enables the model to learn directly from the images, instead of representation, which Meta AI says is a problem with the current LLM models. I-JEPA: Efficient method for Self-Supervised Learning of image features.No need for data augmentation, just masking.Joint embedding predictive architecture, not generative.And it's open source, of course.Blog: https://t.co/knAH1EzJCJPaper: https://t.co/fBG8xxEvb1Code &… https://t.co/jWcYyOoF17— Yann LeCun (@ylecun) June 13, 2023 The Theory Last year, Yann LeCun, Meta’s Chief AI Scientist, introduced an innovative architecture designed to address the significant constraints faced by contemporary AI systems, called the world model. LeCun envisions the development of machines capable of rapidly acquiring internal models of the world’s dynamics, enabling them to efficiently learn, strategise for complex tasks, and seamlessly adapt to novel circumstances. This work by Meta AI is highly based on the hypothesis that common sense information is the key for enabling intelligent behaviour. This knowledge is achieved by passively observing the world which is stored on the background of the mind. Meta believes that self-supervised learning is the path towards human-like intelligence. For this to work, the system needs to acquire these representations through self-supervised learning, which entails learning directly from unlabeled data like images or sounds, instead of relying on manually curated labelled datasets. Meta AI demonstrates the potential of I-JEPA, showcasing the ability to learn competitive off-the-shelf image representations without relying on additional knowledge encoded through manually designed image transformations. Advancing JEPAs further to acquire broader world-models from richer modalities would be particularly intriguing. This advancement could enable making long-range spatial and temporal predictions about future events in videos based on a short context, while conditioning these predictions on audio or textual prompts.
Meta AI successfully trained a visual transformer model with 632M parameters utilising 16 A100 GPUs within a span of 72 hours
Meta AI has announced I-JEPA, a self-supervised computer vision model based on Yann LeCun’s vision of autonomous machine intelligence
["Mark Zuckerberg", "Meta AI", "Open Source AI", "Yann LeCun"]
["API", "Meta AI", "Yann LeCun", "data augmentation", "self-supervised learning", "AI", "Modal", "A100", "ML", "computer vision", "Mark Zuckerberg", "Open Source AI", "R"]
["AI", "ML", "computer vision", "Meta AI", "R", "API", "data augmentation", "self-supervised learning", "Modal", "A100"]
Mohit writes about AI in simple, explainable, and often funny words. He's especially passionate about chatting with those building AI for Bharat, with the occasional detour into AGI.