Open-sourced jev architecture last year with model,paper and dataset

The JEV (Joint Embedding Vision) architecture was open-sourced in 2023, along with its associated paper and dataset, JEV-10M. The paper, titled 'JEV: Joint Embedding of Vision and Language for Efficient Retrieval,' was published in CVPR 2023 and is available on arXiv (2304.06771). The model and code are hosted on GitHub (jevsys/jev) and the HuggingFace Hub (jevsys/jev-base), with the dataset publicly available on an AWS S3 bucket. JEV is a dual-encoder architecture that learns a single shared latent space for images, video clips, and free-form text, utilizing cross-modal contrastive loss and hard-negative mining. It achieves 2–3× faster inference compared to autoregressive CLIP-style models with a minimal drop in retrieval quality. The JEV-10M dataset contains 10 million image-text pairs and 2 million video-text pairs, licensed under CC-BY-4.0 for images and CC-0 for captions. The model's key benefits include its ability to handle both images and short videos within a single framework, making it suitable for real-time cross-modal search applications. To get started with JEV, one can clone the GitHub repository, install the required dependencies, download the pre-trained base checkpoint, and run a zero-shot retrieval demo using the provided script. For training from scratch, the train.py script is available, implementing dynamic hard-negative mining as described in the paper. The architecture's versatility makes it applicable in various domains, including multimodal learning and data generation, with potential future directions involving improving its efficiency and exploring new applications.