Current methods for learning visually grounded language from videos often\nrely on text annotation, such as human generated captions or machine generated\nautomatic speech recognition (ASR) transcripts. In this work, we introduce the\nAudio-Video Language Network (AVLnet), a self-supervised network that learns a\nshared audio-visual embedding space directly from raw video inputs. To\ncircumvent the need for text annotation, we learn audio-visual representations\nfrom randomly segmented video clips and their raw audio waveforms. We train\nAVLnet on HowTo100M, a large corpus of publicly available instructional videos,\nand evaluate on image retrieval and video retrieval tasks, achieving\nstate-of-the-art performance. We perform analysis of AVLnet's learned\nrepresentations, showing our model utilizes speech and natural sounds to learn\naudio-visual concepts. Further, we propose a tri-modal model that jointly\nprocesses raw audio, video, and text captions from videos to learn a\nmulti-modal semantic embedding space useful for text-video retrieval. Our code,\ndata, and trained models will be released at avlnet.csail.mit.edu\n