Evolutionary Robotics and Robot Learning are two fields in robotics that aim to automatically optimize robot designs. The key difference between them lies in what is being optimized and the time scale involved. Evolutionary Robotics is a field that applies evolutionary computation techniques to evolve the morphologies or controllers, or both. Robot Learning, on the other hand, involves any learning technique aimed at optimizing a robot's controller in a given morphology. In terms of time scales, evolution occurs across multiple generations, whereas learning takes place within the 'lifespan' of an individual robot. The long-term goal of Evolutionary Robotics is to create adaptive systems where a population of robots evolve autonomously, optimizing both their physical structure and control system through an evolutionary process. In the context of evolution, adaptability is a trait of the population. Conversely, when it comes to learning, it is the robots' controller that exhibits adaptability. In the end, both forms of adaptation aim to enhance the robots' task performance. Integrating Robot Learning with Evolutionary Robotics seems like a natural fit for improving robot design. Unfortunately, integration requires the careful design of suitable learning algorithms in the context of evolutionary robotics. The effects of introducing learning into the evolutionary process are not well-understood and can thus be tricky. This thesis investigates these intricacies and presents several learning algorithms developed for an Evolutionary Robotics context. My dissertation is structured into three parts: Part I - investigates the complex interaction between learning algorithms and evolutionary processes, provides statistical tools for evaluating different learning algorithms, and explores the dynamics of optimization and the reality gap. The interactions present a counterintuitive finding: learning can negatively affect the evolutionary process. Learning can bias evolution by converging to simple designs that learn quickly, overfitting simulators and exacerbating the reality gap. Part II - investigates model-agnostic learning methods for autonomous robots. Evolutionary algorithms can produce arbitrary robot designs for environments with minimal prior knowledge. This presents a challenging requirement for learning. Nevertheless, robots are expected to be able to autonomously perform tasks 'in the wild'. Here, I cover continuous self-modeling for adaptive feedback control and rapid skill acquisition for quickly learning locomotion. Part III - explores how learning can be extended beyond individual robot. In group settings, swarms of robots can obtain abilities beyond that of any individual robot inside. Here, I demonstrate how such emergent capabilities can be learned and used to solve complex tasks, both in homogeneous and heterogeneous populations of robots. Overall, this thesis offers a comprehensive analysis of Robot Learning within the context of Evolutionary Robotics presented as a collection of peer-reviewed works. Each part of the dissertation combines rigorous theoretical analysis with practical hardware implementations, demonstrating the validity of the ideas presented. As a result, my thesis provides unique insights into how an evolving population of robots can effectively integrate learning. From the 'birth' of a robot, learning about its environment, to its 'adulthood' as a functioning member of a robotic society.