A grand challenge for robotics is generalization; to operate in unstructured real world environments we need household robots that can quickly learn to perform tasks in unseen kitchens, mobile manipulators and drones that can navigate novel spaces, and autonomous vehicles that can safely maneuver through unseen roads with varying conditions, all while minimizing dependence on humans. Recent breakthroughs in natural language processing and vision suggest that the secret to this level of generalization is data — not just the amount of data collected, but its diversity, and how to best leverage it while learning. Especially challenging is that most large-scale sources of data are collected offline, from varied sources. How do we use this diverse, offline data to build generalizable robotic systems?