Asteroids and comets are the remnants of the swarm of planetesimals from which the planets ultimately formed, and they retain records of processes that operated prior to and during planet formation. They are also likely the sources of most of the water and other volatiles accreted by Earth. In this review, we discuss the nature and probable origins of asteroids and comets based on data from remote observations, in situ measurements by spacecraft, and laboratory analyses of meteorites derived from asteroids. The asteroidal parent bodies of meteorites formed ≤4 Ma after Solar System formation while there was still a gas disk present. It seems increasingly likely that the parent bodies of meteorites spectroscopically linked with the E-, S-, M- and V-type asteroids formed sunward of Jupiter's orbit, while those associated with C- and, possibly, D-type asteroids formed further out, beyond Jupiter but probably not beyond Saturn's orbit. Comets formed further from the Sun than any of the meteorite parent bodies, and retain much higher abundances of interstellar material. CI and CM group meteorites are probably related to the most common C-type asteroids, and based on isotopic evidence they, rather than comets, are the most likely sources of the H and N accreted by the terrestrial planets. However, comets may have been major sources of the noble gases accreted by Earth and Venus. Possible constraints that these observations can place on models of giant planet formation and migration are explored.