A central goal of population genetics is to understand how genetic drift, natural selection, and gene flow shape allele frequencies through time. However, the actual processes underlying these changes-variation in individual survival, reproductive success, and movement-are often difficult to quantify. Fully understanding these processes requires the population pedigree, the set of relationships among all individuals in the population through time. Here, we use extensive pedigree and genomic information from a long-studied natural population of Florida Scrub-Jays (Aphelocoma coerulescens) to directly characterize the relative roles of different evolutionary processes in shaping patterns of genetic variation through time. We performed gene dropping simulations to estimate individual genetic contributions to the population and model drift on the known pedigree. We found that observed allele frequency changes are generally well predicted by accounting for the different genetic contributions of founders. Our results show that the genetic contribution of recent immigrants is substantial, with some large allele frequency shifts that otherwise may have been attributed to selection actually due to gene flow. We identified a few SNPs under directional short-term selection after appropriately accounting for gene flow. Using models that account for changes in population size, we partitioned the proportion of variance in allele frequency change through time. Observed allele frequency changes are primarily due to variation in survival and reproductive success, with gene flow making a smaller contribution. This study provides one of the most complete descriptions of short-term evolutionary change in allele frequencies in a natural population to date.