Coastal regions have long been settled by humans due to their abundant resources for livelihoods, including agriculture, transportation, and rich biodiversity. However, natural and anthropogenic factors, such as climate change and sea-level rise, and land subsidence, population pressure, developmental activities, pose threats to coastal sustainability. Natural hazards, such as fluvial or coastal floods, impact poorer and more vulnerable communities greater than more affluent communities. Quantitative assessments of how natural hazards affect vulnerable communities in deltaic regions are still limited, hampering the design of effective management strategies to increase household and community resilience. Drawing from Driving Forces–Pressure–State–Impact–Response (DPSIR), we quantify the associations between household poverty and the likelihood of material and human loss following a natural hazard using new survey data from 783 households within Indian Sundarban Delta community. The results suggest that the poorest households are significantly more likely to endure material and human losses following a natural hazard and repeated losses of livelihood make them more vulnerable to future risk. The results further suggest that salinization, tidal surge, erosion, and household location are also significant predictors of economic and human losses. Given the current and projected impact of climate change and importance of delta regions as the world’s food baskets, poverty reduction and increase societal resilience should be a primary pathway to strengthen the resilience of the poorest populations inhabiting deltas.