Adversities experienced at different life stages impact cognitive aging, with early-life hunger and financial hardship predicting lower cognitive performance in older age. Financial hardship and stress in early adulthood, however, are associated with better cognitive performance.