Trees are long-lived organisms that integrate climate conditions across years or decades to produce secondary growth. This integration process is sometimes referred to as ‘climatic memory.’ While widely perceived, the physiological processes underlying this temporal integration, such as the storage and remobilization of non-structural carbohydrates (NSC), are rarely explicitly studied. This is perhaps most apparent when considering drought legacies (perturbed post-drought growth responses to climate), and the physiological mechanisms underlying these lagged responses to climatic extremes. Yet, drought legacies are likely to become more common if warming climate brings more frequent drought. To quantify the linkages between drought legacies, climate memory, and NSC, we measured tree growth (via tree ring widths) and NSC concentrations in three dominant species across the southwestern US. We analyzed these data with a hierarchical mixed effects model to evaluate the time-scales of influence of past climate (memory) on tree growth. We then evaluated the role of climate memory and the degree to which variation in NSC concentrations were related to forward-predicted growth during the hot 2011–2012 drought and subsequent 4-year recovery period. Populus tremuloides exhibited longer climatic memory compared to either Pinus edulis or Juniperus osteosperma, but following the 2011–2012 drought, P. tremuloides trees with relatively longer memory of temperature conditions showed larger (more negative) drought legacies. Conversely, P. edulis trees with longer temperature memory had smaller (less negative) drought legacies. For both species, higher NSC concentrations followed more negative (larger) drought legacies, though the relevant NSC fraction differed between P. tremuloides and P. edulis. Our results suggest that differences in tree NSC are also imprinted upon tree growth responses to climate across long time scales, which also underlie tree resilience to increasingly frequent drought events under climate change.