The history of our actions and their outcomes represent important information, informing choices and efficiently guiding future behavior. While unsuccessful (S-) outcomes are expected to lead to more explorative motor states and increased behavioral variability, successful (S+) outcomes are expected to reinforce the use of the previous action. Here, we show that humans attribute different values to previous actions during reinforcement motor learning when they experience S- compared to S+ outcomes. Behavioral variability after an S-  outcome is influenced more by the previous outcome than after S+ outcomes. Using electroencephalography, we show that theta band oscillations of the prefrontal cortex are most prominent during changes in two consecutive outcomes, potentially reflecting the need for enhanced cognitive control. Our results suggest that S+ experiences 'overwrite' previous motor states to a greater extent than S- experiences and that modulations in neural oscillations in the prefrontal cortex play a potential role in encoding changes in movement variability state during reinforcement motor learning.