The 2020 US presidential election not only witnessed an onslaught of accusations that elections were fraudulent, but these accusations implicitly and explicitly utilized racial signals to cast non-White Americans as the perpetrators of voter fraud. Elite rhetoric during and after the 2020 election painted predominantly Black and Latino cities as the epicenters of voter fraud, while also suggesting non-citizens were illegally voting in elections. We argue these racially coded accusations resonated with racially polarized White Americans, decreasing their confidence in the legitimacy of the 2020 presidential election. Using individual-level panel data to measure change in voter confidence among White Americans from 2016–2020, we find that confidence decreased between 2016 and 2020, but that this effect was more pronounced among White Americans who harbored greater racial affective polarization, with effects substantively similar to those of political measures of affective polarization. These results suggest that the racialization of election integrity in the 2020 election decreased voter confidence among the racially polarized White electorate. This study adds to a growing literature demonstrating the extent to which election racialization has permeated American politics and perceptions of electoral integrity specifically.