Genomic instability and high mutation rates cause cancer to acquire numerous mutations and chromosomal alterations during its somatic evolution; most are termed passengers because they do not confer cancer phenotypes. Evolutionary simulations and cancer genomic studies suggest that mildly deleterious passengers accumulate and can collectively slow cancer progression. Clinical data also suggest an association between passenger load and response to therapeutics, yet no causal link between the effects of passengers and cancer progression has been established. To assess this, we introduced increasing passenger loads into human cell lines and immunocompromised mouse models. We found that passengers dramatically reduced proliferative fitness (∼3% per Mb), slowed tumor growth, and reduced metastatic progression. We developed new genomic measures of damaging passenger load that can accurately predict the fitness costs of passengers in cell lines and in human breast cancers. We conclude that genomic instability and an elevated load of DNA alterations in cancer is a double-edged sword: it accelerates the accumulation of adaptive drivers, but incurs a harmful passenger load that can outweigh driver benefit. The effects of passenger alterations on cancer fitness were unrelated to enhanced immunity, as our tests were performed either in cell culture or in immunocompromised animals. Our findings refute traditional paradigms of passengers as neutral events, suggesting that passenger load reduces the fitness of cancer cells and slows or prevents progression of both primary and metastatic disease. The antitumor effects of chemotherapies can in part be due to the induction of genomic instability and increased passenger load. Cancer Res; 77(18); 4763-72. ©2017 AACR.