Colorectal cancer is the third most common cancer globally and is a leading cause of cancer death. Gastric cancer contributes significantly to the global cancer burden, particularly in low- and middle-income countries. We reviewed the literature for studies on physical activity or exercise and colon, rectal, and gastric cancers as well as colonic adenomatous polyps. We identified 52 studies of colon cancer, 31 studies of rectal cancer, 23 studies of colon polyps, and 16 studies of gastric cancer. Of the 52 studies of physical activity and colon cancer, 37 found a statistically significant association between increased levels of physical activity and decreased colon cancer risk in at least one comparison. Accumulated evidence suggests that physical activity is associated with a 25% reduction in colon cancer risk. In line with previous reports, we found no indication that the association was more pronounced for occupational versus recreational physical activity, with both resulting in a risk reduction of about 22%. Evidence for other domains of physical activity (i.e., transportation or household physical activity) is limited. Evidence is emerging that individuals who are consistently active across the lifetime may obtain greater risk reductions than those who are only active in recent years. Despite consistent associations with colon cancer, evidence is more limited though suggestive that physical activity reduces risk of colon adenomas or adenoma recurrence. There is clear evidence that physical activity is not associated with rectal or gastric cancers.