Up to 125 students in one undergraduate course are suspected of sharing answers or plagiarising.
Possible cheating was found in about half of exam papers handed in by the class, which has at least 250 students.
If confirmed, the scandal would be the biggest to hit the "Ivy League" of top US universities in recent memory.
Suspicions were first aroused when a teaching assistant noticed several problems with the exam papers, including long passages of identical text in different students' answers, the Associated Press reports.
The university has not named the course or class in question.
Possible punishments for students found to have cheated range from a first-offence warning to being excluded from the university for a year, according to the agency.
"These allegations, if proven, represent totally unacceptable behaviour that betrays the trust upon which intellectual inquiry at Harvard depends," the university's president, Drew Faust, said.
Jay Harris, the dean of undergraduate education, stressed there was no evidence of widespread cheating at Harvard.
"The facts that are before us are that we have a problem in this one course," Mr Harris told AP.
"Looking at the students we have and the work that they do, I would be loath to say this is something that represents Harvard students generally."