For a school with such a strong academic reputation, Yale has a pretty strong athletic program. The University offers options for every interest level: varsity sports for the athletically serious, club sports for the amateur, and IM's for just about anybody. Yale's varsity sports often do reasonably well and are usually comparable to other similar schools. There is also a prominant intermural culture in which the residential colleges compete against one another to win the Tyng Cup. The athletic facilities at Yale are second to none (Payne Whitney Gymnasium is the second largest gym in the world!) and are available to all students for exercise and fitness.
While not everybody participates in sports, practically everybody comes out for the Yale-Harvard football game, even when it's hosted in Boston. During this single weekend of school spirit and revelry, students rally behind Old Eli to show up Harvard and prove that not only does Yale play a great game, but she can also throw a great party.
Essentially, athletics are as important as all other extracurricular activities at Yale, including politics, music, drama, literary interests, and volunteering. Sports are in no way the central focus of the campus. That is the great thing about Yale—there are so many interests to pursue and they are all treated equally by the student body.