Swimmers have to carry around the body mass during the race. A lighter swimmer does not work as hard as a larger swimmer because they have less to carry around. This also applies to running and bicycling (see a pattern here...). The top runners and cyclists are not big burly athletes, but more like svelte and lean individuals, usually average height.
Recognizing this fact, and with the increasing popularity of triathlons here in the USA, the governing body here in the U.S., USA Triathlon, created a separate of competition for people who were larger than most of the field. They call the men "clydesdales" and women "athenas".