You think of swimming as a minority interest and a hobby for the wealthy? Surely not! It's one of the few sports (running would be another) that don't require any special equipment or facilities to get involved in and enjoy and train (although to compete professionally or in the Olympics is of course a somewhat different story). I would think that the only countries that don't have swimming as a popular, everbody-can-do-it-if-they-want sport would be countries where it's never possible to swim outside in natural bodies of water (I guess some desert nations would fall in this category, and maybe a few extremely northern ones?).

Dressage, though, and sailing -- I'd have to agree about the eliteness of those, in general. (Although I can imagine exceptions.)