Why are men attracted to breasts?

Humans are from a family of animals called the Hominids. This includes other animals like the Chimpanzees. Second, Humans are the only Hominids where the breasts remain swollen at all times. All other Hominids have flat chests, except when lactating. Third, Humans are the only Hominids who walk upright at all times.

So, here's the answer. All other Hominids use the buttocks as their sexual attractor. But Humans walk upright and frequently are facing each other. So, Humans evolved permanently swollen breasts as an alternate sexual attractor. You will note that the general shape of the human breasts is similar to the buttocks: rounded with a cleavage between them.

Larger breasts more closely resemble the shape of the buttocks. This is the reason larger breasts tend to be more attractive, not because they "feed the babies better." Also, the breasts tend to swell a bit more when the woman is ovulating. So, larger breasts also suggest fertility.