Men are twice as likely to take the condom off midway through sex due to a poor fit, a poll of 436 in Sexually Transmitted Infections journal reveals.
Ill-fitting condoms more than doubled the chances of them splitting or slipping off, a Kentucky team found.
UK experts said men should know condoms came in different shapes and sizes.
Condom use is increasing in popularity, now rivalling the Pill as a form of contraception, according to a recent poll.
However, there are concerns that couples may turn away from them if they fail to fit comfortably and correctly, and risk unsafe sex instead.
The latest study fuels these concerns.
Out of 436 men questioned by the University of Kentucky researchers, 195 reported that their last sexual encounter involved an ill-fitting condom.
A total of 120 of these said that this had reduced their own sexual pleasure, and 57 said it had reduced it for their partner.
In addition, 34 said they had removed the condom early, while 18 said the condom had broken during sex.
All of these problems were far less frequent among men who reported a well-fitted condom.
One solution, according to the US researchers, would be better public health efforts to make sure that men achieved the best fit.
Simon Blake, from advisory charity Brook, said that, used correctly, condoms were 98% effective and protected against both unplanned pregnancy and sexually transmitted infections.
He said: “Standard latex condoms should fit most people but there are many different types of condoms available in different shapes and sizes and trying different types will be important in finding the ‘right one’.”
Natika Halil, from sexual health charity FPA, said that men, and women, needed to take time to learn the skills needed to put a condom on properly.
“Confidence and the skill of putting on a condom also contributes to how well they are used.
“Men come in all shapes and sizes and so do condoms. When we talk with men on the FPA helpline about condoms tearing, slipping off or being a nuisance to use, one of the main culprits is often something as simple as not using the right size.”
Gill Gordon, from the International HIV/AIDS Alliance, has spent time talking to men in Africa about condom use, and says the size and fit issue must be addressed if condoms are to be used effectively to reduce the number of new HIV infections.
Unlike in Europe and the US, the variety of sizes and shapes of condoms is not available to most men – she told of some resorting to tying an oversized condom on with string, or preferring to use the female condom because regular size versions were so painful.
She said: “I think that the unwillingness of African men to use condoms has been overstated, and that many more would be willing to wear one if they were more comfortable.
“If we are serious about people being able to use condoms consistently, then we need to listen to their needs.”