Fertility is often treated as primarily a women's issue, but both men and women contribute to infertility in about 35% of couples who have trouble conceiving, according to the CDC. In 8% of the cases, the man is the only factor — and some experts say we're not talking about it enough.

"A lot of men equate fertility with virility and sexual potency and it's unfortunate because the two are really not the same thing," Dr. Tara Narula told "CBS This Morning" on Wednesday.

Men often feel if they can't conceive there's something wrong with them and end up staying mum. The way society — and even the medical community — approaches male infertility doesn't help.

"They keep it very private … and it's been documented that not only do they feel they can't talk about it or don't get support from their family, but also in the medical world, they don't get a lot of support," Narula said. "A lot of the support is focused on the women and even online women have community groups and support groups on Facebook and the men really don't. We are in a culture where we don't really ask men how do you feel about not being able to have a baby, or do you want children?"

Narula said the common causes of infertility in men can be broken down into four categories. About 10% to 20% of the cases are considered idiopathic, meaning we have no idea what caused it. About 60% to 80% of male infertility cases are caused by something wrong with the testicle, meaning it could be genetic, acquired, structural, or because of environmental exposure to toxins. About 5% of cases have to do with a hormonal problem, and another 5% are a transportation problem — an issue with the sperm getting out of the testicle.

Some of the most common risk factors for male infertility are obesity, smoking, tight clothing, drugs, alcohol, and even exposure to high heat.

"If you're increasing the temperature of the testicles you can have some issues with sperm production. So that's why part of the questioning of men includes asking about hot tub or sauna use," Narula explained.

© 2019 CBS Interactive Inc. All Rights Reserved.