Experts don't know what causes phyllodes tumors. Women who have a rare genetic disease called Li-Fraumeni syndrome are more likely to have them. They rarely affect men. Women of any age can get phyllodes tumors, but they're most common in your 40s.