The downsides are mainly that your life kinda gets taken over by medicine, so you have to enjoy it. Also I don’t think people talk about this much, but being around sick people all the time a lot of whom have very sad stories can be emotionally taxing.
Just don’t do it unless you have an obsessive passion for pathophysiology and medicine. There are countless cons to being a physician.