From your description, it sounds like you have what is referred to as a "Jones fracture". Your orthopedist was correct in stating these tak a long time to heal. Some require, lengthy casting, some a bone stimulator, and some require surgery.
The cast may be removed, but probably shopuld be followed up with a removable cast brace. When you get home, see a podiatrist or orthopedist.
Vivian Abrams DPM