Is it possible for a cast to slip down your leg and is it a big deal?

Your theory is absolutely correct. Whenever you have surgery, swelling is a normal part of the healing process because of white blood cells that flood the area to kill bacteria. So now it's been 3 weeks, much of the area is healed and there aren't as many white cells, The swelling went down. Don't you just hate casts? I hate the itching. Good luck.