I think I would not be in too big of a hurry to get the PEG out if you are continuing to have esophageal dilatations for radiation side effects. Your doctor is giving you good adice telling you not to take it out until you are able to go 3 weeks without using it. Also make sure you are getting enough calories in orally every day and I know that takes a lot of work when swallowing is difficult and slow at times. You say the PEG tube is "tugging at you", what exactly do you mean by that?
My husband was anxious to get his out after a month of not using it for feeding but was not able to swallow his pills (didn't like them crushed). He ended up having to back to using it because his swallowing became more difficult and he couldn't consume all the calories he needed orally. Do push yourself to consume as much as you can by mouth and only use the tube feeding to top up your calories if you haven't consumed enough through the day.