Just my opinion, but I think the "complications" of the cancer coming back if you do not get radiation will make whatever radiation effects you are currently worried about seem like minor annoyances. My medical team was adamant that perineural involvement meant it was just a matter of time before the cancer came back a third time if I did not have even more radiation. I did not make this decision lightly as I already had the "maximum" radiation and due to the Erbitux, horrendous side effects. But I believe I would be in a hospice now if I had not gotten the follow up radiation.
If you disregard your doctor's advice and the cancer does come back, you will regret it forever.
Charm