You don't say what kind of cancer this was. Some behave very differently than others. SCC seldom invades by perineurial invasion along the nerve pathways, other do it all the time like ACC. So it's hard to comment very specifically. If you've had perineurial invasion, it's not possible to say that the surgeon got everything, because they certainly did not chase any disease down nereve pathways where they have a very difficult time actually seeing the cancer.

Choosing not to do radiation that your doctor is recommending is a dicy decision. They are recommending it for a reason, one that their many year of experience is dictating. It comes with downsides for sure, depending on how much, what type and where. It's not universally a problem, particularly if it is very targeted and avoids vital structures, and could be at lower doses. So there is no one size fits all answer to your question. If you don't do it and this comes back, these cancers are much harder to deal with the second time around. The only people that Ive seen here that did not have radiation as part of their treatments are those that were very early finds at very early stages.