Hisnibs -
I'm on week 3 of my radiation treatment, and it's my first week of discomfort, pain, and fatigue. Weeks 1 and 2 were, relatively speaking, a breeze. Why not plan to go and then see how you feel on that fifth day? You can always decline at the last minute -- everybody would understand.
Good luck with your treatments. I hope they go well for you!