Hi there... so I am not a dr. either... however...

Swelling usually equals tissue perfusion (blood? Lymph fluid?) since it's not allergy related I would look to circulatory issues.
(supported by the fact that by late afternoon it has subsided a bit - you're up - on your feet, blood settles and drains as does lymph fluid - swelling goes down). Maybe look into diseases of the lymph system? Try some lymphatic massage (there are people who do this specifically) see if they can help they can teach you ways to do it yourself.

I have had surgery that impedes my lymph drainage in my neck - because of this my tongue is always a bit swollen... they are all related.

No guarantees of course but to follow it up to see if that might be what's going on.

take care and let us know what happens.