My son loves tinkering with car engines and (when he lived at home) he used to come in looking grubby. He'd been under the car, and over the bonnet for ages and had "fixed" things, so elicited sympathy and offers of drinks when he came in. No one expected anything more from him for the day. However, he'd been in his element, having fun.
When I have fun (writing, reading, sewing) I am being self indulgent it seems.
My brother goes running everyday and my SIL says the same thing - it's like his hobby, his self-indulgence, is somehow more worthy than her knitting and playing Solitaire on her tablet, or doing a crossword. 
I think some women have an in-built guilt complex. I know I do.
I think I am going to have to brazen it out when I next sit down for 'fun'. 