I too have had a taste of what you describe and it is difficult to cope with.
I hope your DH's fracture will mend, so that he will become a little more mobile again. This has happened for us and made a great difference, as we are beginning to get out and about again.
If you DH no longer is able to drive once his fracture mends, I suggest you look into other means of transport, such as an invalid scooter with three wheels or a three wheeled bike, but chose one with an electric auxillary motor. PM me if you need more details.
Is he willing to use a stick or any other aid to prevent falls in the future?
I assume you already have checked the house for rugs with turned-up edges, flexes, lintels and any other hazard he could trip over, plus handles to hold onto when getting in or out of the bath, off the toilet etc.
About adjusting your social life, that is more difficult. You can and should be able to go out to some things yourself, so you don't loose friends and outside interests. To me it sounds as if your DH can manage for himself for a couple of hours a week.
Any chance of interesting him in something you could go out to together? I'm asking because my DH spent two years sitting on his BTM watching TV, but is now that his shoulder is better beginning to take an interest in the outside world again, but it has been uphill work.
To me you do not sound selfish at all, but a concerned and loving wife, facing the problem of a husband who is maybe dwindling into invalidism. You do need and must have some time to yourself to preserve your interest in life and in the last extreme your sanity.
If you can afford it, try to get some help in the house, it will give you another person to talk to, if nothing else.
Try to invite friends in for a cuppa - I hope your DH like mine will make an effort to be sociable and feel the better for it, when someone comes.
Hope this helps. Keep your chin up! PM me if you like.