O dear.
I , only child, moved a long way away from my family, and thought my parents would eventually follow, near to grandchildren, lovely safe part of the country etc. My mother desperately wanted to come, my father flatly refused even when she was ill and he could not look after her, and nothing I said or did would budge him. They died in a nursing home, with me driving 170 miles every two or three weeks, which didn't concern him at all.
Your husband is scared of change, and I don't think he will move.
It isn't fair on your son and it isn't fair on you, but I don't hold out much hope unless you go on your own. Actually, 40 miles is nothing.