I wouldn't do it, but as harmful things go it's pretty near the bottom of the list. I'm happy to wear PJs in the house until I want to go out or take a video call, and if that doesn't happen then I leave them on and change to a clean pair for bed. I've seen comments on here judging that, but I can't see why on earth it matters, or why anyone would care. Nobody sees me except Mr Dog, and he doesn't mind in the least, I save on washing and am comfortable.
Time was when people would talk about you if you went out without a hat, or wore the wrong gloves for the time of year. In years to come this could be looked on in the same way.