I like to be punctual, or early, if arriving early is not going to inconvenience anyone. I agree that being late is impolite.
Once, many years ago, we were invited to Sunday lunch with friends, as a family. My husband made the arrangement as he knew them through work and no specific time was set. We went to church as a family and went straight to their house, arriving about 12.15 ish. It was far too early for them! We didn't expect to eat straightaway but the wife was in a right tizzy. She was still having a good clean and tidy up, hoovering etc and the joint of beef was not yet in the oven, so we went for a walk round the village. We still had to wait a really long time for the meal and we were all really hungry, especially the children. After that I always made sure to check what time we should arrive when we were invited anywhere.