MrBaggs often drives faster than I'm comfortable with as a passenger. I have full confidence in his driving skills. The problem is that, as a passenger, you don't anticipate car movements so you get sloshed about a bit. It's not comfortable.
Once he understood that I wasn't criticising the safety aspect of his driving he was more accommodating, though he does need reminding sometimes.
I drive differently if I have passengers, not more or less safely, just with more consciousness of my passengers' comfort, taking curves in the road more gently, for instance, and accelerating less fiercely (never that fierce! ) than I might when I'm on my own.