If you actually don’t want to go at all, then don’t. You can decline politely using one of the answers suggested, with or without explanations.
Or maybe if you want to go, you could manage a shorter time. If starts at 3.30, then 7.30 is a reasonable time to leave.
I fully understand why you feel it is a problem to stay with them, but it is not clear why staying in a hotel is not an option, either the night after or the night before.
Or simply decline, but suggest meeting half way at lunch time, on another date, especially as a shorter drive is better for you.