I take it your fence is not a solid one?
You could try putting a board along the bottom of the fence or, if the gravel is quite small, you could just leave it and incorporate it into your border.
Gravel is good for drainage and as a mulch to conserve moisture. Once it settles down next door you won’t have much coming through the fence.
It is a solid wooden fence with running boards along the bottom but the gravel has still managed to get through
They are the neibours at the bottom of my garden think the gravel was put down last summer but unfortunately it's just getting worse