It sounds to me as if the builder did not compress the ground and level it correctly, you should not be having this type of problem. My patio was laid years ago and I don't have this problem.
As it's years ago since you had it put down I don't think you have any recall.
Could you get 4 builders round and see what they say and make a decision after about taking them all up or not.
If you need them relaying ask for references next time, ask to see work they have done previously.
Thing is you can't be putting up with the chance of falling so something needs to be done.
BBC Radio 4Extra Scheduling Shake-Up.