Time for a talk, I agree. Grandad is doing the children no favours by being a Lollipop Grandpa.
YWhen children know they can break guidelines and rules with impunity, they don't feel safe.
After all, you are showing weakness if they, the children, can change your mind.
They need to believe you can protect them from everything, cope with everything (even if you think you can't ) You are in charge.
So make your 'no' mean 'no' but not 'because I say so', they need a reason which will make sense to them.
You can certainly be honest with the two older children, and they can help you manage the younger ones.
You (and Grandad) are not there to be manipulated but to keep them safe, happy and healthy. You can only do this with a respectful and kindly set of rules which he must honour as well.
You certainly do have quite a nestful there and to be pulling against other carer-adults is the last thing you need.