You should never just slip away without saying goodbye, as your GS would be totally bewildered if one minute you were there and the next minute you'd gone. Then it would be left to Mummy to explain that Nana's gone home, and surely he would cry then? So it wouldn't avoid the tears, it would only avoid you having to witness his tears. Added to that, he would possibly feel that you didn't think he was important enough, or didn't love him enough to say goodbye.
Young children don't have the ability to control or even understand their emotions. Tears come quickly as a reaction to all sorts of minor upsets and hurts - they are still learning how to cope with them. We shouldn't try to avoid these little upsets, because they do have to experience them in order to grow emotionally.