This is not a situation that I would like but Annali you need to keep right out of it in the hope of retaining a relationship with your little grandson. If your DL and boyfriend are living together then he has presumably got to know the boyfriend and being collected from daycare by him is a practical decision. If that means his dad can no longer collect him and his rights of access are being denied that is a different matter.
It is a very sad situation for you and your son and you can only comfort him and hope for the best.
Forgetting where you left your keys does not mean you are “losing it”