I would be more worried about contamination from previous infant users of the mats than the remote possibility that a few grains of cocaine might be in the crevices of the mat.
Invisible slight faecal contamination from a baby with some infection would be far more likely to make a child ill than the ingestion of one tiny grain of cocaine. As others have said, always wipe the surface down with a disinfected wipe.
I suppose a few, and even the OP said a few, cocaine users may do this, but I would say that the chance of this happening is very remote.
4 Years On…..Health-wise, Has Anything Changed?