In fact, it is an insult to those who don't wear a white poppy, or any poppy, to assume that they don't want peace.
It is getting a bit like those facebook shares "Like this and share it if you . . . . ." implying that if you don't "like" and "share" it then you are a miserable bugger who doesn't give a shit about the thing the poster is promoting. Or shouting out at the theatre that you believe in fairies, because every time a child says they don't believe, a fairy dies.