My daughter was telling me that, rather than buying each other gifts, the members of staff where she works all 'adopt' a child who would not be getting any presents. She Skyped me as she was shopping for the underprivileged, nine year old boy she had 'adopted' to tell me (with tears in her eyes) that all he had written on his 'wish-list' was a basket-ball and books. It is sad that, whilst some children are over-indulged, there are still children who are not so fortunate.
I don't think you are unreasonable at all. I used to take part in these at work and always tried really hard to find something not only suitable, but nice too, often actually spending over the specified amount. However, I did sometimes feel that the person who had given my gift hadn't put in very much effort or thought and had, like you suggest, just found something lying around at home that they didn't want.
I belong to several clubs and group and take part in gift exchanges. The ones where everyone 'buys' a gift worth a certain amount and they are exchanged in various ways i.e. raffle ticket numbers, games etc.
AIBU to expect the gift to at least be new ? Also would be nice to see a bit of imagination put into them. I always seem to get something like the notecards that have obviously been laying around in someones drawer for a couple of years or some cheap chocolates. I know I don't have to take part but I like to get into the spirit of things. Suppose I have to put up or shut up.