Unsolicited door to door salesman making me feel uncomfortable.

Ten minutes ago I was in the garden busy with pottering about etc. Answered a ring at the doorbell (after taking off shoes, cleaning hands etc.) A young man was there and asked if I was familiar with the new houses being built on the corner - no I wasn't and told him this. He said that surely I knew the location the ones with the wooden windows as he was representing the window people that had installed them. I replied that I was sorry I didn't but could I help him with something and what was it he had called for? He replied that he thought I was a being a bit rude and unnecessary I asked again could I help him with something ( I did think that perhaps he needed directions to the site ) he then said well just let's forget it and walked off. Is it me should I have been a bit more polite to this person who disturbed my afternoon then seemed to get annoyed because I didn't want to have new windows installed. Am considering emailing the company to tell them what I think of their salesman but I think that might be over the top.