Don't worry about it, OK, not a problem?

Hi everyone.. I know there have been lots of discussions about grammar, and I am not perfect as I have come to realise... there are two phrases I need to stop saying because I get replies I hate.. and then I feel c**p!
When I am trying to sort someones problems out, or help them in some way I use the above two phrases, as in! "I'll sort that out for you, dont worry about it" or "ok that's fine, i will get that sorted it's not a problem" .... but then i sometimes get the answer back "well actually it is a problem to me" "or well actually I am not worrying about it, but you should" .. then if someone does something wrong and I am trying to make them feel better I say it again "Don't worry about it" and then they will say.. "Actually I'm not worrying about it" and that makes me feel bad too....... is it me.. or have people got a vendetta to make me feel bad!