Quality varies, as with most things. If your cashmere balls with wear, it is wool off various parts of a Kashmir goat rather than what is real cashmere, which is wool taken from the underbelly of the goat only, and the underneath wool at that.
I had a cashmere jumper from Orvis that had obviously been cheaply made because seams started to come apart and stitches started to come undone at the neck even before I had washed it. I sent it back and they refunded my money.
I also have a cashmere tunic that I've had for years and worn a lot. It still looks perfect. So, yes, if you get good quality cashmere, it is worth it, so long as you intend to use it for years.