My mum had big boobs and my older sister does but mine were always small (virtually non-existent during my teens). I endured a lot of jokes about my lack of curves but I thank my lucky stars for small boobs. It never stopped boys fancying me, nor did the family nose I inherited! After breastfeeding, my temporary bazookas shrivelled away to nothing. My late husband referred to them (affectionately) as a couple of empty baccy pouches! They filled out a bit eventually though. I'm happy with them.
Breasts get bigger during pregnancy because the mammary tissue/milk ducts develop. Apart from that, which shrinks away after weaning, they consist mostly of fat so they will get smaller if you lose weight and vice versa. Trouble is, weight is lost all over so it's slow to show. I think it's a good idea to consult your GP about any changes you notice when you start HRT but it's also worth checking your weight and measuring your waist once a week in case it's simply weight gain. Then you know what to do!