OldWoman you are not that much older than your neighbour but in any case it is not unusual for there to be a fixation on older women who are seen as more vulnerable and therefore easier prey.
Find out what work is being carried out, if any, and for how much longer you should expect the scaffolding to remain around their house. Certainly distance yourself from your neighbours and use vertical blinds, which can be angled appropriately so that the neighbour cannot see in, and not leave you feeling closed off. Diarise all instances of creepy behaviour and make sure that family, other neighbours and friends are aware of your concerns. Are there any other neighbours who may be affected by this man? If so then you do need to speak to the police about it. Better safe than sorry!