If you are prepared to have the cat living in your garden on its own terms, to feed it, make sure it has clean water every day, including while you are away on holiday, and that it has a sheltered place to sleep - go ahead and keep on feeding it.
If you do not want to take responsiblity for it, chase it out of your garden now!
It may become tame, or it may not. But by accustoming it to be fed by you, you must realise now that you are responsible for getting it to a vet if it is hurt or ill at any time.
Next point: if it is female it will have kittens - loads of them, around six in an average litter, six or eight times a year, so you will either have to catch the kittens and have them put down, or catch the mother cat when there is not a litter depending on her, and either have her neutered or put down.
However, adorable they are, you cannot find homes for all the kittens she will produce.
If the cat is male, the kittens he sires will not be your headache, unless of course, he persuades his mate or mates to move into your garden, but unneutered tom cats spray their territory and their urine stinks.
So, it boils down to: do you want a cat, or not? If you do, talk to the animal every time you see it in your garden and it will probably become willing to come near you. Have a suitable cat carrier or basket with a lid handy, and as soon as you can, lift cat gently, plonk it in basket, FASTEN THE BASKET VERY SERCURELY and go to the nearest vet, or animal shelter, if you do not want to keep the cat and cannot face having it put down.
If you intend to keep the cat, have it neutered, vaccinated and either tattooed or chipped so you can prove ownership.
And welcome to the world of cat lovers and all the pleasure they bring.