I used to take my youngest to nursery school on my bike and then cycle to work. The worst that happened was when I stopped and put my foot down and with my officey shoes I skidded and the whole bike, with strapped-in 3-year-old at the back tipped over with me under it.
We righted it and carried on our journey unscathed - or, at least, I haven't yet noticed any damage and now he's 28.
I carried all my babies in a sling, or until I got dressed in the morning, tied tightly into the front of my dressing gown. I'm sure there is no danger for them. Your hands are free and instinct would make you protect them if you did trip.
What I do think looks odd is when people have the sling over their outdoor jackets or coats. I always tied it on first, then put everything over the top. Warmer for baby, easier to get your coat off when you get indoors and the straps don't have to go round those extra inches, so are not so tight.