"Wyfe" - the first half of your"wyf-man" - meant "female" on its own, and that was any woman, whether she was married or single. Chaucer's "Wife of Bath" was a woman of Bath (though she had had five husbands since the tender age of twelve - and buried them all) A "wife" in Scots still means any woman as well as a female spouse.
"Just as “history” is not derived from his + story (or as some pun, her + story), as folk etymology would have it, neither is “woman” from womb/woe/wee + man.
The early Old English (OE) wif – from the Proto-Germanic wibam, “woman” – originally denoted a female, and later became the Middle English (ME) wif, wiif, wyf.
By 1175 it was starting to be used to mean a married female, with the two meanings coexisting until the late 16th century. Another meaning of market trader or saleswoman emerged in the late 14th century, which survives in such sayings as “to swear like a fishwife”. "
www.scmp.com/magazines/post-magazine/short-reads/article/3052639/where-word-woman-comes-and-how-it-has-evolved