Merlot Charles had the exmple before him of what happened to his great-uncle David when he persisted - and the empty life that he then pursued, alienated from his family and an exile from his homeland. David had chosen a much more notorious woman than Camilla, and would probably have been a pretty lousy king, vulnerable to blackmail on several aspects of his private life, but his fate would have been very much in Charles' mind.
And at that time he does not seem to have had the much confidence in himself. Marrying Camilla might have been the best thing that he could have done, but that is water under the bridge now.