The gravel soil is actually very good. I know they do well in dry shade, but as a gardener I often found that they did extraordinarily well at the foot of west-facing walls. So I looked them up (many years ago now) and discovered that they are from the Mediterranean, Palestine, Syria, Israel, Greece, Turkey, Tunisia and so on. So you are giving them absolutely ideal conditions in which to grow. They will be setting seed from every flower. So your only solution is to pick every flower as it occurs whilst continuing to pull up every little tendril as you see it in the gravel. I have a gravel path, the only place in which it grows happily and the last place in which I want it to be. I pull them all up and transplant them, and lose nearly all of them. All of the original corms I bought with me have died. They are determined to stay in the gravel.