I'm not given to frequent posting but this thread has really motivated me.
I passed 'A' levels in Chemistry, Physics and Zoology. However, I admit to struggling a bit with the maths that I needed for physics - but there was no joined-up thinking in the teaching of maths to go with the Physics course. Nevertheless, my career has been long and varied, mostly in STEM subjects. In my late 30's/early40's I decided to study for a BEd in Mathematics. I felt that, because I had experience of struggling with maths, it would give me insight into teaching it. The course was amazing, I would come home, completely buzzing with ideas and excitement (my poor children!). I found that I was, in fact, very poor at arithmetic but a powerful mathematician. I tried for years to get a full-time job as a maths teacher but was always overlooked in favour of younger, cheaper teachers. I was a supply teacher for some time and was quite shocked by the way pupils (boys and girls) were streamed and expectations of them were so very low. I clearly remember one girl who sat at the back and put on her make-up during a lesson. I set the class a problem, and asked them to solve it. The girl solved it straight away. She was so surprised and said that it was easy (it really wasn't - no one else got close). The pupils in the class said they were in the 'bottom set'.
In the end, DH and I were struggling financially and I gave up on supply teaching and looked for something which was better paid.
For the last few years, before I retired, I was in a highly paid job that relied on advanced mathematics and physics.
I can see where the idea that physics is difficult because of the maths comes from - after all, I experienced it. But that was down to poor teaching and low expectation, not ability.
Also I want to add that science is not about learning the rules and following them - it's a glorious exploration of the unknown. There is an old joke about the difference between an engineer and a scientist - engineers don't like surprises.