jings it'll be fine, honestly. We've travelled on Brittany Ferries from Plymouth to Roscoff, from Poole to Cherbourg, Portsmouth to St Malo, even Plymouth to Santander, and never felt sick in rough weather. (Funnily enough, I'm more likely to feel slightly nauseous if the sea is flat calm and all I can hear is the thrum thrum thrum of the engines!). As people say, it's all down to the weather, but Brittany Ferries have BIG boats, and that's what makes such a difference. They are very stable even in rough seas.
Yes, make sure you have a good supply of stugeron or kwells. Also, get a can or bottle of coke, open it and leave it to go flat. If you start feeling bad, just sip the flat coke - it seems to settle the stomach. I have no idea why it works, but it does!
Have a lovely time - St Malo is so worth visiting, a really fabulous place. And there are some lovely places to visit nearby - Cancale, Dinard (just across the estuary - there's a road bridge). But particularly worth seeing is Dinan, about half an hour's drive south. It really is absolutely beautiful!