When Osborne talks about shrinking the state/the public sector, it's as if all that money which goes towards providing these essential services is at a stroke saved. But just because some public sector workers are not directly employed by the state, their services are needed and cannot be dispensed with. These home carers, nurses, phlebotomists, cleaners, etc. etc. who are no longer directly employed by the state still have to be paid for out of taxes - but via private companies or not-for-profit organisations or, as is increasingly being suggested may happen in future, by introducing direct payment from those people needing the service.
Workers have been re-employed by private businesses at much lower rates of pay and are often treated very badly. In order to slice off a good profit, companies use every opportunity to cut costs - often resulting in low morale and poor, and sometimes unsafe, service provision. Unless people are going to be charged for many more types of service, I don't see how shifting service provision to private companies can, in the long run, reduce the call on the exchequer - but it will make lots of money for many of the companies that donate to the Conservative Party.