The Physics course was developed to prepare students for uni physics. It is a good stepping stone to uni physics. I am doing advanced physics at uni, the maths in it, although there is more of it, is not necessarily extremely difficult(and probably wont be until later). Basic 2 unit maths would be more than enough so far in the course, much of the harder maths you need in advanced physics is learnt in conjuction with math courses that you learn at the same time. The maths most people imaging physics having is the maths done in latter years which, if added into the HSC, would be way beyond most people's ability. But, like I said earlier, HSC is not to teach students the maths that can be easily taught within one lesson at uni, but rather learn the concepts required, that take much longer to grasp and understand.
.: HSC Physics is good the way it is