Average in Advanced English is not as bad as it may sound. It's a challenging course for those who are not English oriented. If you go to Standard, you will need to be pretty much coming top 5, and still do amazingly well in the HSC in order to equal maybe a slightly above average Advanced student (depending on the cohort). If you compare going badly in Standard compared to going equally bad in Advanced, Advanced will give you a higher mark, evidently. But Standard is slightly easier than Advanced; all skills learned in Standard are the same as Advanced; you still need to analyse a text, give a perspective, answer a question etc.. The texts are just slightly easier and the modules are different.
Your teachers are just worried that you will pull down the whole grade, but really, it's the same statewide. There are going to be cohorts with great and not so great and average students. And to be honest, you only need to do well in Yr 12. Use Yr 11 to build up your skills and knowledge, and you can still go decently well in the HSC.