Your ranks relative to your cohort in each of your subjects are used in determining your Assessment Marks (the mark that reflects your internal performance), which contribute 50% of your HSC marks, the other 50% coming from your Examination Marks (HSC exam marks).
Your chances of achieving an Assessment Mark in the band 6 region if you are ranking low in a subject depends on the quality of your cohort. At a high-ranking school with a more academically capable cohort, your chances would generally be higher, although it would still be a good idea to work on improving your rank as much as possible, so as to maximise your chances of achieving a band 6. At a lower-ranked school with a weaker cohort, a low rank likely limits your chances of achieving a band 6. An Assessment Mark in the band 5 region with a low rank (as you suggested) seems more reasonable. Coupled with favourable HSC exam performance, you can increase your chances of achieving a band 6 in the subject overall.
While your ranks are used to determine your Assessment Marks for your subjects, your Examination Marks depend on your own performance in your HSC exams, and are therefore not affected by factors such as your ranks relative to your cohort or your school rank. With that being said, Examination Marks are also used in determining Assessment Marks. For example, if the highest Examination Mark achieved by anyone in your cohort (including yourself) in a particular subject was 95, the Assessment Mark of the student with the highest internal rank will also be 95.
I hope this helps!