'how small is too small' isnt really a valid question
if youre drawing the line at the minimum it takes for a girl to orgasm, its probably around 4inches considering plenty of girls can get themselves off from just fingering alone. It depends on the girl.
if youre drawing the line at a desired minimum size, its probably around 6" as an average. The bigger the penis, the more visually and physically stimulating it is. Any girl who says she doesnt care for a reasonably sized cock is lying. Anything below 6" becomes less appealing, anything above becomes a welcomed bonus.