Harry should have been genuinely terrified of Voldemort. Even when facing him for the last time he would only just be managing to hide the trembling, because this is, essentially, a vengeful wraith that slaughtered his parents and has haunted him for seventeen years (indeed, I would definitely have gone all in with the "Voldemort's not quite alive." He's regained physical form, but he's almost like a Nazgul now because he has beaten death).
That should all be very frightening for a teenager, let alone the little boy Harry started as.