I agree with you. There's no right answer to it. If a person actually in the trolley problem chose to kill the one, I wouldn't hate them for it. Contrarily if they couldn't bring themself to kill the one and the five died, I still would not hate them.
But here's the thing, the Trolley Problem is a stupid question. It's a mental exercise that's based entirely on eliminating all nuance and choice to pare things down to two options with a single person having all authority to choose life and death, and apparently time to ponder over those options, and yet totally unable to choose a third option no matter what. I would hate the idiot in the trolley for not choosing to yell, blow the horn to alert the people, or some similar action. The Trolley Problem is useful only for vaguely examining oneself, but not very well because you're not in the actual situation and aren't full of adrenaline that's making your reflexes faster than your brain and amping your reactions into fight or flight, and you can take thirty minutes to think it over instead of a quarter second.
Which I tend to think is also the problem with Libertarianism in a nutshell. It tries to simplify extremely complex problems down to the simplest solutions, and in the process loses all the nuance and most of the options that actually make an economy work and make a philosophy interesting. Freedom is good, therefore sacrifice everything else to freedom no matter what.
It's the Diet Coke of philosophies.