I came late to the series, and was midplay during my own near apocalyptic event. that alone was it's own bit of crazy. It's a very weird and unreal feeling to play an end-of-the-world video game to unwind from an actual end-of-the-world event.
ME1 was good, but a pretty run of the mill Bioware game. They used a lot of the same themes (side quests, backstories to uncover through player/NPC interaction, the standard 3-character vic) they had on other products. But it really was ME2 that hooked me, and the known universe suddenly got a lot broader and less clean and clear cut. Cerberus was bad, now they weren't, maybe. Characters like Garrus and Tali got deeper, while new ones like Samara and Mordin were superb additions. You got a lot more interplay, a lot more shades of gray, and a lot more "real life." And I liked ME3 right up to the very end, for much the same reasons.
I firmly agree that the worldbuilding was spectacular. I also liked a lot of the interplay with the other characters, and became vested in them. I've replayed ME3 a few times, and Mordin's death always gets me. I was actually shouting "get out of there!" Legion's death was so sad and unnecessary.
If anything, I liked the sidequests and character building a lot more than the main mission of [Don't] Fear the Reapers. It's a world full of people who very much deserved to live, and that begs for further exploration.
Have not played Andromeda. Can't say I want to invest in the upgraded console to do so.