Yeah, the successor states not really having the resources to expand over much into the rim makes sense. It seems to be heavily implied that the inner sphere itself is pretty heavily underdeveloped during this time: if some entprising clan wanted to go off and stake its own claim to some unclaimed land, most of the inner sphere is so underpopulated that founding a new town for yourself is probably just a couple of hours of rough overland trucking, and then once you get the road built its an hours truck drive to get more supplies/market access.
So, if you want to play colonist, most people just have to move to a different part of the same planet, and not even necessarily all that far, which is much cheaper and less risky than trying to set up on a whole new planet. And, if you do have some situation where the planet just isn't big enough for ya, there's probably a planet thats habitable and mildly settled, or at least has some ruins to salvage to jumpstart your colony a mere jump away from your current planet. Now, I'm not quite sure of the details of BT FTL lore, but if the games are anything to go by, you can only travel fairly short distances by jumps. Assuming each jump is expensive in and of itself, it makes sense to save your money to buy gear and supplies rather than traveling, so if colonizing one jump over gets you a near empty world, no point going, what, half a dozen, a hundred jumps to get to the periphery?
About how far is it from Terra to the Periphery anyways?
The time you would expect pushes into the Periphery was during the Star League, where free real estate in the Inner Sphere was assumedly much less common. Which, we saw an attempt to do so, but that effort was frustrated by the fact that they weren't hitting an empty ring of totally free real estate, but populated and hostile worlds.
Now, why the Periphery itself didn't keep expanding further and further out is the better question. I'm not sure on the timeline, and thus the obvious answer of "they haven't gotten around to that yet" is plausible or not.