Well not having X-Men was probably a good thing, the Mutant storylines tend to not mesh well with the rest of the Marvel Universe and frequently things make no sense there. Spider-Man is probably the most popular Marvel hero so I see little chance he wouldn't have shown up by at least Avengers...