The downside is that people on the sister boards and in general would be even more inclined to think we are fascists, since they can't see what we're talking about for themselves. They will invent stories about rampant racism or worse, child stuff being on it (remember that the SB PM was also baselessly accused of that).
If your first instinct is "well, who cares what other boards think?" Then you should consider that those places are the main grounds for recruiting more members into this site. We can't feed too much into the rumors about us. The politics stuff should probably stay visible to anyone.
Maybe limit posting there to only members with a certain post count, but other than dissuading both trolls (good) and people who will primarily come here for the politics (bad, we need all the people we can, even if they are mainly interested in the politics) I don't see what it accomplishes.