Freedom of speech is a human right as well, and yet we have libel laws. No right is absolute, and in the case of self defense, the fact that you've forced other people to invoke that right in the past seems like an extremely valid reason to demand that you forfeit it in the future, you've proven that you cannot be trusted to wield that power responsibility.
It's the same principle behind banning people that have committed certain financial crimes from ever working in that industry again, they've abused the power they were given before and society has decided that they should not be allowed to do so again, which again violates various rights, such as freedom of association.