Freedom of Speech: What's the Standard of Harms of Speech Rights?

Some people say that there is no, and can be no, threat to free speech from cultural mores.  That's just consequences for your speech, which is other people using their speech to criticize you, or call for your firing or institutional/associational severance, or fire you or boycott you--which in themselves are just people exercising their rights not to associate with or support people they take to be bigots.  

None of this harms your freedom of speech, because it's not the government and people exercising their own freedom in this way does not stop you from speaking.  It's not true that these things means there are things that you can't say.  You can still say them.  These are just consequences of your speech.

But let's remember that government can't actually stop you from speaking either.  They can also only change the incentives, and provide consequences to do so.  

So those who argue that such cultural mores can't threaten freedom of speech are using the wrong standard to measure whether something harms freedom of speech, since under it even government imprisonment for stating certain propositions is not a speech injury since people can still choose to speak.