I think the big difference is that, unlike monogamous gay marriage, there are actually legitimate concerns about polygamy such as:
-long-term stability
-level of abuse of wives, particularly in patriarchal families
-level of child abuse.
-girls as young as 12 years of age being forced to marry.
-incest within polygamous families
-high level of welfare fraud
The only argument against gay marriage seems to be, "it will lead to polygamy!" So if we allowed gay marriage and then a group of people clamored for legal polygamy, we could just be like, "um, no...it's not good."
There are actually lots of polyamorous relationships in the US & Canada already (estimated in the tens of thousands of participants), so it might be a moot point.