How many people do you know that have said "I don't want to (smoke/drink/do drugs/sodomize/etc.) because it's not legal!"
I suspect it's quite a small number. The problem here is that this, too, is one of those businesses (vices, too) that involves no victim; quite frankly, if it were state-subsidized, they'd probably make it easier for women to get out of the business if they feel uncomfortable with it. It's also a great way for California to get out of the red, considering most of these brothels pull in high dollar tricks as it is.
I agree that there is a huge moral downfall here, but it's not like you're going to change the fabric of society by legalizing prostitution. Our society will still be overwhelmingly promiscuous either way, and there will still be prostitution.
Eh. I think it's too similar to the marijuana legalization issue.