Why many think free-mixing of the sexes is forbidden?

If someone is looking for someone who can give speeches, some consider female speakers as a bad idea, given the general opinion among Muslims that there shouldn't be gender-mixing? Is there a Hadith that says that it is forbidden to mix genders in an auditorium or so? If so where is it and is it authentic?