Should one always defend Islam when spoken of negatively?

I wonder, how is it looked upon to refrain from "speaking out" in these situations, in an attempt to avoid conflict. For instance, in a situation where an individual speaks negatively of Islam, is it acceptable to ignore it and move on with you life, or must it be challenged? Although I always feel compelled to do so, in the interest of avoiding conflict (often with strangers), can it be ignored?