Is almost every venial sin also the mortal sin of scandal?

Why isn't every venial sin the mortal sin of scandal? If someone knows you are Catholic and sees you commit a venial sin, wouldn't that be scandal? Where do you draw the line? Or is this like lying where it can be a venial sin but there's no test to know so you have to assume it's always a mortal sin in practice?