My thoughts don't happen to other people so why are they 'not mine'?

Considering the five aggregates and the sense in which they are all not-self. Thoughts (samskāra) are one of the five aggregates so they too are 'not me' or 'not mine'. In one sense this is a statement which accords with my own experience while on and off the meditation cushion. They display an impermanence and often they arising unbidden (or as part of a web of causal relationships that I don't control). In another sense however this doesn't accord with my experience. My thoughts don't happen to anyone else. They are thoughts in my head happening to me. I know (well I think I know) the second statement is a wrong view. So why are thoughts not mine if they never happen to other people.