Random token in my wallet, scam?

Occasionally, I will look at my wallet and see a token I did not interact with has appeared. I typically ignore it. I was wondering if receiving these coins means a scammer can drain my wallet? Should I switch my sol to a new wallet immediately?