How do I stop my dog from having friendly interactions with potentially dangerous people?

I have a one year old Pembroke Welsh Corgi, he loves people and other dogs and he's very socialized and lovable. However this becomes a problem sometimes, there are a lot of bums wondering around my neighborhood and I take him out three times a day, every time these bums would pet and play with him and he would get super excited and it can last like 5 minutes before the bum walks away, as time goes by I think my dog has become friends with these bums which makes me really uncomfortable, I know a lot of them are harmless but I also know that some of them are mental... if I try walking away my dog would lay down and refuse to walk, and I feel rude to pull him away forcefully while the bums are playing with him, that will just me look like a mean guy, I need my dog to stay away from potentially dangerous people, please help. I know my dog will attack the heck out of anyone who tries to kill me, but I doubt that's ever gonna happen...