My suggestion is to not "invite" a conversation with your verbal challenge. Maybe something like "Could you hold up right there?" ....my plan was to turn toward them and challenge them with a loud, "Can I help you?"
Since they were pacing/following you, maybe you just keep moving, like you were. Working to improve your position and checking on them to see if they were closing distance and watching to see if you were being pushed into an ambush.