Yes, you can remove a signal follower's access by going to signal followers page and pressing the red trash can button.