As long as I can see them they can stay. Some of my bigger fish have been caught close to them.
I hate when someone comes along and runs them into the water.