I would never have left, I've fished over brush piles with boats 10' apart, and caught crappie. As long as no one is making lots of noise. It's ever body's lake.