I wouldn't worry about what people think about you "fishing their waters". COE lakes are funded by your tax dollars as well as the locals.