A lot of times fish will school with fish the same size. You just need to move and find some more fish.

I have experienced this several times. Once at Nimrod, caught hundreds, all look like they came out of the same mold.

Just leave em' be and find a new hole.
You can't catch the biguns' if they are not there.