No, i've caught lots of 5" crappie on a 3" jig. Thats what
i use most of the time and it doesn't deter them at all.
Sometimes they just want something smaller though, so You
may want to keep some smaller ones around just in case.