IMHO .... that would depend on "how" you were fishing. If you're trolling, you need extra weight only when you need to get the jig to stay down deeper than you can without it ... in relation to the speed in which you can effectively control your boat. If you're vertical jigging, boat movement causes a problem, so you may have to use a heavier jig.
I, for one, use most of the jigs or jigheads and "pretty colors" plastics that I have ... mostly when I'm casting. They're generally 1/32 & 1/16oz weedless jig heads/plastic bodies. If I'm trolling, it's generally a 1/16oz Roadrunner, Whirly Bee Pro, or a 1/16oz hair/feather jig (not weedless).
Like you, it never seems to be calm when I'm fishing ... and if it is, when or where I'm fishing, the wind seems to know that & comes and finds me

So, I do sometimes add a split shot sinker up the line above the jig (~12") ... but, only on the one rod/reel I have that has 10/2 PowerPro braid. On it, even light taps will transmit as pretty hard "thumps". When using the outfits with hi-vis mono, I don't try to "feel" the bite, but watch the line for any movement that I know I didn't cause. I cast the jig out, raise the rod tip up to about the 10 o:clock position, engage the reel & start my retrieve. I only reel fast enough to keep a slight bow in the line, and I watch the line for any slight jumps/jerks or a sudden slack in the line (when I know the jig couldn't possibly be on the bottom). Any of those "movements" in the line indicates a strike, and I set the hook immediately upon seeing them.
And ... if you like using minnows, or have more confidence in them :
Ain't no law says you can't put a minnow on your jig or jighead/plastic grub or a plain jighead ... & cast, troll, spider rig, vertical jig, deadstick it, or even fish it under a bobber !! You simply run the hook up from under the minnows mouth (in the V shaped section of its throat) and bring the hook out between the minnow's nostrils.
I'm not all that good at math, but I do know that any boat movement is going to produce a angle in your line ... even if you're using several ounces of weight. The prime factor in that scenario is that you understand that 10ft of line out is not putting the jig at a 10ft depth. You have to consider the distance between the rod tip & the surface of the water, when fishing straight down ... and you have to consider that the greater the angle of the line, the less depth your jig is running in. You compensate for those situations by letting out more line, equal to the distance needed to reach the depth you want your jig to be. That's where the "math" comes in

... and, though I'm sure there's an equation that would quickly tell you the depth of your jig, in relation to the length of line out & the angle of that line ... I don't know what it is

But, then I'm not normally trolling or dragging my jigs or minnows around, so I don't fret over it all that much.
... cp
