Slather a good coating of Slab Sauce on those artificial baits & see if that helps.
Seriously though, I'm subject to having many different "flavors" on my hands when out fishing (oil, gas, food, nicotine, etc.) and I still catch fish on jigs. Not that there's anything wrong or "cheating" about using live bait along with artificial bait, as I've used both separately and in combination for several species over the years.
What really matters is you catch fish, have a good time doing it, and return home safely to enjoy the fruits of your efforts.