Did you let it sit on the bottom? That's often where the fish are.
I have not, I almost always use a bobber.
Is it as simple as removing the bobber and having the worm sit off the bottom? I assume that does make it a little harder to detect a bite given that there is no bobber to provide clear indication.