The same thing happened to me, but the bit was in an 1/4" adapter. A 1/4 collet fixed it. But, you say you are using a 1/4" collet already. I can only suggest getting it as tight as you can. and, maybe, roughing up the shank with some sand paper.
'The true soldier fights not because he hates what is in front of him, but because he loves what is behind him.' G. K. Chesterton
Please edit your profile with a name and location so we can better assist you and make for a friendlier forum.