That's maddening, I know.
I have one 1/2" shank straight bit I use to groove with. It does the same thing. I called the seller and they gave me the specs, then I miced mine. It was right on the money.
No other bit gives me trouble, just this one.
Have you checked to insure that it's a true 1/4" all along the shank?
You might try giving your collet a good cleaning. Tho, I did that and it didn't help.
You also might try roughing up the shank with so 80 or 100 grit paper.
Sorry I couldn't be more help.
'The true soldier fights not because he hates what is in front of him, but because he loves what is behind him.' G. K. Chesterton
Please edit your profile with a name and location so we can better assist you and make for a friendlier forum.