Hi
At this point a picture is worth a 1000 words...the sleeve should Not come out,,the bit should slide in easy and should not put marks on the bit shank..
The shank on the bit should not have any scratch marks, it must be true in size...it can spin at 24,000 rpms.and a scratch on the shank will cause it to be out of round..
and will not spin true...
The adapters are a bit over size and a bit under size , that's so they can work in the router collet nut...
==========