Following Mark Somerfeld's suggestion, I drop a half inch grommet into the bottom of the collet so all the bits bottom out at the same depth, about an eighth of an inch from the bottom. I just eyeball it on the 1/4 shanks using a mark if I see one, or must bottom it and then pull it up an eighth or so. I guess you could find a small enough slug of rubber for the 1/4 collets, but haven't found such a thing yet. The small O rings might work out OK, but use a couple of them. Stick's suggestion is fine, but I prefer the grommet.
The more I do, the less I accomplish.