You've had the explanation of what is happening. The router collet is one of the types that is referred to as "non-self releasing". Morse Taper (MT) collets and inserts are also non-self releasing. Many machine tool collets are "self releasing", such as 3C, 5C and R8, which are used in the spindles to hole parts being worked, or in the case of the R8 in milling machines, to primarily hold the tool/cutter that is doing the work.
I'd not thought of it before, but it may be helpful that the router collets are non-self releasing. Gives me a chance to grab the bit before it falls down into the cavity, not that there is far to go.
"If you want nice, clean, fresh oats, you must pay a fair price. However, if you can be satisfied with oats that have already been through the horse, that comes a bit cheaper."