I'm not in my shop, so I can't measure my CRB7 pin. What is the diameter difference between the pin diameter and a 1/4" drilled hole? .020" maybe? That means that the pin will fit in the hole very loosely, but gravity should hold the CRB7 in place. When you cut the circle with the router, and the slop between the center hole and CRB7 pin can cause the router to be off position by 1/2 of the diameter error between the pin and the hole, or .010". Are you really trying to cut a circle that accurately with your CRB7 and router?
I guess I should get back to my shop and take these measurments myself to understand your problem, but to me, you are worrying about a problem when one doesn't really exist, at least not in woodworking accuracys.
Central North Carolina