The problem cannot reside just in the sleeve: it may well be a metric 6mm, but then the nominal 1/4" bit should not fit it at all, let alone slide out. Hitachi does not often make mistakes (have had one for more than 30 years, with 6mm and 1/4" sleeves), but I would not know about your bit set.
I often reflect on how much one can do, if one is unaware that it cannot be done. With enlightenment, often comes paralysis. I built a bunch of bathroom cabinet doors in imbuia (before it became a threatened species) and other doors in mdf, using 1/4" R&S bits in a single-speed router, 1 1/4 hp. Then I read it should not be done, so I went out and bought the necessary, and have never had the urge to make another. Ditto with raised panels.
Denis, the snag is that a number of routers do not come with smaller collets, even in the latest model. I dare say that you are avsolutely correct, but again in ignorance, one can get quite far with the reducing sleeves from the original maufacturer. The only time I have had a problem, is with a push-pull Jacobs chuck (before the muscle chuck was invented) -the sleeve slipped.