I've often pondered the same question, and everytime came to the same conclusion. Don't go out and buy a whole set or a bunch of Bits, buy them as you need them.
When I bought my first Router, I niaively bought an off the shelf combination set of Bits. Two of the Bits are now both worn out and lost, the remainder (around eight Bits) are still sitting new and shiny in the case they came in.
By Buying them as you need them, 1. your never going to have a bit you didn't use at least once, so you got some moneys worth from it and, 2. the initial outlay is less, and the outlays are spaced apart hopefully allowing you to purchase a better quality Bit.
Just my 2c worth, hope it helps.