Generally speaking, bigger is better for rigidity but as usual there will be exceptions. Some that come to mind are:
Small bits, say 1/8th inch plunge bit, will always have the weakest and most flexible portion at the smallest diameter, so there is no real value in going to a 1/2in shank version unless your machine can only handle 1/2in bits.
When using template guides bushes, the shank or cutter generally has to pass through the guide bush. Often not an issue, but with Leigh jigs and bushes it normally limits the user to 8mm (3/8in) or 1/4 in shanks for standard bits and bushes.
Again it's horses for courses.