I don't know if I am 100% sure of the setup you are describing, but I can answer a couple of different ways. If you are using a template guide bushing (not a bearing) to follow your pattern, then you can use any size bit, just make passes that are a little less than the diameter of the bit deep.
If you are using a bearing guided bit, (either a flush trim bit or a pattern bit) use the biggest bit that your pattern will allow. A 1/2 Inch diameter bit is always going to be more rigid than a 1/4 inch bit, but if the details in your pattern are too fine for that size bit, then you will need to use a smaller bit. I have a 1" pattern bit, but it doesn't see much use.
You can make life easier on the router bit by removing as much of the material either with a bandsaw, larger diameter router bit, etc. so that the smaller bit doesn't have that big of a load on it when cutting thicker stock.
Hope this explanation isn't too confusing!