Welcome to the club Spuddo - you might try your drill-press on the aluminum with a clamp-on fence and taking 2mm @ a time. I do this on a regular basis when I do not want to crank up my 1960's Bridgeport mill. Normally I use a end-mill with side cutting but I have used a regular router straight bit at 3500 rpm. Another option is go to AliExpress.com - Online Shopping for Electronics, Fashion, Home & Garden, Toys & Sports, Automobiles from China.
and search for "router speed control" - the Australian cost is under 50$ including shipping. Speed controllers come in several varieties, rheostat , triac, etc. If you have a speed dial on the router, you are stuck and any external speed controller is likely to the fry the router electronics.
Hope this helps - Baker