A couple of possibilities. How was the stock held down? If clamped around edge, possibly not flat everywhere or lifting do to upcut. This may be a good application for the blue tape and superglue method.
How did you zero off the top of material? If you use touch off puck, make sure that the thickness is set correctly. Mine seems to have not read some types of bits as consistently, have mine back off to 1", and double check with a 1-2-3 block.
Several sharks that I have seen tend to have a little flex in the gantry do to the unsupported rails, possible that "grabbing" of the aluminum is pulling the entire Z carriage down a slight bit. If you push up or down on the collet, is there any movement?
Do you know the alloy of the aluminum? I have had better luck with harder alloys (6061) than with softer (3003) alloys. The soft ones are gummy and grab more.
When cutting aluminum, I like to lubricate the bit with this spray. Use it all the time, but it seems to make a more noticeable difference with aluminum. May not be as necessary if you were using wd40, I have been cutting dry most of the time (or just air blast).
Tubular latex pressure vessel configuration engineer