If you have a drill press you could make a fence to square it up, bore out most of the mat and fine tune it with hand chisels and a hammer, or do it all by hand and learn to appreciate the horror our ancestors put up with.
I've been in the construction trade since 1978, too often I came across the need to drill perfect straight and square holes to align whatever, too often I failed.
Years ago I made up a bunch of drill blocks off my drill press to take to the job. Simple squares with a series of holes equal to all the bits I carried, nearly absolute perfect holes every time.
You could do something similar, a jig that is square to and fits over the stile, clampit to the stile and drill through it to remove most of the stock.
Never bite the hand that looks dirty!
The more you know the more you're worth