Nick may be on to something. The arbor could draw to one side or the other if there is lateral play in the bearing. The blade may remain tight to one side on the right hand cut, but move from one side to the other on the left hand cut causing it to appear that the tracking is not parallel to the grooves.
That is a rare defect and reaching. If that were true, you would notice any of that by the kerf in the bottom of the sled. As you cut from one side it would be tight against the kerf on one side and have room on the other side of the kerf, between the blade and the kerf. Remember that a blade cut the original kerf and should still be a zero clearance kind of affair... That kind of wear will want to pull the work "to" the blade, towards where the most force is applied... Aloose bearing also has the tendency to wander to the limits of it's looseness and follow the grain, resulting in an un-straight cut.
But if an arbor bearing is loose, they are not like wheel bearings. They are not tapered bearings. They are in fixed races(, most of them are sealed bearings) and not adjustable. There is no adjustable preset. If they are off, they are off and should be replaced. If they have lateral movement because of race wear, they will also have wear enough to pivot slightly on it's axis. And that kind of wear, you can "feel" by trying to move the blade by hand while it is attached... (unplugged for safety)
Thank you Nick and Charles for your suggestions. Charles I favor your suggestion of slight deflection in blade and arbor. I do see a slightly wider groove at the head of the groove on the side that isn't square.
Nick, I have been careful to keep the pressure straight forward. Also, I get the same behavior with all three of my blade types. I wiil look for a lifting on one side. On my recent project I was crosscutting a half inch thick piece of maple which is 4 inches wide. The maximum deviation from square is about 1/32 over the 4 inches.
Let me explain a different way so you can visualize something... Think of a loose bearing, bent blade or blade deflection in a different perspective... Say you have a bent blade. It would act like a wobble dado. Assuming your fence was true (as you implied), the cut would be true with a kerf wider than the blade. If the arbor bearings are loose, the arbor will deflect with pressure "each" side-- be a wider kerf and have burning one the side of the most force. The toe of the blade gets pushed away, as the heel of the blade gets pushed back against the work. That is the side the heel of the blade will be deflected and scuff against the work. If the bearings have play, it will have both burning, wide kerf and have rough saw marks. If you change sides to cut, those characteristics will change and move accordingly. You see blade deflection more often in high angle bevels and miter cuts... not usually straight up and across a crosscut or rip (unless. of course, it is really really dense
Blade deflection "cut's" will not result in a straight up cut nor a consistently "straight" cut across through wood... Meaning, if a blade is going to deflect, it does not deflect consistently across the whole cut... nor does it delfect just on one side of the blade more than the other (changing directions or sides) rather you see a flutter in the cut and it is not consistent saw marks. Wood is just not consistently dense and you are not using a power feeder. You are not going to get a controlled, "consistent rate of deflection" outside a CNC Saw. I see blade deflections... but if a blade deflects on one side, it deflects as much on the other (normally). Unless... as someone mentioned on one side being dull... but those have a wandering kerf that pulls on the work... and you see that in the cut... More common or possible would be that the fence is off by a small amount.
0.03125" out at 4 inches... 0.09375" out per foot... 0.5625" out at 6 feet. That's would be like my 6 foot crosscut fence being off over a half inch out at 6 feet. It adds up.