The grain caught the bit and pulled you in. I would get reasonably close and just nibble at it. The smaller the cut the less force you have to counteract. Plant one hand on the board for stability and take small arcs off by guiding with the other. Try practicing just the letters you have trouble with until you get the method perfected.
Someone I consider a master woodworker once told me that a master woodworker is not someone who never makes mistakes. He is someone who is able to cover them up so that no one can tell.