Hi,
In general the router should be running at it's highest speed with that size bit. The problem you're having can be due to having the bit just a little to deep... hard to tell from the pictures but you need to have a cutting edge at the surface of the wood...
If I were doing this I might opt for using a straight bit for the first cut then come back with the dovetail bit. It would help with getting rid of the chips.
Even after doing all this you may end up with the "fraying" but a piece of sandpaper on a block of wood should clean it up with one or two passes.
Ed
In general the router should be running at it's highest speed with that size bit. The problem you're having can be due to having the bit just a little to deep... hard to tell from the pictures but you need to have a cutting edge at the surface of the wood...
If I were doing this I might opt for using a straight bit for the first cut then come back with the dovetail bit. It would help with getting rid of the chips.
Even after doing all this you may end up with the "fraying" but a piece of sandpaper on a block of wood should clean it up with one or two passes.
Ed