Welcome to the Router Forums.
This might be related to the wood you are using for your project. Some woods will leave a lot of areas like this to clean up, others will cut so clean you won't have much cleanup or sanding.
Softer woods tend to tear and leave fuzzies you need to deal with while harder woods cut more cleanly. Also if the wood has a higher moisture content you will get more fuzzy areas that need cleaning.
When using softer woods you can set up your toolpaths to cut close to the lines then make a final pass to help clean up the fuzzies depending how bad they are like David suggests.
Also like Ronnie says you can get cleaner pockets if you use a larger tool to clear the majority of the waste.
Your BRAIN Is The Most Important Power Tool In Your Shop. Turn It On Before You Turn On Any Other Power Tool.
A Disability Is Only A Disability If You Let It Be One