I see mostly a bad top layer.
This comes in my opinion from too less top layers or maybe (you speak about grinding and scratching) bad bridging.
Top layers should hide the obvious and is a likely evil, additionally it adds strength.
Bad bridging causes a not consistent connection to the support walls (infill) and lifts up in variable spaces.
To increase the binding between the infill and the first infill top layers it would help to slow down, as you already mentioned @mikeyman2171.
Maybe a bridging test would help to see the best settings in this case..
Once again, this seems really similar: