A few more thoughts, do you see any bed wobble like this:
I don’t think this could be your issue but nobody knows:
Your tool sits tight in the collet, no movement in Z, right?
Your quick swap kit sits in the end position, no movement in Z, right?
Asking this because I have experienced these issues.