The touchscreen calibration routine is well understood, it’s just a set of gcode it automatically issues. You could try and run a calibration by issuing the gcode line by line and see on which exact step it fails.
I posted the code for someone else to do an 11x11 grid here: Is automatic level compensation working properly?
You would change the 11 in the first command to 3 for a 3x3, or 5 for 5x5.