Hi Mark,
I've been coming up to speed on the AOX for just a few months so I'm hardly an expert. Using a mag 8.6 for guiding sounds perfect. In my experience anything brighter than mag 10 will do (at least in the clear skies typical of El Sauce). Your early description of the problem includes a note that two mag 8 stars were on the sensor. It does sound to me that it might be possible that the sensor is being oversaturated producing a star image which is quite wide and not a pinpoint. Perhaps the two stars can between them saturate the sensor to the point that Maxim is having trouble distinguishing the centroid of the selected guide star. I'd try setting the ACP s/n to 3.0 (works for me) and a shorter minimum exposure time to say .03 just to see what happens. Playing with the binning may be a way to separate the two bright stars on the guider sensor. Or you can just find a new guide star! At any rate, good luck and please keep the forum updated on the resolution.
Cheers,
John