James -- I was hoping to keep this simple, but in the process I sort of ducked some of your questions, so...

Can you tell me where in the ACP the error message: "SHUTTER FAILED TO CLOSE: This is a dome controller problem" comes from?
It's internal to ACP.

What is it checking for to determine there was an error?
Read for yourself. It's easier than me trying to explain what code does in English :-)
Code:
    If g_bDomeLive And g_bDomeAutoHomeClose Then
        If g_DomeCtrl.CanSetShutter Then                    ' If we have a shutter/roof
            '
            ' If unsafe to close till scope parked, park scope first!
            '
            If Not g_bDomeClosedScopeSafe And g_CorrScope.CanPark Then
                bDisconnected = g_CorrScope.AssurePark(DiscScope, True)    ' Do best to park, use long wait extra safety
                bScopeParked = True                             ' No need to do this again below
            End If
            '
            ' Close the shutter if supported. Note that this will cause the TI dome to home.
            ' Thus, this needs to be done before trying to park, as the last thing we want
            ' the dome to do is move to the park position.
            '
            If g_DomeCtrl.ShutterStatus = shutterOpen Then  ' If Open, close it
                '
                ' NOTE: CloseShutter() also has the logic to first park the scope, and
                '       that already honors g_bDomeClosedScopeSafe, not parking the
                '       scope if that is True.
                '
                g_DomeCtrl.CloseShutter                     ' Asynchronous
                Do While Not (g_DomeCtrl.ShutterStatus = shutterClosed Or _
                                    g_DomeCtrl.ShutterStatus = shutterError)
                    g_Util.WaitForMilliseconds 1000
                Loop
            End If
            If g_DomeCtrl.ShutterStatus = shutterClosed Then
                agt.Speak "Shutter is closed."
            Else                                            ' If not closed PROBLEM!!
                agt.Speak "Can't close the dome! Shutter error or in motion."
                g_Console.PrintLine "**SHUTTER FAILED TO CLOSE! This is a dome controller problem**"
                g_Console.PrintLine "**Error: Shutter error or shutter still in motion**"
            End If
        End If
        '
        ' Now, park the dome if possible, otherwise just home it. The TI
        ' dome will already be at home as a result of closing the shutter.
        '
        If g_DomeCtrl.CanPark Then
            If Not g_DomeCtrl.AtPark Then                   ' Unless already parked
                g_DomeCtrl.Park                             ' Asynchronous
                Do While Not g_DomeCtrl.AtPark
                    g_Util.WaitForMilliseconds 1000
                Loop
                agt.Speak "Dome has been parked."
            End If
        ElseIf g_DomeCtrl.CanFindHome Then
            If Not g_DomeCtrl.AtHome Then                   ' Unless already homed
                g_DomeCtrl.FindHome                         ' Asynchronous
                Do While Not g_DomeCtrl.AtHome
                    g_Util.WaitForMilliseconds 1000
                Loop
                agt.Speak "Dome has been homed."
            End If
        End If
    End If
Is it possible for me to add a few lines there to make it try again?
Unfortunately no it is internal and very mature code. Plus I am not going to add "guess what's wrong and try it again" code anywhere. For more on this see Policy for Working Around Device Errors, Quirks, and Limitations.

Assuming I cannot fix this possible driver issue, is there a way to make ACP call my script to open/close the dome via the serial port?
There is a way... write another ASCOM driver for the AstroHaven. I am not joking here. I won't become involved in any low-level programming for devices I know nothing about, and for which I am not being paid, so you would need to offer me a standard interface... so you would need to write your own driver.

Again I would press the issue with AstroHaven.

PS The hooks via which the geostat stuff runs does not have a way to make up for failing dome control. It's not in that part of the logic.