Dome error status

[Bob Denny]

OK, yes Skype and Thursday afternoon your time, around 2100 UTC?

[Stephane Basa]

Perfect

[Stephane Basa]

Bob,

Unfortunately the problem came back last night: the dome is in error and the whole system is stuck. Let me explain:
- We launch the system via the java scipt given in Startup.txt.
- The telescope connects, but the dome goes in error.

By analyzing the ASCOM logs, we see that everything is good on the side of the dome. What is disturbing is that the status of the dome is not asked when ACP puts the dome in error. With Michael, there is something that escapes us (I attached some comments from Michael just below) and it now seems clear to me that you have to take a look. This is not satisfactory because it prevents activating the scheduler with the quiet mind.

Stéphane

ACP indicates an dome shutter error...:


17:24:20 Open dome shutter.
17:24:20 ** #domeopen - Unexpected shutter status 4
17:24:20 Shutter : ERROR MESSAGE.
17:24:20 Please contact expert
17:24:20 **Failed to open the Dome Shutter
17:24:20 **Script Error (Tracking has been stopped)**
17:24:20 Source: ACP Observatory Control Software
17:24:20 Message: ==== Startup failed, cannot continue
17:24:20 Location: line 212 column 5.


... but never asks our driver about the shutter status


2018-01-11 17:23:55.199 I => execution(Dome.setConnected(..)) arguments('true') (classes.ck a:77)
2018-01-11 17:23:55.199 I <= execution(Dome.setConnected(..)) arguments('true') return() (classes.ck a:77)
2018-01-11 17:23:55.199 I => execution(Dome.canFindHome()) (classes.ck a:77)
2018-01-11 17:23:55.199 I <= execution(Dome.canFindHome()) return(class java.lang.Boolean:true) (classes.ck a:77)
2018-01-11 17:23:55.200 I => execution(Dome.canPark()) (classes.ck a:77)
2018-01-11 17:23:55.200 I <= execution(Dome.canPark()) return(class java.lang.Boolean:true) (classes.ck a:77)
2018-01-11 17:23:55.200 I => execution(Dome.canSetShutter()) (classes.ck a:77)
2018-01-11 17:23:55.200 I <= execution(Dome.canSetShutter()) return(class java.lang.Boolean:true) (classes.ck a:77)
2018-01-11 17:23:55.201 I => execution(Dome.canSetAzimuth()) (classes.ck a:77)
2018-01-11 17:23:55.201 I <= execution(Dome.canSetAzimuth()) return(class java.lang.Boolean:true) (classes.ck a:77)
2018-01-11 17:23:55.201 I => execution(Dome.canSetAzimuth()) (classes.ck a:77)
2018-01-11 17:23:55.201 I <= execution(Dome.canSetAzimuth()) return(class java.lang.Boolean:true) (classes.ck a:77)
2018-01-11 17:23:55.201 I => execution(Dome.canSetAltitude()) (classes.ck a:77)
2018-01-11 17:23:55.202 I <= execution(Dome.canSetAltitude()) return(class java.lang.Boolean:false) (classes.ck a:77)
2018-01-11 17:23:55.202 I => execution(Dome.getName()) (classes.ck a:77)
2018-01-11 17:23:55.202 I <= execution(Dome.getName()) return(class java.lang.String:NTM) (classes.ck a:77)


After that time, ACP does not ask the Dome for more values (eventhough the error appears 25s later)


This was the same last time, so we do not understand where ACP gets the shutter error from if is does not ask our driver about it.

[Bob Denny]

Looking at your MyOpenShutter code, and the log StartupObs-xxxx.log, it is clear that the ShutterStatus reports 4 (ShutterError) already when the function is called:

Snap1.png

You can see that the first thing logged after the call to MyOpenShutter()

17:24:20 Open dome shutter.
17:24:20 ** #domeopen - Unexpected shutter status 4

Thus no calls to ACP for dome activity are made within MyOpenShutter()! That is why nothing is logged in the dome controller. This logic seems troublesome on several levels. I think it is trying to make up for the Dome controller reporting ShutterStatus=4 (ShuttterErr0r) when it should not be doing that. The 15 second delay after the call to OpenShutter()... why? And another 10 seconds, again for Shutter Error? I think the "problem returned" because the dome controller is reporting ShutterError (4) immediately when MyOpenShutter() is called.

The real solution to this problem is not to try to accommodate buggy behavior (reports ShutterError during "normal" operation) in the startup script. It is going to choke again later when ACP Scheduler tries to open or close the dome. It really needs to be fixed at the controller level, then the simple and reliable code can be used, and also the opening of the dome in the startup script can be avoided (so the Scheduler can operate normally always).

[Stephane Basa]

Bob, I admit to being even more lost after your answer. I agree that it is due to the fact that a "ShutterStatus=4" is returned. However, when you lool at the ASCOM logs, there is no error somewhere: how such an flag can be raised? Is-it really coming from the dome or is-there a trick somewhere (my question is probably, but there is something that really escapes me)? To be honest I am not (yet) convinced that it is coming from the controler: I really need more elements to discuss with Michael.

For the scheduler, I can let it manage the dome. But, when schools are observing, it is desactivated and we need something automatic to open the dome. I then used once of your script and slightly adapted it. If you have suggestions, they are more than welcome.

Stéphane

[Bob Denny]

Stephane, I am sorry I tried to explain the best I cold. The logic in your startup script... it is skipping all of the code that opens the dome because the dome is showing an error already, before that function has even been called. That's what the yellow arrows are showing. Let me see if I can explain in English:

1. The dome is already in error state
2. Call MyOpenShutter()
3. The dome is available and it has shutter controls, so
4. Look at ShutterStatus. Is it 1 (ShutterClosed)? No, so follow the first yellow arrow (skipping all of the Dome Open code)
5. Look at ShutterStatus. Is it 0 (ShutterOpen)? No, so follow the second yellow arrow (skipping the return 2)
6. We are now at the last "else" so print "** #domeopen - Unexpected shutter status " + "4" (ShutterError, the dome is already at error status, right?)

Indeed this is what is in the StartupObs log. There has been no activity to try opening the dome because it is already in error state. If the ShutterStatus is 4 when MyOpenShutter() is called the effect will be to instantly print the "** #domeopen - ..." message.

Somehow the dome needs not to report ShutterError, ever, unless there really is some sort of error (stuck shutter, broken cable, frozen, electronic failure, etc. etc.). This is a basic rule: Do it right or report an error. Reporting an error should/must never be part of routine operation.

If someone else is reading this and sees that I have missed something obvious, or that I am using poor communication skills, please feel free to jump in and help.

[Stephane Basa]

Bob, thank you very much for your response always very educational, but obviously it's me who am not clear. I totally agree with your analysis, but what I do not understand is the following: the controller does not indicate any error, while the ASCOM driver returns one. How can that be? Who manages this status?

Somehow the dome needs not to report ShutterError, ever, unless there really is some sort of error (stuck shutter, broken cable, frozen, electronic failure, etc. etc.). This is a basic rule: Do it right or report an error. Reporting an error should/must never be part of routine operation.

I also totally agree and will rethink my program. But like you, I would like to understand where the ShutterError comes from. This can possibly hide a bigger problem (there are rarely miracles in computer science ).

Stéphane

[Bob Denny]

the controller does not indicate any error, while the ASCOM driver returns one. How can that be? Who manages this status?

ACP asks the driver for it's ShutterStatus with a standard one line call for the valuue of Dome.SHutterStatus. This is coming from the driver. ACP does not "filter" or otherwise interpret it.

I would like to understand where the ShutterError comes from. This can possibly hide a bigger problem (there are rarely miracles in computer science ).

Ultimately it is coming from the dome control system, shutter position sensing. I know nothing about the insides of it, from the mechanical shutter to the code that handles the requests for ShutterStatus. Somewhere in all of that, the value of ShutterStatus is coming back 4.

[Stephane Basa]

I think we now have a fairly close analysis and I will now look for the ASCOM driver in more details. I asked Michael about this and of course I will give you more infos.

[Stephane Basa]

Bob,

With Michael, we have made good progress on this problem and we think we have identified the stage that causes concern. It happens when the telescope is connected through the command:

Telescope.Connected = true;

During this step the telescope connects and makes a unpark: we saw clearly in the logs that a connect in ACP triggers the isAtPark inquiry and an UnPark if the former is true.

What disturbs us is who asks for this UnPark? This does not come from the Astelco driver and the ASCOM manual seems to say that it is not a basic command. Could you say more on the implementation of this command (is-it encapsulated in ACP)? It will help us to identify the guilty.

Thanks,
Stéphane