Filter wheel hangs, ACP doesn't recover and needs restart

Hi Bob

I'm seeing some strange failures to recover on error that I could do with some help on - It's not urgent, but it would be useful to know if you can help with a 'quick fix'. ACP is version 8.0 build 6 service release 1, and Maxim DL is 6.13.

I am seeing an issue where my filter wheel occasionally locks up. I'm pretty sure it's a dodgy DC cable, though it might be a dodgy driver :). Nothing you guys can do about that!

However, if the lock-up happens, then ACP gets into a very weird state that needs a complete restart to recover from.

The cascade of events that ends up with me needing to restart is:

- ACP working along just fine, tries to take an image (usually a pointing update, I guess because the dodgy cable gets loose after a slew to a new target)
- Maxim tries to move the filter wheel and fails.
- Maxim looks OK after this failure, but any attempt to take images with it after this gives a filter wheel error. Power cycling the wheel usually fixes Maxim, but sometimes Maxim needs to be restarted to recover.

Now, that's a bit painful and I'm troubleshooting it separately. I don't imagine it's anything to do with ACP though. Hopefully if I sort out my filter wheel power, then I won't hit this again!

My worry is that ACP doesn't really seem to handle this failure very well, and wondering if you've got any suggestions:
1) ACP doesn't notice the failure, and just sits waiting forever for the exposure to finish, leaving the mount tracking. Not sure if that's an ACP issue, or Maxim failing to return the right status. Log attached from ACP, but there's no detail in there that I can see - you will see a log at 12:07:08 UTC taking the pointing exposure, then nothing. The next log is at 16:09:46 UTC when I manually stopped and restarted Maxim, around 4 hours later to try to recover:
16:09:46 **Pointing update error from Microsoft VBScript runtime error:
The remote server machine does not exist or is unavailable: 'Camera.CameraStatus'

I imagine this means that Maxim simply hasn't returned an error message or something to ACP, but it would be really nice if ACP had some sort of timeout to detect this rather than sitting there forever with the mount tracking. It knows that it's trying to take a 10 second exposure, so if that doesn't return for 4 hours, then presumably you could run a timer that fires and takes recovery action? Even if that's just to shutdown - anything would be better than letting the mount continue tracking, and possibly hitting the pier. Could you look at adding something like this?


2) The other symptom is that if I manually stop/restart Maxim to recover from the filter wheel lock up issue, ACP doesn't notice that the old instance of Maxim has gone, and doesn't try to re-attach to the new instance of Maxim. Any attempts to take images result in the same 'remote server machine does not exist or is unavailable' error. In fact, even if you go to the ACP UI, and try to 'disconnect' the camera, you get the same error message in a popup window. That effectively means that once Maxim has terminated and restarts, ACP is stuck unable to communicate with it, and you can't even disconnect/reconnect from within the ACP UI to recover. That means that I then need to restart ACP as well. I'm not sure what logs would be useful to help troubleshoot this - but do you have any suggestions?

Thanks in advance!

Stewart.

Hi Bob

Been reading around the forum, and saw your response to a similar issue http://forums.dc3.com/showthread.php...-keeps-running

I now get why a timer probably wouldn't be that helpful, which leaves me troubleshooting and reporting to Maxim for problem (1) I think. Great explanation!

Would still appreciate any help you can give on the second issue where ACP doesn't notice that Maxim has been restarted though. Thanks!

Stewart -- Thank you so much for taking the time to look for the answer here. I'm happy that you found it. Your note prompted me to link to that post of mine. from a generic "why?" post in the Company and Product Info section. I was somewhat inspired I guess :-) :-)

As for why ACP doesn't know that MaxIm has been restarted... The component programs talk to each other through the COM service of the Windows OS (one of Microsoft's best innovations IMO, though it started out to be something else like so many great ideas). Once ACP opens a "channel" to MaxIm it has a channel to that copy of MaxIm (in Windows terms a process with a unique ID). If that copy of MaxIm goes away, there is no mechanism to pro-actively notify all of the other programs that may be using it. But the next time one of them (in this case ACP) tries to talk to it, an error is sent back to the caller "process is gone". Starting a new copy really can't set off a chain of events that would automatically connect everyone who was using the old copy to the new copy... what if the new copy is for a different purpose etc? Some programs that offer COM services can have multiple copies of themselves, each serving a separate client. Who would know whether the one just started is for those who lost the old one or for someone else interested in something a little bit different?

Again, I should point out that if MaxIm and ACP were somehow joined at the hip, or were all one program, the original underlying problem would have taken the whole thing out.

Hi Bob

Thanks for replying so quickly!

I guess I can understand that if ACP doesn't get a notification that Maxim has gone, then it can't do anything about that. However, if I then use the 'Camera/Disconnect' option in ACP, and that then gets the 'Process is gone'' notification from Windows, wouldn't it make sense to at least disconnect? The problem is that what happens right now is that ACP stays 'connected' to the COM instance that it (now) knows is broken - it's impossible to disconnect, because using the disconnect just gives the error message. Which in effect means you need to restart ACP (and by implication, everything else that is using it's objects).

It would be really nice if the disconnect camera option actually did disconnect when it gets the 'process gone' notification. After all, it's definitely disconnected :-). At least that way, you could 'connect' and have COM start a new instance of Maxim for ACP's use at that point, without having to restart ACP. Am I missing something obvious here? Probably :-)