A new problem with scheduler and dome opening

[Dick Berg]

HI Bob,

I'm not sure what's going on here yet, but I just wanted to say something in advance of your releasing 8.2.2, in case there's a bug.

For the last two nights, the initial startup of the observatory has failed with a DOME FAILURE. I've attached the two logs. Each night was the same issue. Everything started up as it's supposed to, but the dome remained closed after startup. See attached Dome log file for tonight. In that log there's no report of the dome opening at the time of the dome failure. In the ACP Control Panel, the DOME annunciator is flashing grey, and the Scheduler reports Failure in the System Status, with the Dispatcher Running box unchecked. To reset and get going, I quit Scheduler, click on the "Close" box in the dome control panel to forcibly reset ACP, and then start Scheduler again. At this point, StartupObs runs successfully to completion, opening the dome. After this restart, all is well for the whole night, with weather events opening and closing the dome as they should.

This is a new problem since I installed the trial 8.2.2 scheduler.exe file. I didn't report this last night, thinking it was a one-off glitch. But since it happened again tonight, I thought you should be aware.

The log 20190721@010945 is the log for the night of the 20th. The other log 20190721@172045 is from tonight just now.

[Bob Denny]

I don't know how to interpret your driver log, but what I can say is that I am seeing ShuttterStatus = ShutterError. This cannot have been caused by 8.2.2 changes. I am reluctant to post internal code as it is a magnet for "discussion", but this is definitive. It works, and has zillions of miles on it. This is a property DomeOpen, returning true/false if the dome is open or not, and trapping on weirdness as well:

Snap2.jpg

[Dick Berg]

Bob, My bad on the dome log - it was the wrong one.

I understand your reply. Of course there's no issue with the status reporting, and for some reason my driver reported "error." This was the same problem months ago that I/we got through. The dome and dome driver has been working flawlessly ever since.

Here's a combined running log of last night startup, which failed (shutter error). I thought you (others) might be interested to see it. The log incorporates the scheduler log, dome log, and startupobs log, all ordered in time, along with my running commentary. I prepared this this way mainly to clear my own head about the order of events. I know you can't solve this problem for me, and I believe you when you say the 8.2.2 changes don't affect this code. So I have a problem of my own to deal with.

I guess it is just coincidental that this error is reappearing now at the time I plugged in the test scheduler.exe 8.2.2. You can close this thread - I will poke around and let you know what I find.

Thanks for looking at this.

[Bob Denny]

Ok I hope you can find it.

[Dick Berg]

Bob,

Me too!

Same error tonight at startup, but in bad weather; it disables Scheduler. Have to quit Scheduler, restart, and then it works fine. Still raining though, so I don't expect to get any farther tonight. It's a puzzle.

[Dick Berg]

Bob,

My most humble apologies. &

You know, it pays to keep your eyes open once in a while when running this software, unless we become accustomed to trusting the automation is working the way we think it always works.

As a first test, I decided to turn off all the software - weather, scheduler, etc..., turn off all the hardware but the dome and telescope, and just run ACP's control panel to connect the telescope and, as it does in turn, the dome. Got the dome error right at the git-go. There, in front of my eyes, is the DOME annunciator flashing and the dome control panel reporting "shuttererror." The problem I've been having is clearly not an issue with scheduler.exe 8.2.7, as I might have thought .

So unless you made some error in the latest ACP maintenance release "Version 8.2 (build 9, V8.2.1 Maint Rel)" - highly unlikely - I'm going with the notion that I monkeyed around in the dome driver about the same time I installed this release (July 15), and created some peculiar condition that fails on connecting the dome. When I use the ACP Dome Control Panel's "close" button to force-close the already closed dome, this resets the shutter status to "closed" and restores the functionality without further problems.

I have all the previous versions of the dome driver. I will back-rev one at a time until I get to the one that works, then compare it with the next later version to see what I did wrong. It's probably a misplaced semicolon or something else stupid.

I'll post one more note when I've found the problem, just to let everyone know.

[Dick Berg]

Nothing wrong with scheduler.exe and nothing wrong with the dome driver either! (Oh me of little faith!)

None of the back revs worked, and that was strange. I fixed this problem by rebooting the observatory computer. Drat - what a lot of wasted gray cells and lost sleep getting to this conclusion.

This is actually probably the worst of many outcomes simply because there's no way of knowing why something somewhere in the OS glitched. Could be that I installed the latest ACP/Scheduler updates successfully but didn't reboot the computer at the time and should have. Could have been a cosmic ray - my favorite villain for the unexplained. Could have been something hardware or software that I turned off or on at the wrong time.

Back on the beam now. I just can't wait for the next update.

[Bob Denny]

OK, thanks for letting us know! Yeah gremlins are the worst! One ting, the computer ought to be rebooted about once a week to prevent the SystemTicks timer from rolling over....