ObservaDone Causes Fatal Errors

[Mark Williams]

Hi,

I've been working through several issues relating to migrating to a new telescope control system and new Win7 computer. I've had a couple of issues with ACP and Scheduler in the process:

1.) I've been getting "out of memory" errors while acquiring images. As per other posts, I disabled custom file naming and the problem improved, although I still had a couple of of runs crash with windows popup saying "out of memory" and a console message:

05:16:37 Image finished
05:16:38 ** ACP Auto-Calibration overrides ImageSet calibration flags.
05:16:40 Plate-solve IX Ori-S001-Johnson V-R001-Johnson V_dupe-2 final image.
05:17:42 **Script Error (Tracking has been stopped)**
05:17:43 Source: PinPoint.Plate
05:17:43 Message: No FITS image file is attached. at line 1304 column 21.
ACP console log closed 03-Jan-2014 05:17:43 UTC


I've attached the full run log. These crashes seem to be be associated with plate solving, although plate solving is working well otherwise. I've increased the virtual memory and heap memory on the computer just see if that made a difference.....it did not seem to.

BTW, on the subject of custom file naming....any hint on how to avoid the trouble with "out of memory" errors thrown by AcquireImages? I wish I understood what was happening.....

2.) Went to bed last night at midnight, leaving the scope running with no more targets until it was time for dawn flats. When I got up to watch the dawn flats, Scheduler was dead. The Scheduler log attached seems to indicate that ACP.BoltwoodII server was not responding to a Weather.Available call. This seems to have occurred intermittently through the night and finally triggered a crash. The Cloud Sensor II works fine with the ACP.BoltwoodII server, although it seems to take a long time when I connect in to ACP for the data to be available (say 15 seconds). I also attach the full log.

Any ideas?

Thanks,

Mark

[Mark Williams]

Hi again,

More testing tonight and a couple of more crashes.

1.) The dusk flats failed after taking 1/5 images, with a message " Probably an error in the flat plan at line 822 column 17." The log is attached. The plan ran perfectly at dawn, but this is the second night in a row it has failed similarly at dusk.

2.) I got another out of memory error when starting a run after a long inactive pause in Scheduler. The Scheduler log is attached.

3.) I recovered by closing and restarting ACP, Scheduler, FM, Clarity. Thing then ran smoothly for two targets and then the third target failed with a cryptic console message "NO DATABASE Out of memory." I 've attached the Scheduler log, but I could not find a run log for the target that failed. Scheduler kept running so I resubmitted the plan with identical results. I let Scheduler run until the next target, but it failed too.

Can anyone make sense of this?

Thanks!

Mark Williams

[Bob Denny]

Wow you have some serious issues with that system. I don't think I've ever seen "out of memory" errors, nor the even crazier one "No more threads can be created in the system." The problems appear in Scheduler's log at times where it is asking for weather from ACP, and that causes ACP to ask for weather from Boltwood/Clarity. It is normal for Clarity to take a long time to start up. The "No more threads can be created in the system." may be coming from Clarity, or it may be a "red herring" (false message) caused by some other error relating to that out of memory. Can you capture a picture of that "out of memory" error? Or at least tell me what is in the title bar if it is a popup? That would help identify the source.

Again, I think it's time to look at your system live. We would not necessarily need dark/clear skies. I'll be available starting this coming Thursday night.

[Bob Denny]

What's going on with the UserAction "Wait for dome slew to complete" before every image in the flat acquisition process? Is there something special with your dome? Could the UserAction be a source of problems (well, it could, but I'm asking if yours could maybe be causing some of this)?

[Mark Williams]

Bob,

As always, thanks for the prompt response......

Next Thursday is fine. I'll be back at the observatory then anyway. Name a time. In the meantime, I'll try a few things like operating with the dome and weather simulators to see if I can narrow the issues. And I'll try just ACP with and without Scheduler involved. I'll see if I can capture the popup messages, although to be honest they have been random and incomprehensible (to me).

Would it be worth trying to reinstall ACP and/or Scheduler over the existing installation?

On the dome....I had some problems with the dome chasing the scope and imaging starting before the dome caught up, so getting more that a few pictures of the inside of the dome--so I slowed things down by waiting after each slew. It was working perfectly OK for the last couple of years with the old computer and RCOS drive. So I don't think it's the problem. But don't get me started on the dome driver issue.....it a saga!

On the rotator, the RCOS PIR is installed with their TIM controller. But with the square KAF 16803 CCD, framing isn't much of an issue, and I use it just to orient to 0 PA after I've had the camera off....a chore that takes about 30 seconds. That said, I have integrating the rotator on my to-do list.

Thanks again and happy new year!

Mark

[Bob Denny]

Well, I just scheduled an evening with another customer for Thursday night. Dang it!!! I saw his before yours. This is the week for "stump the chump" tech support issues, and wouldn't you know, I have three days in the field coming up.

The dome should not be an issue. If it is chasing, then ACP is not managing it! If it is a Digital DomeWorks HomeDOme or ProDome, and if you are using the ancient TI Digital DomeWorks Control Program (DDWCP) and its supposed "ACP support", that would explain your pain. That stuff was from 14 years ago and was for ACP 1, and it never worked anyway. Just guessing here. Try me on the dome. If DDW, it's easy. Get the latest driver from TI and install it. Configure it from ACP. Remove the hack in the UserActions and go. Problems solved. Same for MaxDome, ACE, M1OAsys, Foster.

Try this: During the day, turn on Simulated Images and turn off autoguiding in ACP. Then you can play with it, do runs, etc with exposues of 1200 seconds which will really be 120 seconds. You can image some random part of the sky as long as the RA is within a couple of hours of the Local Sidereal Time, right?

Try disabling the dome altogether first (you don't need sky or weather), the reenable it and try substituting dome simulator for the dome. Keep working with it till you can pin down the culprit. Maybe we can solve this without an online session. I'll try!!

[Mark Williams]

Bob,

I'll be available Friday and the rest of the weekend, if that works. I can also do daytime as well if we keep the mirror covers or dome closed! In the meantime, I'll try the suggestions.

On the dome, it's an 4m Observadome that doesn't have a OEM ASCOM Driver. Dan Azari did reverse engineered a wrap-around ASCOM driver for me in 2008 that works ok except that dome slew doesn't start until telescope slew stops and Acquire.Images sometimes starts imaging before the dome stops. I have the source code and I've made a few minor bug fixes over the years as I've found them. Suggestions are most welcome!

Let me know what day/time is good.

Mark

[Bob Denny]

Groan: I found this via a Google search ha ha

ACP Locks Up with Observadome Driver

[Mark Williams]

Yes....this problem was finally "solved" by changing some timing loops in the dome driver. But I've never been convinced the Observadome driver handles the communication between dome hardware and ACP entirely robustly. Mark

[Bob Denny]

I would love to close this one

ACP/Scheduler Crashes

As yet another effect of the ObservaDome dome control system. It seems like they charge a hell of a lot of money for their domes and they should make it possible to integrate with the software packages that use 13 year old integration standards that are in very wide use!! If their customers don't turn the screws on them, they won't do it.