Solid State Disk Issue (was "ACP Script error while...")

[rspostrspost]

Bob,
Over the past months, I have been running SN searches, but recently, I've been getting the error as in the log below, with the same error message each time it fails. I can continue by simply restarting the plan, but need to eliminate the route cause. I've pasted in the end of the log.
Running SkyX with Paramount ME, MaximDL, Win 7 x64, direct USB connection between ME and computer running AquireImages script on ACP. Plan is located in documents\Plans. Any suggestions on the cause?
Regards, Dick
05:45:12 (taking 30 sec. exposure, Clear filter, binning = 1)

05:45:12 (using Monochrome readout mode)

05:45:12 (starting exposure)

05:45:47 (can pre-slew to next target)

05:45:47 Start slew to NGC7454...

05:45:51 (exposure complete and image downloaded)

05:45:51 Calibrating image...

05:45:51 Image finished

05:45:53 Plate-solve final image

05:45:53 107 image stars found

05:45:54 72 catalog stars found

05:45:54 Solved! 30 stars matched.

05:45:54 Average residual is 0.24 arcsec.

05:45:54 Pointing error is 0.881 arcmin @ angle 29.45

05:45:54 True focal length is 296.2 cm.

05:45:54 True binned plate scales (arcsec/pix): H = 1.04 V = 1.04
05:45:54 True image center (J2000): 23h 00m 37.4s 12° 47' 53.12"

05:45:54 Imager sky position angle is 359.9 deg.

05:45:54 Image FWHM is 2.4 arcsec (2.26 pixels)

05:45:54 (avg FWHM = 2.31 arcsec)

05:45:54 ==== Starting target NGC7454 ====

05:45:54 (wait for slew to complete)

05:45:56 (slew complete)

05:45:58 (no slew, scope already within max error of target)

05:45:58 Ready for NGC7454 (# 191 of 306 in set 1)

05:45:58 **Script Error (Tracking has been stopped)**

05:45:59 Source: ACP Observatory Control Software

05:45:59 Message: at line 1230 column 36.

ACP console log closed 25-Aug-2012 05:46:00 UTC

[rspostrspost]

When I look at this error, it seemed to me that the problem is not likely software as it was running correctly for over a humdreded images before failure and it has only recently started. There have been no changes in hardware or software behond adding WatchDirectory to zip and ftp files. I ran the smiulator with a run plan which had failed and it ran to completion; I will run it often on cloudy nights. The simulation ran with SkyX, MaximDL, AMC Dome control. My best guess is the USB line caused the error. I switch to using the RS232 line as the connector is supperior to USB and less likely to have a corrosion problem as the seal is superior. The USB has been connected without being touched for serveral years.

In looking at the error, does it make sense that the error could be caused by a failure of the communication from the ME to the SkyX? Could there be a confict between the download of Maxim with the slew that could cause the error?
Dick

[Bob Denny]

Dick --

I got a note from Ajai this morning indicating that you're unhappy with the service here. I really apologize. I see that your first post was on the 31st. It's unacceptable that I didn't get back to you here, my only weak excuse is that the Labor Day weekend intervened and I lost the marker for your post which I had read but didn't know how to fix at that time (it did look like hardware!). Then your last post on Wednesday made me think that you are fighting a hardware problem. I'm writing you and Ajai and Tim a response via email.

[rspostrspost]

Bob, thanks for the call. Today, I tried to run a plan using SkyX with builds 4470, 5424, 5511, 5649 (the build I was running for 1000s of images before this problem started), 6019 and 6109. All failed after none to 2 images after slew.
Start slew to NGC5198...
(wait for slew to complete)
(slew complete)
Ready for NGC5198 (# 1 of 33 in set 1)
**Script Error (Tracking has been stopped)**
Source: ACP Observatory Control Software
Message: at line 1227 column 36.

I also tried Sky6 and got the same error. I'll go back to Bisque with this information, but at this point, the only piece of the puzzle which is remaining is the mount which can otherwise run 400 pt TPoint model automated and run 600 slews with AP4. Thanks for the help. I'm sure I'll be back on this one.
Dick

[Bob Denny]

Dick, we need to engage a bit more closely. At a minimum, I need to look at the special AcquireImages script you are using, and the complete log file, etc. Line 1227 in the standard script is a call to update the completion state data at the head of the observing plan. If you are using the standard script, then maybe the observing plan is write locked?? It could be that simple? I'm happy to help but I need to be able to sink my teeth into it a bit more, and for that I need info.

edit: we're on the phone with Remote Assistance doing daylight test.

[Bob Denny]

I see you're looking. I'll post more detail after I hear back on your tests.

[rspostrspost]

Bob, I've continued to run without error, so I conclude that you solve it. Please post your explaination. For ACP users, I would advise not using a SSD C:/ drive in their control computer.
Thanks for your heroic effort today.
Regards, Dick

[Bob Denny]

OK, thanks for the report. I'm in the office overnight tonight to try to catch the TheSky X slewing/flipping problem red-handed :-) The session lasted 5 hours. The physical problem is in the code that updated the plan state, the two lines of info that ACP writes back to the plan to make it "continuable" after being stopped or failing part way through. The logic is (in English)

1. Open the Plan file
2. Read it all into a variable
3. Close the plan file
4. Strip off any old plan state info in the first two lines
5. Create the two lines of plan state
6. Create a new file with the same name as the plan file, overwriting any old plan file
7. Write the new plan state info plus the stripped plan (the original plan)
8. Close the new plan file

It fails in step 6, a call to FileSystemObject.CreateTextFile(name, true). Furthermore it fails with no error information, not even an error code!! I did not have any protection on that statement because I had already read the plan file above so that means all of the directories needed are OK. An error here is "impossible".

The failures were random. It might fail on the first target, or after successfully updating the plan state 50 times. We went through several theories on what was causing the CreateTextFile() to fail, and we could never confirm them because the Windows FileSystemObject was not returning any error information at all, not even an error code!! Finally, we got to talking about the solid state disk (SSD) he had installed around the time the problem started happening. I thought of a possibility that Open-Read-Close-Create/Overwrite-Write-Close all on the same file path in rapid succession might be exacerbating a timing problem in the SSD. So I added a 1 second delay between the Open-Read-Close and the Create/Overwrite-Write-Close.

This is what he's testing now. It makes me squirm because it's a hated "software victory over hardware". The fact that FileSystemObject is raising an anonymous error makes me believe the error is actually happening deep in the SSD's driver or hardware, in a place even the FileSystemObject's developers never expected or would never defend against.