Exposure not completed in Dhyana 400BSI camera

[Takanori Sakamoto]

Hi Bob,

We changed to a new CMOS camera Dhyana 400BSI (http://www.tucsen.com/Home/Wiki/index/dataid/29.html) recently. Since then, we are having the issue that the exposure of the camera is not completed (downloading of the image to the PC has not been started) and ACP is stucked after the message saying "(taking 10 sec, exposure, R filter, binning = 1)". This issue doesn't happen everytime. As shown in the Sceduler log file below, it goes without the issue for several exposures. According to the company of the camera (Tucsen), the issue doesn't happen running the camera in MaxIm DL (I am also sort of comfirming this by running the camera in MaxIM DL by exposing in the continuous mode). The camera is connected to MaxIM DL via ASCOM driver provided by the company.

I believe the issue is happening due to the communication between the ACP and the ASCOM driver of the camera. I know that the ASCOM driver of the camera is not well written. For example, 1. "cooler on" information is not displayed at the information of the setup tab of the MaxIM DL camera control window, 2. progress of the camera exposure is not shown as a bar at the exposure tab of the MaxIM DL camera control window, and much more. As can be seen in the attached log,

16:09:57 (taking 10 sec. exposure, Clear filter, binning = 1)
16:10:10 (starting exposure)
16:10:12 (exposure complete)
16:10:14 (exposure complete and image downloaded)
16:10:16 Image finished

the actual time between "(exposure complete)" and "(starting exposure)" is 2 seconds and not 10 seconds which is the actual exposure time of the camera. I am guessing the handshake between the ACP and the camera driver is not going well due to the badly design of the ASCOM driver.

According to my several trials, this issue happens within 2 hours after starting of the automatic observation by the ACP Scheduler. Is there any good approach to understand the fundamental cause of the issue? Since the company doesn't know/use the ACP, they don't have a way to reproduce the issue. It would be great if you can help debugging the issue, so that I can tell the company to fix the provided ASCOM driver.

GRB-1-20220211@160905.log

[Bob Denny]

Hello —

Let’s start by having you disable slew during image download (ACP Preferences, General tab). This causes ACP to avoid using the MaxIm two-phase image downloading feature. If the problem ”goes away” then we can focus on the ASCOM features that are specific to the two-phase downloading.

[Takanori Sakamoto]

Hi Bob,

Sorry for taking so long to test your suggestion and report the outcome.

We disableed slew during image download and ran the ACP Scheduler. With disabling this feature, we now see that the ACP is waiting for the camera exposure to start. When we hit the start botton of the MaxIM DL manually, ACP starts to process the rest of the tasks. This doesn't happen all the times during the observation. But this happens several times during the night. Do you have any clue how to debug the issue?

Regards,
Taka Sakamoto

[Bob Denny]

Hi Taka — This will be some sort of timing issue within the camera or its driver. With the overlapping turned off, ACP always calls one function, Expose() and then waits for imageReady. If Expose() returns then by the interfacing rules, the camera has said “I have successfully started the exposure”. If it cannot do so for any reason, it must raise an exception. ACP is prepared for this and will handle it with a plan failure. What you are describing is that the camera accepts the Expose() call but does not do it and does not raise an error, leaving ImageReady False “forever”. If the camera were to start the exposure then fail midway, then it must raise an error on reading ImageReady. In any case a “silent failure” is always a bug.

[Takanori Sakamoto]

Hi Bob,

Thanks for your reply. I will contact the vendor of the camera and see what they can do for us.

Regards,
Taka Sakamoto

[Bob Denny]

Thank you. Please let me know the outcome.

[Takanori Sakamoto]

Hi Bob,

We collected the logs using the software requested by the vender and captured the PC screen captured video at the time of the incident. We provided them to the vender. Here is the reply from the vender.

ACP issues the expsoure start command to MaxIm DL firstly and MaxIm DL calls our camera's SDK through the ASCOM driver,
then SDK controls the camera to exposure. The flow chart is like this.


When the ACP stuck, you need to manually push the start button of MaxIm DL, and then ASCOM Driver, TUCAM SDK, finally Camera works.
That means they all work normally from MaxIm DL to Camera.


So the ACP stuck has two possible reason:
1、ACP didn't issue the exposure start command actually at that time;
2、ACP did issue the command but MaxIm DL didn‘t respond.


However, about these two possible reason, we currently has no way to verify, it's beyond our capabilities.
Looking forward to your comment.

Catch.jpg

So, basically, they don't have any clue what is going wrong (we spent two months communicating with them and zero outcome).
Do you have any suggestion what I should ask to the vender?

Just in case you are interested, below is the captured movie when we reproduced the issue.
https://drive.google.com/file/d/1TDc...ew?usp=sharing

Regards,
Taka

[Bob Denny]

Hello Taka -- The video was a good idea. I assume the DebugView window is for the camera. What I see in ACP is that it calls MaxIm to start the exposure (Camera.Expose()) but that function never returns. Camera.Expose() is supposed to return immediately with the exposure in progress. At that time ACP will print "(starting exposure)". This happens immediately after the return from Camera.Expose(). When it gets stuck, that "(starting exposure)" message never prints in ACP so it is certain that the call never returned from MaxIm.

These images are from your video at 01:13

Snap3.png

I don't know the insides of MaxIm but I will guess that Camera.Expose() results in calling the camera to take an image. In fact, if the debug log is from inside the camera then it was called for the 10 second image:

Snap1.png

However, the camera appears to be instantly reporting that the image is ready. Lines 299 and 600 are both for the same instant of time (+/- one second). This is an "impossible" condition. I am going to guess that MaxIm is not expecting the image to be ready the instant that the camera returns from starting the image, and that MaxIm getting stuck somehow. The instant image ready looks like the first clue to me. The ImageReady() = True should not happen until at least 10 seconds have passed, or so it would seem.

I hope this helps.

[Bob Denny]

I got an email message on our ‘info’ address from Dhyana. Here’s where we are. I’ll let Colin know about this thread so he can see the analysis so far.

---

Hello Derek -

I assume this is for our mutual Japanese customer Taka Sakamoto. Thanks for contacting me. I can’t explain the problems. Our software uses many of the “scripting” features of MaxIm. We call CCDCamera.Expose(), check CCDCamera.ImageReady, and more. I am copying Colin Haig of Diffraction in case he has thoughts, or is possibly familiar with your camera and if it uses a “native” driver that needs to be programmed specifically for MaxIm.

— Bob

On Aug 5, 2022, at 05:25, Stephanie Denny <stephanie@dc3.com> wrote

Begin forwarded message:

From: 王瑾 <derekwang@tucsen.com>
Date: August 5, 2022 at 1:44:07 AM MST
To: info <info@dc3.com>
Subject: Problem

Hi Sir/Madam,

This is Derek from Tucsen sCMOS camera company in China.

We have a client in Japan who use our Dhyana 400BSI sCMOS camera for his observatory. During his usage, he use ACP to contro Maxim DL to control our camera. For your information, our camera support MDL via our ASCOM plug-in.

Now the situation is, it is no problem for our camera to run on MaxIm DL alone. But if running both ACP and MaxIM DL, sometime the automatic operation will pause, and the client has to continue via manual operation.

Hereby pls find the vedio: https://drive.263.net/link/kbHbdTvFQBJi8cP/

From our side, we have tested the camera data, every time, no matter it is automatic start or manual start, our camera will immediately response, and now the question is how does the ACP control signal to MDL to switch between starts and stop?

Thanks for your support.

[Colin Haig]

Hello Taka,
My name is Colin Haig, and I work for Diffraction Limited, makers of MaxIm DL Pro.
Bob and I will discuss this today.

Some suggestions:
1. Turn off Windows USB Selective Suspend and prevent Device Manager - device Power Management from turning off the devices to save power.
Details here on the Diffraction support forum:
https://forum.diffractionlimited.com...-suspend.7848/

2. In MaxIm Camera Control Window, Setup Tab, try turning off the Advanced Threading options.
threading_off.png

3. If the problem happens again, we will need the MaxIm log from the same date as the ACP logs.
In MaxIm, View... Log Window.
Right-click in the Log Window.
Turn on Show Timestamps
Right-click in the Log Window again.
Open Log Folder.
Send us the log file that corresponds to the date when the problem happened.
There may be several logs with filenames like 20220808.log