Flats: Constant illumination time for each filter/rotator set

[Gregory Nelson]

I am getting residual calibration artifacts in my light frames coming through from flats, due to the fact that each flat set for a filter and rotator position angle is taken to a target illumination ADU and the exposure time changes from one flat to the next to try to keep that target value. This leads to a set of flats with different exposure times and it is difficult to take that many flat darks to match to each flat frame. What I want is a set of flat frames that starts at target illumination ADU and then keeps exposure time constant for the rest of that set. I can then take a dark file that matches that exact exposure time.

What I need is strategy where Autoflat.vbs searches for the starting exposure time as it does now, but when the exposure is found that reaches target ADU, the exposure is kept constant for that set of flats. This means that the illumination will get lower for each flat at dusk or greater at dawn. But each flat will be calibrated exactly the right exposure dark and stacking will handle the illumination differences.

In the AutoFlatConfig.txt I know I have the ADUAcceleration_AM and ADUAcceleration_PM factors. If I set these to 1, I believe that the script will do constant exposure for a set of flats, but will the script properly search search for the right starting exposure time for a set?

If no, does this mean I need to edit the AutoFlat.vbs script to get the result I want?

Any insight you have for the path to solve this issue would be helpful.

[Bob Denny]

I work with very finicky science astronomers that do photometry, to include almost all of the stations on the AAVSO Net. I'm not an expert. However all of my science astronomer customers are happy with the quality of calibration frames they get using either sky-flats or panel flats. I think Eric Dose can probably expand on this. I believe the trick is to use bias frames together with the flats and darks to balance out the residual dark current changes across varying exposures. I'm unsure that you can even get enough flats to be useful by starting at the shortest exposure (dusk) without noise become a factor, and at dawn, starting with the longest exposure without saturation becoming a factor. I just don't know but this is the first such request I have gotten.

I hope Eric, or Dick Berg, or one of our other science astronomers can chime in here.

[Gregory Nelson]

Bob,

Lots of this traditional logic on flats and flat darks and constructing optimized darks from a master dark and a master bias scaled to exposure time. comes from years of experience from CCD cameras. My camera is CMOS and the fact that every pixel has an amplifier makes bias frames at very short exposures very nonlinear and very unrepeatable....an even worse problem. I have a couple of data sets acquired with the skyflat script with apparently excellent looking sky flats, that unfortunately these under correct and introduce artifacts from the sensor into my final image because I use an averaged dark that only matches the median exposure time of a set. And if I attempt to acquire Bias frames so I can created "optimized" darks, it leaves these chip patterns in my lights that come from CMOS Bias frames. Lots of recent info is out there written about CMOS chips and Bias frames and it's pretty unanimous that bias frames with optimized darks are not good.

For example, on my last image, the skyflats had a total variation in exposure time of 30-60% time from the first in a group of 7 to the last. While that is only 2-3 seconds for most flats (except the one taken in very dark skies, at t total exposure of 4 seconds this is a huge relative variation in the stacked flat, given that I only subtract one average dark frame close to the median exposure time. and given a flat is used multiplicatively to calibrate the light frame, this residual dark current which carries chip pattern noise amplifies the effect in the calibrated light frame. And I see that in every target. One "easy" solution is to do panel flats, but my observatory is not geometrically configured well for panel flats and good luck finding an illuminated panel that big for my CDK20.

A significant way to reduce that noise in sky flats is to have a dark for each flat that matches it's exposure time. Hence, the need to expose for a fixed time once the correct exposure range is found for a flat (almost like treating a sky flat like a panel flat). Over the time of acquiring a small set of flats for a filter, the sky variation will introduce much less noise to the flat and importantly no pattern noise from a mismatched dark. And even though the flats will have differing illumination, this is normalized out in the median stacking equation and carries no pattern noise. Bottom line: lower pattern noise to the final image if the dark and flat are exactly the same exposure time. The Old CCD Chips had very few patterns except the occasional linear defect. The New CMOS chips have extensive patterns that look like checkerboards.

I've acquired many sets of ACP sky flats in groups of 7. Once a filter/rotator position starts acquiring, the time increase/decrease from the first to the last flat in the set is 30-50%. I am sure that of I started acquiring a group of flats at 50% ADU, the final flat in that group would be 25% ADU or greater. Hence the question whether the logic to find the starting ADU for a group is coupled to the logic to change the exposure time. For instance, if I set the ADUAcceleration_AM and ADUAcceleration_PM factors to1.0, what happens?

I can show you more of the effect of what's happening. For now I am looking generate a test set of flat data directly in Maxim to prove out my observations. But ultimately I'll need a solution because I can't process any images due to the flats being off for my equipment combination.

[Gregory Nelson]

OK,

I "tricked" the script into acquiring the flats I need with about75% yield, 25% I'll have to throw away. This is good enough for now and I see how I can optimize my "trick" to get a higher yield, albeit at the price of more time neede to get all the sky flats data.

I'm not conversant in the VBS scripting so I can't dive in for a permanent fix and I'm not asking for one. My trick was to set the ADUAcceleration constants to 1.0 and make the tolerance wider. This setup ran through the algorithm to produce the fewest changes in exposure time during a flat series (filter/rotator combination). If the average ADU dropped below minimum tolerance midway in a series, the exposure was changed. I suspect that logic is separate from the adjustments driven by the ADUAcceleration factor. In any event, with the right combination of target ADU and tolerance, I can maximize yield in a series of number of frames with the same exposure, which is what I need. I'll share some before and after pictures once I get the needed data sets to show you what happens for these CMOS sensors and how flats acquisition strategy can eliminate that issue.

[Bob Denny]

I lied. Chris Elledge also asked about fixed exposure flats. I gave him the same info. I suspect this is a recipe out of the past or driven by some limitation of software (not ACP ha ha). Maybe I am wrong as I said I am no expert! I await Eric Dose and/or Dick Berg and/or ???. If needed I will reach out to Arne Henden, former USNO, former Director of AAVSO , and designer of the APASS Project (which is being done with ACP Expert).

[Eric Dose]

There is very little doubt that for non-rotated flats and flat corrections centered on good light panels, the current math works very, very well indeed. No changes needed that I'm aware of. There has to be some accommodation for minute-to-minute variations in light panel brightness, which are real even for high-end panels. I'm getting vignetting corrections good to 1-2 millimagnitudes (0.1-0.2 % flux), every night for years, without even trying very hard. To hold constant the exposure time per session, per filter, giving up the constant flux advantage, would make for a change I'd need to see tested. As an new option in ACP Preferences it might be fine.

I wouldn't know whether this extends to rotated flats and flat corrections; to figure that out I would have to know the excruciating details of how the flats are taken, combined, and applied. This complication is one of a dozen reasons why I've never gone near a rotator.

Arne might have some insight on flats through rotators, though I don't know whether he's ever used one, either. The APASS project doesn't, and I'm pretty sure AAVSOnet doesn't either, at least not up to a year or two ago.

[Gregory Nelson]

Eric,

The issue I'm working to correct isn't connected to the rotator (although that does make things a bit more complicated), its the CMOS chip pattern noise that gets imprinted on flats if the exposure times for flats and flat darks are not matched. The time variation introduced by the skyflat routine make me need a different dark file for every flat frame. I would rather have all flat frames at same exposure time (matched exactly to a dark) and allow the stacking to handle the variance in lumination in each frame from the trilight sky.

I completed an experiment this evening and "tricked" the Sky flats routine into getting me a data set I need with a few frames I'll have to throw away. As I work up some examples I'll share in case anyone else acquires a CMOS sensor bringing these problems here.

This really isn't an ACP problem, its a new issue arising from very sensitive CMOS sensors with pattern noise.

[Eric Dose]

I wouldn't know about CMOS. Maybe there does need to be a constant flat-time option in ACP. But it would have to be an option, not a behavior change.

Arne and Gary Walker have done some detailed noise evaluations in high-end CMOS cameras. That's where I would start. They may already have figured this out.

If it's critical for you right now, you can always take flats in MaxIm directly. It's pretty fast, and most people don't take flats every night.

[Bob Denny]

Eric -- Thank you so much!

Greg -- Eric's right. Arne Henden and Gary Walker are the people to start with. Both are pioneers with CMOS and high-precision photometry. Gary is the source of the initial need for the SubexposureInterval addition to the ASCOM Camera standard (initially in 2018) working with the FLI Kepler imagers. Do you know these people? If not I can make introductions.

[Gregory Nelson]

Bob,

I don't know them probably because I haven't been in precision Photometry circles. But I'm always willing to learn from an expert!

I'll have a data set tomorrow that for a few filters I was able to get constant exposure time flats with all frames within target illumination by +-15% (see my notes above on how I "tricked" the script into getting these). This should allow me to calibrate the flats correctly and minimize the residual pattern noise that the flats are carrying into my light frames. I'll let you know how that works out.