Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Hybrid View

  1. #1

    Default Automatic resubmission of failed plans

    How do I get the scheduler to automatically resubmit failed plans? The Scheduler Concepts help doc states:

    "A failed Plan can be automatically rescheduled to run again in the same or another night."

    I see the "resubmit on weather failure" option in the configuration, but I don't find any other options.

    I have a project with ~10 plans to gather exposures on a target. They all failed in rapid succession because passing clouds caused a pointing solution error. This was the only target available at the time, and the scheduler made no further attempt to try those failed plans. It also didn't try again the next night until I manually did a "resubmit failed plans."

    -Russ

  2. #2
    Join Date
    Nov 2005
    Location
    Virgil, NY
    Posts
    5,153

    Default

    Russ,

    Resubmit on weather failure works. Why it didn't for you could be possibly explained by the plans or observations failing due to other constraints. You'll want to check that out. The scheduler log file can help explain. Get more detail in the log by switching the Log Level on the Scheduler panel to Debug, but make sure in the Configuration you set the Wake Interval to something bigger than zero, like 30 (seconds), or else you will get tens of thousands of lines in the log that will be useless and hard to search through. Debug mode will explain if Plans or Observations fail constraints, which ones and why.

    Regarding "another night," when building a Project or Plan using the Astro-Imaging or Single Series web forms, set the Monitor Interval in the Timing section to "1". Even though a plan has failed on the original night, on the next night the plan will be resubmitted as if you had done it manually. Don't worry that the thumbs-down icon is shown in the Schedule Browser - it will be resubmitted. Put a date in the Must Run Before field so your Plan won't run every night forever!

    monitor.jpg

    If you already have a Plan in place, use the Scheduler Browser to edit the Plan portion, where you can change the Monitor interval to "1". You can even put in a Must Run Before date to tell Scheduler when to stop daily resets - follow the example date format below exactly - and it's different from the one above. Here's what it looks like:

    monitor003.jpg

    Note: you may not see "Best Efforts" in your Scheduler Browser Plan display - in my example here I've set up several consecutive Observations (not shown) that are linked and provide a way to make sure some parts of the Plan will be successful until a failure due to a constraint (like the horizon).
    Dick
    www.VirgilObservatory.us
    Pier-mounted Meade 12-inch SCT "classic"
    w. focal reducer to f/5.3 ~ FL 1630mm
    Optec TCF-S focuser
    SBIG CFW-8A and ST7-XME
    FOV ~ 15' x 10'
    H-alpha, BVRI, RGB & Clear filters
    MaxIm and, of course, ACP!
    AAVSO Code: BRIC

  3. #3

    Default

    Hi Dick,

    Thanks for the detailed reply... I'll study all of these.

    But my failure wasn't due to a "weather failure," in ACP parlance, as I understand it. It failed due to not being able to find a pointing solution. Yes, it happened because of clouds, but I think "weather failure" means that weather became unsafe. That didn't happen in this case.

    Also, how do I get a failed plan to be automatically resubmitted *on the same night*, as the help file says is possible?

    Thanks,

    Russ

  4. #4
    Join Date
    Nov 2005
    Location
    Virgil, NY
    Posts
    5,153

    Default

    Russ,

    I guess I misunderstood your comment about weather. I think "automatically resubmitted on the same night" only is true for true weather events. Your weather event was not enough to close the observatory.

    If you're watching online, and catch the failures first-hand, you can use the Schedule Browser itself to resubmit them right away.

    monitor004.jpg

    Resubmit All Plans essentially starts the whole Plan over. If there are constraints in place, some of those resubmitted plans may not run at a later time.

    About the failed plate solve, usually Scheduler will go on to do imaging anyway. There is a setting in ACP/Preferences/PinPoint&AllSky tab that can skip targets if the pointing update fails - if that's checked you're done.
    Dick
    www.VirgilObservatory.us
    Pier-mounted Meade 12-inch SCT "classic"
    w. focal reducer to f/5.3 ~ FL 1630mm
    Optec TCF-S focuser
    SBIG CFW-8A and ST7-XME
    FOV ~ 15' x 10'
    H-alpha, BVRI, RGB & Clear filters
    MaxIm and, of course, ACP!
    AAVSO Code: BRIC

  5. #5
    Join Date
    Oct 2005
    Location
    Mesa, AZ
    Posts
    28,774

    Default

    Dick, as always thanks for your help.

    Russ, "weather failure" is a Plan that is failed due to one of its Observations encountering an unsafe weather condition and terminating before it's completed. In that case the Plan will be rescheduled for the next night. Other failures, including "passing clouds" causing guiding or other failures, will not get automatically rescheduled. The Scheduler is not that smart. In the beginning I allowed auto-reschedule of any failure, but it quickly became evident that it was a bad idea. Most non-weather failures are equipment/USB/etc. failures and in that case the scheduler would fail Plan after Plan, rescheduling and failing again and again. Just last June, we had a situation where an equipment failure caused many Plans to be failed. After all these years, the idea came up of having the Scheduler just stop after "too many" failed Plans in a row. That one is on the queue.

    SCHEDULER-1583 - Operator Intervention on "too many" ACP Observation Errors

    The "passing clouds" problem is one that's difficult to address rationally. One customer created a constraint that was supposed to force the scheduler to shoot through "sucker holes" using a 10.6 micron IR all-sky camera and zonal analysis. It looked good on paper but didn't turn out to significantly improve productivity on nights with scattered to broken clouds.

    With all that said, what caused your Plans to fail? You can see the ACP run log from the web schedule browser spreadsheet. Find your Project, within that the Plan that failed and see its list of Observations, open the Observation that failed and caused the Plan to fail. and you should see a link to the log. Read the log to see what specifically caused the Observation (and thus the Plan) to fail.
    -- Bob

  6. #6

    Default

    Hi Bob,

    Thanks for the reply. Yep, understood re weather failures. As previously mentioned, the help file states, "A failed Plan can be automatically rescheduled to run again in the same or another night."

    Exactly what settings need to be set to accomplish automatic resubmission on the same night? I find only two that seem to be related, and I have them both enabled:

    - Scheduler -> Configure -> Dispatcher -> Resubmit on Weather Failure = true
    - Main Scheduler Page -> Prefer failed plans = checked

    Re my particular failed plans, I'll refer to my original post. These plans failed due to failed pointing solutions, due to passing clouds. I have about 10 duplicate plans with the same target and observations. Scheduler tried them all in rapid succession, and they all failed for the same reason. I see no downside to Scheduler trying failed plans again on the same night, particularly if it has nothing else to do (which was true in this case... it just went into an idle state with no more jobs to do). So again I need to know what settings to set to accomplish automatic resubmission of failed plans on the same night.

    Thanks,

    Russ

  7. #7
    Join Date
    Oct 2005
    Location
    Mesa, AZ
    Posts
    28,774

    Default

    Thanks for the reply. Yep, understood re weather failures. As previously mentioned, the help file states, "A failed Plan can be automatically rescheduled to run again in the same or another night."
    Can you point me to that? I think it is a holdover from the very old days before it became apparent that this was a bad idea.

    Please submit a couple of ACP Run Logs that are from those failed plans. I think you are using the web UI? If so in the Schedule Browser table list, drill down to the Observation that failed and see if you see a link leading to the log. If you are not logging in with a username/password ("Use Web Browser" button, user Observatory Operator) then these logs are in My Documents\ACP Astronomy\Logs\Scheduler\ddddmmyy\...
    -- Bob

  8. #8

    Default

    Scheduler -> Help -> Contents -> Detailed Description -> Scheduling Concepts -> description of "Plan"

    I'm trying to be a good new user by doing as much RTFM (read the fine manual) as possible... it's problematic when the manual is wrong.

    Not being able to automatically try failed plans again on the same night might be show-stopper for me. I see zero downside to the scheduler having another go at a failed plan if it has nothing else to do.

    Here's a log from one of the failed observations:

    ACP console log opened 09-Aug-2020 03:16:19 UTC
    This is ACP version 8.3 (build 1, Release 8.3)
    Unlicensed copy - 30 day demo mode
    Licensed to
    03:16:19 This is AcquireScheduler V8.2.8
    03:16:19 Custom image file path/names are in use
    03:16:19 [PreFlip=-1800 PostFlip=120 AF=180 PU=60]
    03:16:19 Hardware GEM pier side reporting is available.
    03:16:20 Initializing AcquireSupport V8.3.1
    03:16:20 Weather safe, server is Boltwood Cloud Sensor II
    03:16:20 Telescope is ACP->Takahashi FSQ-106 EDX4, driver V3
    03:16:21 MaxIm DL is version 6.22
    03:16:21 Imager is SBIG Universal
    03:16:21 Using focus offsets and star mags from FilterInfo.txt
    03:16:21 Selecting filter L from previous ACP run at 20200809@031258 UTC
    03:16:21 Imager readout modes:
    03:16:21 0 is Raw
    03:16:21 1 is Raw w/RBI pre-flash
    03:16:21 Guider is DL Imaging (rotated off-axis)
    03:16:21 Guider plate scale is 1.9 arcsec/pix. Guided dither with main imager pixels
    03:16:21 Max unguided exposure 30 sec.
    03:16:21 and exposure agregation is disabled
    03:16:21 (assuring that FocusMax is running now...)
    03:16:26 Calculated unbinned plate scales (arcsec/pix): H = 3.50 V = 3.50
    03:16:26 Calculated field of view (arcmin): H = 238.8 V = 238.8
    03:16:26 Plate solving with GSC 1.1 (ok for wider fields)
    03:16:26 Mount supports tracking rate offsets
    03:16:26 Imager rotator support enabled
    03:16:26 ACPS Observation Blue Horsehead RGB Moon Test(1 of 1)
    03:16:26 (belongs to Project Blue Horsehead, Plan Blue Horsehead Moon Test)
    03:16:26 (turning on sidereal tracking)
    03:16:26 Pointing updates are enabled
    03:16:26 Start slew to Blue Horsehead RGB Moon Test...
    03:16:28 Start rotate to PA 247.0 deg (raw rotator angle 337.6)
    03:16:28 (wait for slew to complete)
    03:16:57 (slew complete)
    03:16:57 (wait for rotator)
    03:19:52 (rotation complete)
    03:19:52 [flip check: Tn=-1440s HAc=4361s GW=T HAz=2849s DWz=T WF=no]
    03:19:52 Doing auto-center...
    03:19:52 (taking 2 sec. exposure, L filter, binning = 1)
    03:19:52 (using Raw readout mode)
    03:19:52 (starting [RBI] + exposure)
    03:19:55 (exposure complete)
    03:20:00 (exposure complete and image downloaded)
    03:20:01 Image finished
    03:20:01 Plate-solve pointing image.
    03:20:01 63 image stars found
    03:20:01 275 catalog stars found
    03:20:03 ** No matching stars found. Check your estimated center-point RA/Dec, and your image scaling and quality.
    03:20:04 Start slew to offset...
    03:20:05 (wait for slew to complete)
    03:20:13 (slew complete)
    03:20:13 (taking 2 sec. exposure, L filter, binning = 1)
    03:20:13 (using Raw readout mode)
    03:20:13 (starting [RBI] + exposure)
    03:20:17 (exposure complete)
    03:20:21 (exposure complete and image downloaded)
    03:20:23 Image finished
    03:20:23 Plate-solve pointing image.
    03:20:23 58 image stars found
    03:20:23 399 catalog stars found
    03:20:25 ** No matching stars found. Check your estimated center-point RA/Dec, and your image scaling and quality.
    03:20:25 Start slew to Blue Horsehead RGB Moon Test...
    03:20:26 (wait for slew to complete)
    03:20:32 (slew complete)
    ACP console log closed 09-Aug-2020 03:20:33 UTC

  9. #9
    Join Date
    Oct 2005
    Location
    Mesa, AZ
    Posts
    28,774

    Default

    03:20:26 (wait for slew to complete)
    03:20:32 (slew complete)
    ACP console log closed 09-Aug-2020 03:20:33 UTC


    OH!!!!! Ah ha! This is what Marco Rocchetto has been complaining about and now I have another customer with this! Oh no! This isn't a termination for plate solve failure, it "just exited" after slewing. Marco has seen this happening when a slew starts and when an image finishes. This silent exit is right at the top of my list, and unfortunately it's in part of ACP that hasn't changed in many years, the interface to the Windows Script system. Well now I have another data point. I've been trying so hard to solve this. The script engine just exits. Why? Anyway I'm working this as my #1 priority. I have no completion schedule. So far it's just you and Marco that are having the problem.

    PS in the future I'd rather the logs be attached to posts rather than being pasted into the post. Here are instructions on how to attach files.
    -- Bob

  10. #10

    Default

    Interesting... perhaps my assumption of plate solve failing due to passing clouds was incorrect. Are the pointing images saved anywhere so I could inspect them?

    Let me know if I can help debug this. I'm fairly handy software-wise... helping the DL folks debug a couple of nasty MaxIm bugs at the moment.

    BTW, how do I get all of the log file time stamps to be in local time? UTC is cumbersome.

    -Russ

 

 

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Resubmit all failed plans?
    By Voskresenskiy Anton in forum Pre-Sales Technical Questions and Help
    Replies: 1
    Last Post: May 3, 2013, 19:58

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •