Scheduler crashing - The RPC server is unavailable.

[Marco Rocchetto]

Hello

In one of our installations we started to have random ACP Scheduler crashes. The engine logs says:

2019-10-06 22:58:08.6: No pending plans that can be started now.
2019-10-06 22:58:08.6: Dispatcher cycle time: 0.0 sec.
2019-10-06 22:58:25.8: ACPSequencer: transient error getting DispatcherEnabled.
2019-10-06 22:58:25.8: Unable to cast COM object of type 'ACP.UtilClass' to interface type 'ACP._Util'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{773C2BAA-487D-43C5-A5D7-E543B7D61DCC}' failed due to the following error: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA).
2019-10-06 22:58:25.8: ACPSequencer: transient error getting DispatcherEnabled.
....
2019-10-06 22:58:39.1: **EXCEPTION IN SCHEDULER:
2019-10-06 22:58:39.1: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)
.....
2019-10-06 22:58:39.2: Traceback:
at ACP.UtilClass.get_Scheduler()
at DC3.Scheduler.ACPSequencer.set_DispatcherStatus(St atusValue value)
at DC3.Scheduler.Engine.set_Status(StatusValue value)
at DC3.Scheduler.Engine.Run()
2019-10-06 22:58:39.2: DOME/ROOF FAILURE: In addition, attempting to close your dome/roof this error occurred:
2019-10-06 22:58:39.2: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)
2019-10-06 22:58:39.2: Running
2019-10-06 22:58:39.7: ACPSequencer: transient error getting DispatcherEnabled.

Note that we have no dome or roof connected or configured !

I read this RPC server error might due to issues with USB devices, but from the log is very difficult to debut - All devices are working OK.

The config is: The Sky X with Paramount, Maxim DL with FLI camera and filter wheel, FocusMax with Lakeside. AutoFocus enabled and working fine.

This has happened over the last 3 days and nothing else has changed. It happens either shortly after startup, or after taking a few images (e.g. after one hour of startup). Please see attached a few logs.

Thank you
Marco

[William Bristow]

If Windows 10 is the OS then the following may be applicable, in particular with reference to RPC errors there are known problems with Microsofts recently released and then recalled October 2019 ‘Feature Update’ 1909 and Septembers quality update for 1903.

Windows 10 bi-annual ‘Feature Update’ 1909 that was released just a week ago has been causing problems and was pulled from general distribution after four days however some users will have received this update and already automatically installed it.

In addition Septembers monthly quality update for release 1903 has caused RPC errors to appear.

MS are due to release a ‘Quality’ update tomorrow, 8th Oct, which is supposed to fix the known problems so far, unless MS find other problems and delay the QA update. It’s also been rumoured that the 1909 bi-annual feature update will be re-released with fixes tomorrow, 8th Oct but this is not confirmed and given the mess that Win 10 updates have been the last six months this may slip again.

There is also an already released manual update that fixes some of the known issues, KB4524147, which you could download and install yourself from the MS update centre and see if this includes the fixes needed for ACP but probably the best thing is to roll back Septembers 1903 quality update, or Octobers 1909 feature update, if you received that, and delay its reinstallation for a few weeks giving MS a chance to fix 1903/1909 properly.

[Marco Rocchetto]

Hi William

Thank you for your message.

Updates 1909 or 1903 have not been installed yet.

The last update was

2019-09 Cumulative Update for Windows 10 Version 1809 for x64-based Systems (KB4524148)

On 4th October 2019.

I'm quite sure problems started on 4th October....

Best regards,
Marco

[William Bristow]

Hi Marco

1809 is last years bi-annual 'Feature' release (from October 2018).

1903 is from March this year so it seems you are a bit behind and there were problems with that release that impacted RPC and that required several monthly 'Quality' patches over the last six months for a fix.

There are several options:

You could roll back the 2019-09 Cumulative Update for Windows 10 Version 1809 for x64-based Systems (KB4524148) update, and that would most likely fix the issue but if it were me, make sure you are connected to the internet and run the DISM.exe /Online /Cleanup-image /Restorehealth command from an elevated command prompt window (open a CMD window with a mouse-right-click and choose "Run as Administrator") then after DISM completes successfully run the sfc /scannow command to ensure your Windows installation is running with the latest file versions, finally then run Windows Update tool manually to ensure you have the latest 'Quality' and/or 'Feature" package installed.

You may find there are one or two 'Optional' updates waiting for you that are pertinent to fixing the RPC issue.

Both these tools, DISM and SFC Scannow can take up to an hour or more to complete depending on the speed of your internet connection and it may seem that nothing is happening, you need to wait patiently until a message appears in the CMD window to say completed or errors occurred.

The computer will need rebooting several times if DISM or SFC /SCANNOW finds and replaces faulty windows files.

Details can be found here:

https://support.microsoft.com/en-gb/...date-readiness

Alternatively, wait until tomorrow and see if MS releases 1909 and try to update manually by forcing a Check for Updates from the Windows Update app but see the following caveat.

If your Windows 10 Update system is broken and DISM will not run or Windows Update is not offering the latest 'Feature' and 'Quality' updates then there are special remedies for this and you may need the source Windows 10 installation disk to repair your current Windows environment, there is a reference to this in the linked web document.

If you don't have the installation media or it is contained on a hidden recovery partition of the C: drive then there are Windows Tech Level tools to repair the Windows 10 update system and I have used these but it would be better if a Windows Tech ran these remotely for you, unless you are really careful using the manual update repairs you can corrupt the Windows OS so badly you have to reload the OS from scratch.

William.

[Marco Rocchetto]

Hi William

Thank you - I uninstalled and rolled back KB4524148, but I didn't run DISM.exe nor sfc... Right now I just disabled updates for 30 days. Why are you suggesting to run them, and then apply again the latest quality and feature packages updates (which would likely reinstall KB4524148 again)?

Everything was working fine before this update, so rolling back might be enough ?

Thanks
marco

[William Bristow]

Hi William

Why are you suggesting to run them, and then apply again the latest quality and feature packages updates (which would likely reinstall KB4524148 again)?

Everything was working fine before this update, so rolling back might be enough ?

Thanks
marco

Hi Marco.

Yes, un-installing the KB4524148 should be enough.

The suggestion to run DISM and then re-run updates was so that you might skip KB4524148 entirely and jump straight to 'Feature' update 1903 for which the KB4524148 would not be required as this is only applicable to 'Feature' build 1809 from last year.

As you have a months grace then the latest bi-annual 'Feature' update to 1909 should be released by then (due for release tomorrow) and you may be able to jump straight to that, and by that time Bob will have had some feedback about how reliable this latest Win 10 1909 release will be.

We have found on some of the campus students computers that the Win 10 'Feature' updates don't seem to be distributed automatically even though Win 10 standard and pro editions should do so, and it is necessary to manually use the 'Check for Updates' button to start the 'Feature' updates download. The Quality updates don't seem to be affected and always appear to be installed automatically as they should for those Win 10 editions.

Regards.

William.

[Marco Rocchetto]

Great, thank you!

Just to be on the safe side, I rolled back a full disk backup to last week, prior to Win update and when system was working fine. I then disabled updates for 1 month

I will now wait for the Feature update 1909 and use DISM to jump straight to that, but only when Bob confirms this is safe to do

Best regards
Marco