A recent update released for Windows Azure Pack Web Sites has an undocumented change. That change is the requirement for the File Server Resource Manager role installed on nodes serving as a Publisher node and any Web Worker Nodes.
When I needed to provision a new Web Sites v2 cloud, I found it strange that all of the Publisher and the Web Worker roles were stuck in an infinite auto-repair loop. Looking at the “Role Log”, and I found the following errors:
Id = 3737, Date = Sat May 10 2014 21:43:42 GMT-0700 (Pacific Daylight Time), Level = 1, Server Name = worker, Message = System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.253.0.163:80
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at HostingController.Helper.CheckWorkerHttpAvailability(String& errorMessage)
Failed to run method 'Microsoft.Web.Farm.ControlServiceOperationProvider' on server 'worker'. Exception in response stream
Operation is not valid due to the current state of the object.
Exception in response stream
Operation is not valid due to the current state of the object.
Id = 3767, Date = Sat May 10 2014 21:44:18 GMT-0700 (Pacific Daylight Time), Level = 1, Server Name = worker, Message = Failed to run operation 'Repair'. Operation failed to complete.
Failed to run method 'Microsoft.Web.Farm.ControlServiceOperationProvider' on server 'worker'. Exception in response stream
Operation is not valid due to the current state of the object.
Exception in response stream
Operation is not valid due to the current state of the object.
Id = 3766, Date = Sat May 10 2014 21:44:18 GMT-0700 (Pacific Daylight Time), Level = 1, Server Name = worker, Message = Failed to run operation 'SetupRole (7.7.10699.8)'. Operation failed to complete.
Failed to run method 'Microsoft.Web.Farm.ControlServiceOperationProvider' on server 'worker'. Exception in response stream
Operation is not valid due to the current state of the object.
Exception in response stream
Operation is not valid due to the current state of the object.
Id = 3765, Date = Sat May 10 2014 21:44:18 GMT-0700 (Pacific Daylight Time), Level = 1, Server Name = worker, Message = Failed to run operation 'SetUpWorker (7.7.10699.8)'. Operation failed to complete.
Failed to run method 'Microsoft.Web.Farm.ControlServiceOperationProvider' on server 'worker'. Exception in response stream
Operation is not valid due to the current state of the object.
Exception in response stream
Operation is not valid due to the current state of the object.
Id = 3764, Date = Sat May 10 2014 21:44:18 GMT-0700 (Pacific Daylight Time), Level = 1, Server Name = worker, Message = Failed to run operation 'ServiceControl'. Operation failed to complete.
Failed to run method 'Microsoft.Web.Farm.ControlServiceOperationProvider' on server 'worker'. Exception in response stream
Operation is not valid due to the current state of the object.
Exception in response stream
Operation is not valid due to the current state of the object.
Id = 3763, Date = Sat May 10 2014 21:44:18 GMT-0700 (Pacific Daylight Time), Level = 1, Server Name = worker, Message = Failed to run operation 'RunRemote'. Operation failed to complete.
Failed to run method 'Microsoft.Web.Farm.ControlServiceOperationProvider' on server 'worker'. Exception in response stream
Operation is not valid due to the current state of the object.
Exception in response stream
Operation is not valid due to the current state of the object.
And if you look a little further in the Event Viewer, you’ll find this error from a .NET assembly source that starts with Microsoft.with.a.long.string.I.cant.even.remember:
The Dynamic Windows Process Activation Service encountered an error during startup. Exception: 'System.IO.FileNotFoundException: Could not load file or assembly 'srmlib, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
File name: 'srmlib, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
at Microsoft.Web.Hosting.ProcessModel.Util.SetDirectoryQuota(String directory, String quotaTemplateName, Int64 quotaLimit)
at Microsoft.Web.Hosting.ProcessModel.DynamicWas.SetupTempDirectories()
at Microsoft.Web.Hosting.ProcessModel.DynamicWas.Initialize()
at Microsoft.Web.Hosting.ProcessModel.DynamicWas.StartService(Object state)
WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
Doing some research on Google, you’ll find that srmlib is a .NET assembly for the File Server Resource Manager role. Install that role on your server and viola! Your web worker/publisher will repair itself and change it’s status from “Installing…” to Ready.
Unfortunately from what I’ve scrounged up in the Windows Azure Pack Web Sites documentation and the update release notes, none of them even documented such change. I’ve also complained about this on the Windows Azure Pack forums but as of now, there isn’t any reactions as of now. From my understanding on how web app’s works it’s understandable why the Publisher and the Web Worker requires access to the FSRM API, but why was this undocumented?
Thank you for you post. It was very helpful. I had the same problem and now my Web Worker is working fine, after installing the “File Server Resource Manager”
OUTSTANDING. Had exact same issue — this solved it. Thank You !