SCCM Status Messages Stopped Processing

symptom

Last few days the Status messages have stopped processing on the Central server and shows the below error.

"The SMS State System message file processing experienced a severe error and suspended file processing. Review the statesys.log file for further details." with Message ID 6101.

If I look at the Incoming inside inboxes\auth\statesys.box I can see a huge list of unprocessed SMX files. Reboot usually fixes any such issue but in this case even that has not helped.Also there was no upgrade or changes done on the server but someone we have a unprocessing inbox. Checking the SQL doesnt give out an specific issue. Also I haave checked SQL and is running fine. Even tried manually running the comments but without any luck,

The statesys.log keeps showing the below

Remote client resync generated for client GUID:78ACD068-BA2B-4DFC-A092-82A4AF47826F; type = 20, update/insert result = 1 SMS_STATE_SYSTEM 3/17/2011 7:12:32 AM 6460 (0x193C)
CStateMsgReporter::DeliverMessages - Queued message: TT=700 TIDT=0 TID='Resync Initiated' SID=1 MUF=0 PCNT=0, P1='' P2='' P3='' P4='' P5='' SMS_STATE_SYSTEM 3/17/2011 7:12:32 AM 6460 (0x193C)
CStateMsgReporter::DeliverMessages - Created state message file: e:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\1kf7lf4i.SMX SMS_STATE_SYSTEM 3/17/2011 7:12:32 AM 6460 (0x193C)
CStateSystem_PS::ResyncClients - Machine 52PS5D1 is missing state messages, sending resync to site FFI SMS_STATE_SYSTEM 3/17/2011 7:12:32 AM 6460 (0x193C)
Remote client resync generated for client GUID:DF536761-C9CD-4DDA-88A2-DD6B03A4DA2C; type = 20, update/insert result = 1 SMS_STATE_SYSTEM 3/17/2011 7:12:32 AM 6460 (0x193C)
CStateMsgReporter::DeliverMessages - Queued message: TT=700 TIDT=0 TID='Resync Initiated' SID=1 MUF=0 PCNT=0, P1='' P2='' P3='' P4='' P5='' SMS_STATE_SYSTEM 3/17/2011 7:12:32 AM 6460 (0x193C)
CStateMsgReporter::DeliverMessages - Created state message file: e:\Program Files\Microsoft Configuration Manager\inboxes\auth\statesys.box\incoming\1k2oneu7.SMX SMS_STATE_SYSTEM 3/17/2011 7:12:32 AM 6460 (0x193C)
CStateSystem_PS::ResyncClients - Machine abcsert is missing state messages, sending resync to site FFI SMS_STATE_SYSTEM 3/17/2011 7:12:32 AM 6460 (0x193C)

 

solution

The solution seemed to be to remove all the "large" SMW and SMX files from the inboxes\auth\statesys.box\incoming and inboxes\auth\statesys.box\process folders.

Here are the steps I followed

• stop the SMS_SITE_COMPONENT_MANGER service in order to stop the SMS_STATE_SYSTEM component so it will stop trying to process new files.
• stop the SMS_EXECUITVE service in order to stop it from processing the files hanging the flow.  It might eventually stop on its own, but I gave up and killed the smsexec.exe process (I know, not a good practice).
• rename inboxes\auth\statesys.box\process to inboxes\auth\statesys.box\process.bak
• rename inboxes\auth\statesys.box\incoming to inboxes\auth\statesys.box\incoming.bak
• wait for SQL to calm down (use very little CPU)
• start the SMS_SITE_COMPONENT_MANGER service which automatically starts the SMS_EXECUTIVE service
• the inbox folders are automatically recreated
• move the "small" (less than 10KB) files back to incoming and process respectively.  SCCM processed these about as fast as Windows Explorer could move the files
• I'm slowly adding the "medium" (10 - 300KB) files back to the original folders, then I'll consider the "large" (300KB+) files.

SCCM processes the "small" files very quickly.  But anything over 300KB seemed to take much much longer than expected.  This seems like a bug to me; perhaps SP2 and the hotfix will resolve that quirk.

I did a little timing test.

• SCCM processed 3,928 5KB files totaling 16 MB in the process folder in 30 seconds.
• It took more than 4 minutes to process as single 1.93 MB file in the same folder.