Monday, January 17, 2011

Solution to problem : ERROR ExecMethod(): Failed to execute AppCreate2.

If a package gets stuck in the "Install Retrying" state at a BITS-enabled ditribution point.
 DP_Status
And you see the following error message logged in the Microsoft Configuration Manager\Logs\distmgr.log on the server:
"ERROR ExecMethod(): Failed to execute AppCreate2. error = The service cannot be started, either because it is disabled or because it has no enabled devices associated with it."
It might be because of the "COM+ System Application" service is set to anything other then automatic startup. It may be set by a group policy or it could have been set manually.
More info can be found here: http://support.microsoft.com/kb/917485

Wednesday, January 12, 2011

Understanding Site to Site Communication in SMS/SCCM

Found this great article here:

http://blogs.msdn.com/b/steverac/archive/2010/07/16/understanding-site-to-site-communication-in-sms-sccm.aspx


 

Understanding Site to Site Communication in SMS/SCCM

I’ve been intending for a while to write a post on the internals of site to site communication in SMS/SCCM.  The internals really haven’t changed much since SMS 2.0 so no matter what version of the product you are using, the detail here will apply.
For some reason when I think of site to site communication I always think in terms of package distribution.  Certainly site to site communication comes into play with package distribution assuming the package is being staged to distribution points at a child site – but package distribution is only one item that requires solid site to site communication.  Other examples include site control file updates, status messages, state messages, discovery records, inventory, etc. 
Information going site to site does so based on priority.  These priorities help ensure ordering so critical data goes first.  There are three priorities – high, medium and low.  Site control file updates always go high priority, status messages by default go low priority and most everything else goes medium priority by default.  Some priorities are configurable (packages, status messages) but most are not (site control file updates). 
So what if I have a very large package in progress at medium priority and I want a new package to be deployed with high priority?  Will SCCM pause sending the medium priority package in preference to my high priority job?  No.  Once sending is underway it continues until completion and then the highest priority item after that will be transferred next.  The ‘scheduling’ of site to site jobs is handled by a component strangely named the scheduler!  More on that as we go along.
So what needs to be configured for site to site communication?  Two things are needed - an address (to tell us where the other site is) and a sender which is used for transferring the data.  There are several senders to choose from but the two most common are the LAN sender and the courier sender.  I’ve only seen another type of sender in use once years ago. 
The courier sender is a mechanism that allows sending of data between sites using external media – such as a DVD, CD-ROM, flash drive, etc.).  This sender is typically used in scenarios where large amounts of data need to be transferred and the WAN environment is insufficient to handle the load.  Further discussion of the courier sender is beyond the scope of this post.
The LAN sender is the most common sender choice and simply means that data will be transferred between sites using the network. 
The discussion of senders and address opens up a whole different branch of this topic related to strategies for placing addresses.  In a three tier hierarchy, for example, should the central site have direct addresses for all child sites – including those at the third tier?  The answer almost always is no – instead of direct addressing the central site benefits by taking advantage of fan-out distribution (if you aren’t sure what that is, see the ‘using fan-out’ section of this document)
OK, with that introduction let’s get into the topic.  For simplicity we will use a two tiered hierarchy for discussion.  CEN is the central site and PRI is the primary child site.  The concepts discussed also apply to 3 and 4 tier hierarchies and secondaries follow as well.  For purposes of discussion, a two tiered structure keeps things simple.
On CEN we have an address configured for PRI – so we know were it is and can send data to it – and we have a sender configured.
image
We also have an address configured on PRI back to CEN
image
The address settings are where all of the configurations are available.  On the general tab we can configure items such as the destination computer name and the account to be used for site to site communications.  If left blank the computer account is used – and that often is preferable so we can avoid maintaining passwords.  If you prefer a user account can be specified here and must be specified if sites are separated by an untrusted security boundary. We also see a choice here for address priority
image
The schedule and rate limit tabs are used to configure when the particular address will be available and whether or not it can run at 100% or needs to be throttled.  Note that if any throttling is selected the ability to send multiple packages to a site is disabled. 
The sender is separate from the address.  The sender node defines all senders in use at a site.  In our example we just have the standard sender.
image
On the sender advanced tab we can specify concurrent sending settings between sites but, as mentioned earlier, if address  throttling is enabled the settings are overridden and only one thread is active per site.
image
OK, so thats all of the configuration to be done in the UI.  Remember that these two items need to be configured on both sites that are communicating.  With this done, lets see how site to site communication works behind the scenes.  For our walk through we will trace a package as it leaves distribution manager en route from CEN to PRI.
The flow of events is as shown. 
image
Note that to illustrate the movement of data through the system I have artificially paused processing along the way.  If you look at your equivalent folders you likely won’t see data in many of the folders I’m showing unless there is a problem or the site is very busy.
Our test package is created and sent to a distribution point at our child site.  We see in the distribution manager log file that a minijob has been created that will send the content to PRI.
Distmgr.log
--------------
Created minijob to send compressed copy of package CEN0000D to site PRI.  Transfer root = \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK.   
STATMSG: ID=2335 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=STEVERACCEN SITE=CEN PID=2580 TID=5112 GMTDATE=Tue Jul 13 01:58:23.632 2010 ISTR0="CEN0000D" ISTR1="PRI" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="CEN0000D"   
Package CEN0000D is new or has changed, replicating to all applicable sites.   
Sent package changes to site PRI for package CEN0000D
   
Again a reminder that we started our site to site communications demo using distribution manager content – but it could just as easily have been collection replication, site control file replication, status message replication, state message replication, etc.
The minijob is placed in replmgr.box which is the component that handles replication.  The contents of replmgr.box is show below:
image
There are 5 folders here.  

Incoming This is where all data that is incoming from a parent or child site is placed.  Replication manager has the job of figuring out what kind of data it is, which component should handle it and where it should be placed – and then the content is copied there for final processing (more on this once we get over to the server side.  This folder is not part of our example on CEN (but will be on PRI) so we will skip for now.
Outbound
This is the location where all data is placed by various ‘sending’ components that is destined for another site.  Data is categorized as to priority and placed in the appropriate subfolder under Outbound – either low, normal or high priority.  Our minijob – known as a replication job - was created here  in the normal priority folder as shown.

image
On a side note (rant) – I can’t count the number of administrators that seem to think deleting files in the various inboxes is an OK thing to do.  In many cases you will get away with it – once you enter the site to site mechanism (and we have just entered it) you better know exactly what you are doing before deleting.  Best rule – don’t delete unless you KNOW.  Sometimes it’s required but many times its not.
OK, so what is this RPL file – if we open it in notepad we can see some detail about it.
image
Note:  It’s OK to look at files in an editor but don’t try to edit or save them.  Many  files contain binary information that will not save correctly and will render the file unusable.
Much of the file contents appears to be gibberish but some content is readable – we see details such as the name of the package – software dist flow – the source directory for the package (and the content that will be transferred) – pkgsrc\rockstar demo CEN, the compressed PCK directory - \STEVERACCEN\SMS_CPSC$\CEN0000D.PCK, the security rights assigned to the package, the destination for the package – the destination site PRI – Primary Child, the sending side CEN and the thread involved – Distribution Manager.  Oh, before you go look for a folder called SMS_CPSC$, it’s a hidden share – it’s actually the SMSPKG folder as shown.
image
Note:  If we jump ahead there is already a .job file created in the schedule.box folder as shown – this is the minijob that will act on data in replmgr.box\ready – but more on that in a bit.
image
Process Data moves from outbound to process where everything is checked and made ready for transfer.  Typically you won’t see files in the process folder unless there is a problem.  The processing was so quick I wasn’t able to catch it to show here.
Ready
This folder contains data that is ready to be sent.  Once data arrives here the next steps are to schedule it for sending and to send.  If you look in the ready folder on a reasonably active site you will typically see folders with data inside that are destined for another site in the hierarchy.  The highlighted area below shows that the contents of this folder are destined for site PRI.
image If we look inside the folder we see another .RPL file.
 image The filename has changed from the one we found in the incoming folder – does that mean the contents of the file have changed?
image Nope, same file – except now it has been processed, validated and is ready for sending.

History
The history folder maintains a history of all transactions that have flowed through replication manager – either incoming from other sites or outbound to other sites.  If we look in the history folder we see a .TRS file- known as a transaction file – which records movement through the replication manager inbox.  In some scenarios information will fail to replicate due to invalid values in this file.  In earlier versions – such as SMS 2.0 – this fact wasn’t logged in replmgr.log but is now.
image
All of the activity just described is recorded in just a few lines from the replmgr.log
Scanning low priority outbound replication directory.    
Did not find any additional replication files.    Minijob created.  Priority 2, transfer root (\\STEVERACCEN\SMS_CEN\inboxes\replmgr.box\ready\2_gg1xd8.PRI).
   
As we leave the replmgr.box note that content will remain here until notification is received that sending has completed.  This is important to ensure pointers to content are not removed until sending is finished.
The next step is to schedule the jobs for sending.  If we take a look at the schedule.box we see jobs waiting for processing.  We also see a few folders. 
image
We will go through the folders in the order of processing but first we need to get processing started!  The log snip below shows the processing loop as schedule handles these jobs.  I’ve added commentary to the log to point out a few key points of interest.
SMS_EXECUTIVE started SMS_SCHEDULER as thread ID 3344 (0xD10).   
========  Starting Processing Cycle ( 07/13/10 18:07:40 Pacific Daylight Time) ========   
Updating in-memory send request list...   
Deleting in-memory send requests that are no longer in inboxes.   
Deleting in-memory slave send requests that are not referenced by a  master request.
  
Deleting any completed jobs
     <Scheduler cleans up any completed jobs at the start of the 
      processing loop>.
   
========  Processing Routing Requests  ========   
========  Updating Send Request List  ========   
Updating in-memory send request list...   
Deleting in-memory send requests that are no longer in inboxes.   
Deleting in-memory slave send requests that are not referenced by a  master request.   
========  Building Outbox Send Request Lists  ========   
Examining 0 send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests...    Sorting list of 0 send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests by priority and time...   
Examining 0 send requests for outbox \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN...   
Sorting list of 0 send requests for outbox \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN by priority and time...   
Sorting complete, updating perf counters with number of master send requests...    
     <Check to see if there are any jobs that are in process of   
     sending in the outboxes\LAN folder – remember we are using
     the LAN sender>
========  Checking Status of All Send Requests  ========   
========  Scheduling All Unscheduled Send Requests  ========   
========  Cleaning Up Routing Packages  ========   
========  Finished Processing Cycle ( 07/13/10 18:07:40 Pacific Daylight Time) ========   
Sleeping for maximum 10 seconds.   
========  Starting Processing Cycle ( 07/13/10 18:07:50 Pacific Daylight Time) ========   
========  Processing Jobs  ========   
Updating the in memory job list...   
0 jobs found in memory, 2 jobs found in job source.
     <Nothing is in memory or in progress so we look to schedule
      anything new>
   
<Activating JOB 0000004F> [Software Distribution for Software Dist Flow, Package ID = CEN0000D]   
    Destination site:  PRI, Preferred Address: *, Priority: 2   
    Instruction type:  MICROSOFT|SMS|MINIJOBINSTRUCTION|PACKAGE   
    Creating instruction file: \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\0000004F.I16   
    Transfer root:  \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK   
    Begin to calculate signature on \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK   
    Signature on \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK is 3ABF1ADA72390CF25F3499D925D2EA2FB60C3D3B0801FF6B85E7F8135A1B3B8CDB1052AC6DDCC1F6E273BACB9545FBEB16120F19E7A6DA2540482CDEC393AEA225D3A1E1CBE115843B44C4361090507E6AC7D95B4D769DA211A0E1B89EE6902490345E8938DCE5AFD6F359A4A8A9A2C27B532450844356DF433886DEE97AEFAB   
    Hash algorithm ID on \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK is 0x800C   
    Instruction (and package) file created.  Mark job active.   
    <JOB STATUS - ACTIVE>   
<Updating JOB 0000004F> [Software Distribution for Software Dist Flow, Package ID = CEN0000D]   
    Destination site:  PRI, Preferred Address: *, Priority: 2   
    Created new send request ID:  20027CEN   
     <OK, so now we have TONS of information.  We see a new job
     being activated along with identifying info – in this case a
     package – what the destination site is – in this case a direct
     child but it could be that we are sending this to a third tier site
     in which case that site code would be seen.  We also see the
     instruction file being created and the fact it is placed in the
     tosend directory. The location of the compressed content is also
     seen along with the signature on the package.  Finally we see
     the job marked active>
 
<Activating JOB 00000050> [Inter Site Replication]   
    Destination site:  PRI, Preferred Address: , Priority: 2   
    Instruction type:  MICROSOFT|SMS|MINIJOBINSTRUCTION|REPORTING   
    Creating instruction file: \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Imh        Transfer root:  \\STEVERACCEN\SMS_CEN\inboxes\replmgr.box\ready\2_gg1xd8.PRI        Creating package file: \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Pj8        Begin to calculate signature on \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Pj8        Signature on \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Pj8 is 4FE19318CFC83252BA271A43260CB0DAEC2D513028E4F6AFEA0E4DEAE56C62A4FC5650CB40AB2EDEB064B618C74242351552DA12B95EEACFE1066C49C55F844C4DE051CC583C7D4498CF741AB3721631E9C8877533959E153C633FACACE41D77EB204A48CBD4E33B9591A452E0138B0BACA7B37DA4B2B7D7E5564FA280E71602        Hash algorithm ID on \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Pj8 is 0x800C   
    Instruction (and package) file created.  Mark job active.   
    <JOB STATUS - ACTIVE>   
<Updating JOB 00000050> [Inter Site Replication]   
    Destination site:  PRI, Preferred Address: , Priority: 2   
    Created new send request ID:  20028CEN   
Updating the in memory job list...   
2 jobs found in memory, 2 jobs found in job source.   
Job count is 2.  
     <The second job is activated – this is a replication job – note
     some of the differences but still basically the same structure –
     now we have both jobs activated>
 
========  Processing Routing Requests  ========   
========  Updating Send Request List  ========   
Updating in-memory send request list...   
     <When we mark the job active we also create the send request
      in the tosend folder – the next step begins to evaluate our
      progress sending – in this case the sender is stopped (on
      purpose for this blog entry) so no activity is logged – we will
      start it shortly.>

====  Found 2 send requests in outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests.   
Send Request 20027CEN   JobID: 0000004F   DestSite: PRI   FinalSite:      State:  Pending   Status:            Action:    None  
Total size:       80k   Remaining:       80k   Heartbeat: 18:07   Start: 12:00   Finish:   12:00   Retry:         SWD PkgID: CEN0000D   SWD Pkg Version:    1    
    
<We see the status of our two send requests – one above and
     one below.  This status is very useful troubleshooting
     information when looking to see if data is flowing through the
     sender or if there are large jobs in progress>

Send Request 20028CEN   JobID: 00000050   DestSite: PRI   FinalSite:      State:  Pending   Status:            Action:    None   Total size:        1k   Remaining:        1k   Heartbeat: 18:07   Start: 12:00   Finish:   12:00   Retry:   
Deleting in-memory send requests that are no longer in inboxes.   
Deleting in-memory slave send requests that are not referenced by a  master request.   
========  Building Outbox Send Request Lists  ========   
Examining 2 send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests...    Sorting list of 2 send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests by priority and time...   
     <Checking our outbox for send requests and routing them to the
     appropriate sending folder if the aren’t already there – in this 
     case the LAN folder>

Examining 2 send requests for outbox \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\
LAN...   
Sorting list of 0 send requests for outbox
\\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\
LAN by priority and time...    Sorting complete, updating perf counters with number of master send requests...   
========  Checking Status of All Send Requests  ========   
====  Checking send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests.   
    Checking unscheduled send request 20027CEN   
    Checking unscheduled send request 20028CEN   
     <These send requests just came in so are not scheduled yet –
     working on that>

========  Scheduling All Unscheduled Send Requests  ========   
Starting routing analysis for 2 requests   
Routing Analysis: Combining requests for same destination site and package...   
Completed routing analysis  
     <Checking the best route through the hierarchy for delivering
     content – this really is important if more than two tiers deep –
     this is where fan out distribution comes into play>
 
====  Scheduling send requests.   
Starting routing analysis for 2 requests   
There is direct address to site PRI, no need for routing for send request 20027CEN.   
There is direct address to site PRI, no need for routing for send request 20028CEN.   
Routing Analysis: Combining requests for same destination site and package...   
Completed routing analysis   
    Scheduling send request 20027CEN.   
        Selected outbox.  From: C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests to \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN.   
    Scheduling send request 20028CEN.   
        Selected outbox.  From: C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests to \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN. 
     <We have gone through all of our checks and finally the job is
      scheduled for sending.and we cleanup>  

========  Cleaning Up Routing Packages  ========   
Time to clean up completed jobs and send requests list.   
Deleting completed jobs.   
Deleting job data that's not associated with valid jobs.   
========  Finished Processing Cycle ( 07/13/10 18:08:00 Pacific Daylight Time) ========   
Sleeping for maximum 300 seconds.
   

Note:  Obviously there is a lot going on here and you would see even more if you looked at a production site with multiple sites and lots of activity.
Note:  Scheduler is a single threaded component.  That means that when it starts it has to work it’s way through the entire processing loop before it starts again.  On a very busy site it is common to see that one full loop of logging is not captured in a single log file.
So we’ve seen the processing in the log – what about the contents of the folders at the end of the processing loop?  As mentioned earlier, lets take them in order of processing.
If we look at the .job file in the root of schedule.box we notice some interesting things.
image
We definitely see some familiar content based on examining the RPL file but we see three new pieces of information that are worth noting.  First note the 0000004F identifier.  This is both a consistent identified that helps us link files together in schedule.box and also the filename of our job file.  We also see here the filename of our file in the tosend folder – 0000004F.I16 and also the filename of the send request for this job that has been placed in outboxes\lan to initiate sending – 20027CEN.  Review the other elements of the file to see what you can identify.
ToSend The to send folder is the staging area showing what content is to be sent to the destination.

image
Following our filename our job of interest is 0000004F.I16.  Reviewing the contents of this file we see the same identifier and more pointers back to our source.
image
Requests
Requests is a folder that we see as part of processing in the log entries above but it’s hard to catch data in this folder since requests is a quick stop along the way to outboxes\lan.
Outboxes\LAN
The outbox folders – in this case LAN since we are using the LAN sender – is where our send requests are placed to flag sender to wake up and begin sending the content defined in the tosend folder. Note that the .CPB and .DAT files are always present in this folder.
image
If we look at our appropriate SRQ file – 20027CEN.SRQ as found earlier in the .job file content – we see this file pulls everything together so the sending process can begin.  We will watch the sending process take place next.
image
UID Before leaving the schedule.box folder we should stop a the UID folder.
image
In this folder we have two files – a .REQ and a .JOB.  These are files used to track the progress of various items as they are sent through the scheduler.  These files are critical to the operation of scheduler and should never be deleted.  If they are you can recreate them as blank text files.  If you have one file but not the other just take the filename of the one you have, increment it by a hundred or so and create the missing file with that new filename.  If neither file are present you can recreate both – use a filename for both that would likely be above the value you might expect there.  I like to use a filename of 00100000 since it would take a really long time for a site to send that many jobs so the filename is likely safe and leaves plenty of room to grow.
OK, so lets SEND ALREADY!!!!  When we bring the sender back online the content is immediately seen and sending starts.  The relevant portion from sender.log is below with comments added.
Wrote 60123 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.PCK at position 21504   
     <Starting to send the PCK file referenced earlier>
Sending completed [C:\SMSPKG\CEN0000D.PCK]   
Finished sending SWD package CEN0000D version 1 to site PRI   
STATMSG: ID=3533 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_LAN_SENDER" SYS=STEVERACCEN SITE=CEN PID=2108 TID=3972 GMTDATE=Thu Jul 15 14:21:21.646 2010 ISTR0="CEN0000D" ISTR1="1" ISTR2="PRI" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=3 AID0=400 AVAL0="CEN0000D" AID1=410 AVAL1="1" AID2=409 AVAL2="PRI"   
     <It’s not a big file so that quick and we are done>
Attempt to create/open the remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP   
Created/opened the remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP   
Attempt to create/open the remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP   
Created/opened the remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP   
Sending Started [C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\tosend\0000004F.I16]   
Attempt to write 1024 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP at position 0   
Wrote 1024 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP at position 0   
Attempt to write 32 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP at position 1024   
Wrote 32 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP at position 1024   
Sending completed [C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\tosend\0000004F.I16]    Finished sending SWD package CEN0000D version 1 to site PRI   
STATMSG: ID=3533 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_LAN_SENDER" SYS=STEVERACCEN SITE=CEN PID=2108 TID=3972 GMTDATE=Thu Jul 15 14:21:21.658 2010 ISTR0="CEN0000D" ISTR1="1" ISTR2="PRI" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=3 AID0=400 AVAL0="CEN0000D" AID1=410 AVAL1="1" AID2=409 AVAL2="PRI"   
     <We have sent everything down for this package – the package
     itself and the instruction file>

Renaming remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP to \\steveracpri.startrekng.com\SMS_Site\20027CEN.SNI   
Rename completed [\\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP]    Sending completed successfully
     <Rename the file to SNI so despooler on PRI can work with it>

Note:  At this point all sending is complete from CEN to PRI – now PRI has to process that sent data forward. 
Notice also that we are sending to SMS_Site – that is a share name that translates to the inboxes\despoolr.box\receive folder on PRI. 
image
All sending, regardless of content, is sent directly to the SMS_Site folder on the destination site servers.  The content is sorted further from there.
How is it sorted further?  If we look at the despoolr.log on PRI we will see our .SNI file being processed.  Before we jump into the details of the log – a log reading tip.  Many components in SCCM are multi-threaded meaning they can work on multiple things at once.  This is potentially confusing as the entries in the log may get mixed between various lines of processing.  One way to keep things straight is the thread ID that is recorded in the log.  If we look at despoolr.log we see this very thing. 
image
Note that we see thread 5FC (1532) is being spawned to handle processing of our 20027CEN.SNI file.  There are several other despooling threads spawned after that and then the second hightlight shows thread 5FC starting to work.    You can also filter based on the thread ID – you have to use the decimal thread ID – which will display only the parts of the log that are processed on the thread ID you are interested in.  Thats what I did for the log snip below.
image
Just a couple of tips to help you keep things straight.
Now to the log showing processing of our .SNI file – with commentary added.
Verifying signature for instruction C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\despoolr.box\receive\ds_9rem4.ist of type MICROSOFT|SMS|MINIJOBINSTRUCTION|PACKAGE   
     <Processing an instruction file which is our package> 
CPublicKeyLookup::CPublicKeyLookup("CEN")   
CPublicKeyLookup::CPublicKeyLookup("CEN") Initializing to file: C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\hman.box\pubkey\CEN.pkc   
CPublicKeyLookup::GetNextKey() Getting Iteration: 2   
CPublicKeyLookup::GetNextKey() Checking C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\hman.box\pubkey\CEN.pkc for Key0   
CPublicKeyLookup::GetNextKey() No Match Found, Trying C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\hman.box\pubkey\CEN.pkp   
CPublicKeyLookup::GetNextKey() Found Key: 0602000000A40000525341310004000001000100FD8307BF5E9F68D42C559EBD8183CF87F200C949130B9A16C54271CB03B657755D13DB82B55DE87A6290AC62C99367BC44A32EA8B77688E70A46C3FB5576C24CC3F3929F9E1F51F9A07CACC9BD11831D73526F3A0B6DBD3287FEA2653EAA5E3682B64183F2383857EC3A1590430C2EDFA6894A7A5D17938B95674F88D096E6D1   
             
     <The section above is trying to lookup the public key info for site
      CEN so we can validate this data came from a trusted source>
Begin to calculate signature on C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\despoolr.box\receive\ds_9rem4.pkg using hash algorithm ID 0x800C   
Verified package signature from site CEN   
CPublicKeyLookup::CPublicKeyLookup("CEN")   
Signature checked out OK for instruction coming from site CEN, proceed with the instruction execution.   
     <The package is checked against signature to verify it is OK>
Executing instruction of type MICROSOFT|SMS|MINIJOBINSTRUCTION|PACKAGE   
STATMSG: ID=4407 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DESPOOLER" SYS=STEVERACPRI SITE=PRI PID=1224 TID=1532 GMTDATE=Thu Jul 15 14:21:26.489 2010 ISTR0="CEN0000D" ISTR1="1" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="CEN0000D"  
     <Executing the instruction and decompressing the package> 
Received package CEN0000D version 1   
Use drive C for storing the compressed package.   
Insert package CEN0000D to the distribution source.  
     <Here we are forwarding the package to distmgr.box.  This is
     where it gets interesting – in every other case I know of we
     forward to replmgr.box/incoming for further processing – I’ll
     show an example of that shortly>
 
Stored Package CEN0000D. Stored Package Version = 1   
Forward package status for pkg CEN0000D to site CEN   
STATMSG: ID=4400 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DESPOOLER" SYS=STEVERACPRI SITE=PRI PID=1224 TID=1532 GMTDATE=Thu Jul 15 14:21:30.723 2010 ISTR0="CEN0000D" ISTR1="\\STEVERACPRI\C$\SMSPKG\CEN0000D.PCK" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="CEN0000D"   
Despooler successfully executed one instruction.    

And there you have it – despooler is finished and now the package is being nicely processed further by the distribution components.  So what about other types of replicated items – the log snip below shows another random transaction (not package related) where despooler forwards to replmgr.box/incoming to allow it to further process.
Signature checked out OK for instruction coming from site CEN, proceed with the instruction execution.   
Executing instruction of type MICROSOFT|SMS|MINIJOBINSTRUCTION|REPORTING   
Moved C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\replmgr.box\ut19p8x7.TMP\1ebn31f1.RPL to C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\replmgr.box\incoming\v8xgi1h6.RPL    Despooler successfully executed one instruction.
   

As you can see – the trick to following the flow on the destination site is seeing which inbox despooler forwards to after processing.
If replmgr is used to handle the file type all it does is look at the content and forward it on – something similar to the below log snip.
Moved incoming replication file to C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\objmgr.box\INCOMING\CEN_24.SDM   
Moved incoming replication file to C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\ciamgr.box\CEN_2.CAR   
Moved incoming replication file to C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\objmgr.box\INCOMING\CEN_11.CID
 
Before we wrap up – lets go back to discussion of despooler.  What exactly does a .SNI file look like anyway?  A sample .SNI file is below.
image
Beyond SNI’s there may be other accompanying files that need to be processed as part of the instruction – in this case there is also a .pck as shown.
image
Not a great deal to discuss about the files above but I show them for completeness of the flow.
So there you have it – site to site communication end to end.  Will this flow exactly match your scenario?  Likely not – but generally should be similar.  Hope this helps!

Tuesday, January 11, 2011

Troubleshooting packages that just won't install on DP's

Found this good article here:
http://nikifoster.blogspot.com/2009/12/troubleshooting-packages-that-just-wont.html













These are the steps:


1. First delete the package from that DP using the console.
2. Wait for the package deletion message to arrive at the DP (keep monitoring the distmgr.log file).
3. Wait for this message to reach the central site (in the console, it should show version number as 0). This is important.
4. Run this query on the sccm database. If there is a child primary site, then run it on both:


update pkgstatus set sourceversion = 0, status = 2 where id = 'packageid' and sitecode = 'xxx'


5. Now, reassign the package to the DP. This will cause the whole package to be resent and the package will be installed successfully at the DP.


Monday, January 10, 2011

SMS Package deployment to Distribution Points-compressed files haven't arrived yet

Found this article on internet for troubleshooting package-DP problems:
http://blogcastrepository.com/blogs/mattbro/pages/687.aspx

First off, the obvious things to check or try for a misbehaving package:
  1. Go to System Status->Package Status and browse to the Package/DP having issues. Right-click on the DP and choose “Show Messages“-> “All“. This should show you some useful information on what might be going on.
  2. Confirm that the Source Directory for the Package actually exists.
  3. Try re-pushing the Package to the DP.
  4. Make sure the permissions on your DP are good. I found one DP where the Administrators group had their permissions applied only to the root of the Dist share. This caused problems with any packages that were configured to store themselves in a nested subfolder. SMS was able to create the first level folder but was not able to create the subfolders under that folder.
(Let me preface the next part by saying that almost all of our remote DPs are also CAPs and MPs - I'm not exactly sure what components will be on a separate server if you have these functions split up. The same methods should work though)
Ok, now on to the fun stuff. After doing the things above, I was left with about 50 instances of packages that didn't want to deploy properly.
After doing a bit of digging, I found the following errors in DISTMGR.LOG on the remote DPs for every package that wouldn't deploy:
Processing incoming file E:\SMS\inboxes\distmgr.box\INCOMING\Z3AWAX9I.PKG.
Package 009000B7 requires a newer version (3) of source files and the new compressed files haven't arrived yet, current version is 2, skip E:\SMS\inboxes\distmgr.box\INCOMING\Z3AWAX9I.PKG
My friend Brian Tucker told me about a method he has used in the past to fix these issues. I tried it and it had a good success rate of fixing the issue. The only thing I didn't like about it is the fact that it forces the package to re-deploy too ALL DPs. Sometimes you have a really big package that you don't want to saturate your entire WAN with. Also, I wanted to understand why changing the sourcepath forced the package to redeploy.
This leads us to PCK files. PCK files are the compressed copies of each package that SMS sends to each DP. Once the PCK gets copied to the DP it gets extracted onto the Dist share. At least, that is what is supposed to happen. For the majority of the packages that were misbehaving, this wasn't occurring. I was able to fix the majority of them by using the following process:
  • Manually copy the PCK from \\primaryserver\smsdrive\SMSPKG to \\DPServer\smsdrive\SMSPKG
  • Make sure the Read-Only attribute is set on the PCK file. I also removed the Archive attribute so it would be like all of the other PCK files.
  • Run PreloadPkgOnSite.exe to force the package to decompress and register itself propery at the DP.
(PreloadPkgOnSite.exe comes with the SMS 2003 Toolkit 2. All you have to do is give it the Package ID as a command-line parameter)
  • Re-push the package to the DP. You can use the SMS Console for this. This will not copy the compressed content to the server again since we just manually did this. This will just give SMS a little kick to help things along.
During my testing with PCKs I was able to determine why Brian's fix (linked up above) works most of the time. Whenever you change the sourcepath on a package, it regenerates the PCK on the Primary. The new PCK is then deployed to all of the SMS servers. It seems to go ahead and push itself on through on the DPs that were having problems fairly well after this. I have a feeling this is because the package has a new source version so the problematic DPs start the entire deployment process over again. Also, I found that I didn't have to permanently change the sourcepath on the package. All I had to do was just add a number to the end of it and then go right back into the sourcepath and change it right back to what it originally was set to. This way you don't have to actually rename the folder on the file server.
While testing this process I did run into a few cases where updating the source path did not generate a new PCK. If you are ok with redeploying to all DPs, I found that you could go ahead and delete/rename the PCK on the Primary (located at \\primaryserver\smsdrive\SMSPKG ) and then update the sourcepath and detailed above. This will definitely force it to regenerate the PCK on the Primary.
Now for the remaining packages... I had a few that were giving me the following error when I tried to run PreloadPkgOnSite.exe on them:
The compressed package path for package 0090002C is already set in the database
OR this is the site where the package was created.
 There is no need to use this tool for this package on this site.
Apparently these packages got just a little bit further into the deployment process before they bombed out. I was able to get around this problem by deleting \INBOXES\DISTMGR.BOX\packageid.PKG on the DP. Then I was able to run PreloadPkgOnSite.exe on that DP for that package without any problems.
You might have to re-push the package to the DP before running PreloadPkgOnSite.exe. If you get the following error, just re-push and wait a few minutes:
Failed to get the specified package 0090002C in the database. Please check if you have instance rights to that package.
You should see the following when it works properly:
Forward package status for pkg 0090002C to site 009
****** Successfully set the Compressed Package Path on this site ******
****** Successfully forwarded the information up the hierarchy ******
Note: At any time, feel free to go into the SMS Console and re-push the package to the DP. It really seemed to help kick things through once I did some of the other steps to fix the underlying issue. I always did a re-push after running PreloadPkgOnSite.exe.
That's pretty much it. These steps might not work for every situation but I have a feeling that most people running into this problem can fix the majority of their issues with these steps.

Other misc. info I found while doing my testing:
  • Each package also gets the following files generated when it gets deployed:
    • \SMS\inboxes\pkginfo.box\packageid.NAL
    • \SMS\inboxes\pkginfo.box\packageid.PKG
    • \SMS\inboxes\pkginfo.box\packageid.ICO (this is a folder)
    • \SMS\inboxes\colleval.box\packageid.CLF
    • \CAP_XXX\pkginfo.box\packageid.NAL
    • \CAP_XXX\pkginfo.box\packageid.PKG
    • \CAP_XXX\pkginfo.box\packageid.ICO (this is a folder)

I messed around with deleting these files to help things along but it didn't seem to make a difference.
  • Removing the package from a DP (via the SMS Console) and then re-adding it never seemed to help things much.