Archive for the ‘scom’ Category

using pscp.exe in a batch file and getting around the host key

March 19, 2013

I was trying to automate an scp file copy for work and having a hell of a time with the host key. Even though I was logged in as the same user that the script is running, when it’s run non interactive, for some reason it cannot find the host key in the registry, and fails. Some info.

  • putty and  pscp.exe keep their host keys in – [HKEY_CURRENT_USERSoftwareSimonTathamPuTTYSshHostKeys]
  • if you run your pscp.exe with the –batch command, you cannot input any console input
  • in order to get this to work, I had to echo a “Y” to it like thisecho y | C:_scriptspscp -sftp –v  -etc. etc

Basically, If you try to echo the Y with the –batch option, then you can’t because the –batch option makes it ignore any input. In order to make this secure, I ran it once the way I have listed above, and once the host key was stored in the right place, I removed the echo and added the batch again.

Advertisements

strange .net runtime error and the fix

July 20, 2011

We had done a POC for Foglight by Quest back in March and ended up not purchasing the product because we already had SCOM in place. After the uninstall and ever since, we have had some issues.

  • errors logged whenever an app pool starts/restarts containing this description
  • .NET Runtime version 2.0.50727.3603 – Failed to CoCreate profiler.
  • .net health monitoring would log an app pool start but not log an app pool stop or shutdown
  • I had the suspicion that this was related to random app pool restarts during the day on one of our webservices
  • other debuggers would not attach to an app pool and work (like avicode that comes with SCOM)
  • I looked for how to fix this since March, of course I was too stubborn to call Quest, so that’s my fault. Much googling didn’t resolve much so when I finally did figure this out, I wanted to post what I found. It turns out that debugging is set in an environment variable. Seems like it’s a session variable because it’s set in the parameters of a service, so that way it runs with the service in the user context of the service. This is something I had never run across before so it seemed kinda odd.

    In the end we had to remove the reg key “Environment” (and the contents of the key) from the two locations. This key is what sets the debugger to enabled and tells it which debugger to use.

    • [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesW3SVC]
    • [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesIISADMIN]

    Once these have been deleted, all you have to do is IISRESET the server and the error is gone and random app pool restarts have ceased. As of this writing I have not tried to reinstall the SCOM AVIcode to the webservice, but I’m certain that it will work now.

    AV exclusions

    May 25, 2011

    This comes up often and I can never find everything in one place. This is a good post with a bunch of info and links.

    SQL query to tell if SQL logins are using Kerberos or NTLM

    March 17, 2011

    Not explanation needed here, if you know what you are looking for, this is it.

    SELECT 
    s.session_id
    , c.connect_time
    , s.login_time
    , s.login_name
    , c.protocol_type
    , c.auth_scheme
    , s.HOST_NAME
    , s.program_name
    FROM sys.dm_exec_sessions s
    JOIN sys.dm_exec_connections c
    ON s.session_id = c.session_id

    MSMQ MP for SCOM–has issues..

    February 21, 2011

    This is just an example of the kind of stuff I find in the MSMQ MP for scom. Is this alert about the SIZE of the messages or the NUMBER of messages?

    image

    scom missing performance counters

    January 7, 2011

    I recently had to make some changes to my scom test environment server. It’s a single virtual machine with scom, sql, ssrs, the works on it. Despite what one would think, it has performed well with about 30 agents. Recently I added about that many more and it pushed up the ram to the 4gb max that I had given it. When I added more I realized I had used 2008 x32 for the OS (oops) and was not able to see the extra ram. After some research I found that I could do an upgrade from 2008 standard to enterprise, which is what I did. Afterwards, scom wouldn’t run because the SDK service would not start. This is the event log message.

    Log Name:      Operations Manager
    Source:        OpsMgr SDK Service
    Date:          1/7/2011 2:25:48 PM
    Event ID:      26380
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:     

    Description:
    The System Center Operations Manager SDK Service failed due to an unhandled exception. 
    The service will attempt to restart.
    Exception:

    System.TypeInitializationException: The type initializer for ‘Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess’ threw an exception. —> System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
       at System.Diagnostics.PerformanceCounter.Initialize()
       at System.Diagnostics.PerformanceCounter.set_RawValue(Int64 value)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager.CreateCounter(String categoryName, String counterName)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager.CreateCounters(PerformanceCounterCategoryDescriptor categoryInfo)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager..ctor(String instanceName, PerformanceCounterCategoryDescriptor categoryInfo)
       at Microsoft.EnterpriseManagement.Mom.Internal.SingleInstancePerformanceCounterCategoryDescriptor.CreateManager()
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkPerfCounterManager..ctor()
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess..cctor()
       — End of inner exception stack trace —
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess.Initialize()
       at Microsoft.EnterpriseManagement.Mom.Sdk.Service.SdkSubService.AsyncStartCallback(Object notUsed)
       at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="OpsMgr SDK Service" />
        <EventID Qualifiers="49152">26380</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2011-01-07T19:25:48.000Z" />
        <EventRecordID>10025424</EventRecordID>
        <Channel>Operations Manager</Channel>
        <Computer>INFMGT52.accessgeneral.com</Computer>
        <Security />
      </System>
      <EventData>
        <Data>System.TypeInitializationException: The type initializer for ‘Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess’ threw an exception. —&gt; System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
       at System.Diagnostics.PerformanceCounter.Initialize()
       at System.Diagnostics.PerformanceCounter.set_RawValue(Int64 value)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager.CreateCounter(String categoryName, String counterName)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager.CreateCounters(PerformanceCounterCategoryDescriptor categoryInfo)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager..ctor(String instanceName, PerformanceCounterCategoryDescriptor categoryInfo)
       at Microsoft.EnterpriseManagement.Mom.Internal.SingleInstancePerformanceCounterCategoryDescriptor.CreateManager()
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkPerfCounterManager..ctor()
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess..cctor()
       — End of inner exception stack trace —
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess.Initialize()
       at Microsoft.EnterpriseManagement.Mom.Sdk.Service.SdkSubService.AsyncStartCallback(Object notUsed)
       at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)</Data>
      </EventData>
    </Event>

    After some research I found this article from Microsoft that contains instructions on how to fix this problem.

    Dashboarding using the SCCM Dashboard solution accelerator example

    January 5, 2011

    I will turn this into a more useful blog post when I have a few minutes but for now it is just a collection of links about dashboarding using the System Center Configuration Manager Solution Accelerator. The short story is that this runs on top of Sharepoint 3.0, which is free, meaning you can run this for free. It’s not specific to SCCM so you can dashboard any sql data you want! Cool!

    http://technet.microsoft.com/en-us/library/ff369719.aspx

    http://systemscentre.blogspot.com/search/label/Dashboard

    http://garyhay.blogspot.com/2010/07/sccm-dashboard-queries-1.html

    http://blogs.msdn.com/b/shitanshu/archive/2010/04/08/part-1-how-microsoft-it-using-configuration-manager-dashboard.aspx

    http://blogs.msdn.com/b/shitanshu/archive/2010/04/11/part-2-using-configuration-manager-dashboard-for-software-update-deployment-readiness-patch-tuesday-checklist.aspx

    http://blogs.msdn.com/b/shitanshu/archive/2010/05/02/part-3-sql-queries-used-for-creating-custom-configuration-manager-dashboard-in-microsoft-it.aspx

    http://blogs.msdn.com/b/shitanshu/archive/2010/10/09/part-4-sql-queries-for-creating-configuration-manager-client-health-and-problem-management-dashboard.aspx

    http://64.4.11.252/en-us/edge/system-center-configuration-manager-2007-dashboard-part-2-configuration.aspx?query=1

    SCOM Windows 2008 R2 Printer MP

    December 10, 2010

    Ran across this custom MP by Kevin Holman. Nice! Thank you!

    cleaning up the default management pack

    December 10, 2010

    SCOM has what I feel is a major bug in that it will allow you to save items (monitors, rules, overrides, etc.) in the default MP. Doing this is bad for a lot of reasons, and not only does SCOM allow you to do this, but it is the default option as well. In my case it turned out that an occasional lack of attention allowed me to do this and then removing MP’s later becomes a huge pain in the rear. Anyway I found this good article on how to clean up the mess.

    SCOM: powershell run space failed to start

    December 7, 2010

    I have been getting these messages since day 1 and tried various things that didn’t work to resolve.

    Below I am pasting an example rule with full text so that if someone is searching they will find it. This is one specific alert, I was having an issue with all non microsoft powershell scripted discoveries. For me this was 99% from the XSNMP SCOM Management Pack. To be clear the MP was not the cause of the problem, only the one that tried to run PS (and not work) the most.

    PM

    Log Name:
    Operations Manager

    Source:
    Health Service Modules

    Event Number:
    22400

    Level:
    1

    Logging Computer:

    User:
    N/A

    Description:

    Failed to run the PowerShell script due to exception below, this workflow will be unloaded. System.NullReferenceException: Object reference not set to an instance of an object. at System.Environment.GetEnvironmentVariable(String variable, EnvironmentVariableTarget target) at System.Management.Automation.ModuleIntrinsics.GetExpandedEnvironmentVariable(String name, EnvironmentVariableTarget target) at System.Management.Automation.ModuleIntrinsics.SetModulePath() at System.Management.Automation.ExecutionContext.InitializeCommon(AutomationEngine engine, PSHost hostInterface) at System.Management.Automation.AutomationEngine..ctor(PSHost hostInterface, RunspaceConfiguration runspaceConfiguration, InitialSessionState iss) at System.Management.Automation.Runspaces.LocalRunspace.DoOpenHelper() at System.Management.Automation.Runspaces.RunspaceBase.CoreOpen(Boolean syncCall) at Microsoft.EnterpriseManagement.Modules.PowerShell.RunspaceController.RunScript(String scriptName, String scriptBody, Dictionary`2 parameters, PowerShellOutputType outputType, Int32 serializationDepth, IModuleDebug iModuleDebug) at Microsoft.EnterpriseManagement.Modules.PowerShell.PowerShellProbeActionModule.RunScript(RunspaceController runspaceController) Script Name: MemoryPctUtil.ps1 One or more workflows were affected by this. Workflow name: xSNMP.Cisco.Rule.CollectMemoryPoolUtil Instance name: I/O Instance ID: {X} Management group: X

    Event Data:

    < DataItem type =" System.XmlData " time =" 2010-12-03T19:15:30.1742570-05:00 " sourceHealthServiceId =" X" >

    < EventData >

    < Data > X </ Data >

    < Data > xSNMP.Cisco.Rule.CollectMemoryPoolUtil </ Data >

    < Data > I/O </ Data >

    < Data > {X} </ Data >

    < Data > MemoryPctUtil.ps1 </ Data >

    < Data > 300 </ Data >

    < Data > System.NullReferenceException: Object reference not set to an instance of an object. at System.Environment.GetEnvironmentVariable(String variable, EnvironmentVariableTarget target) at System.Management.Automation.ModuleIntrinsics.GetExpandedEnvironmentVariable(String name, EnvironmentVariableTarget target) at System.Management.Automation.ModuleIntrinsics.SetModulePath() at System.Management.Automation.ExecutionContext.InitializeCommon(AutomationEngine engine, PSHost hostInterface) at System.Management.Automation.AutomationEngine..ctor(PSHost hostInterface, RunspaceConfiguration runspaceConfiguration, InitialSessionState iss) at System.Management.Automation.Runspaces.LocalRunspace.DoOpenHelper() at System.Management.Automation.Runspaces.RunspaceBase.CoreOpen(Boolean syncCall) at Microsoft.EnterpriseManagement.Modules.PowerShell.RunspaceController.RunScript(String scriptName, String scriptBody, Dictionary`2 parameters, PowerShellOutputType outputType, Int32 serializationDepth, IModuleDebug iModuleDebug) at Microsoft.EnterpriseManagement.Modules.PowerShell.PowerShellProbeActionModule.RunScript(RunspaceController runspaceController) </ Data >

    < Data />

    </ EventData >

    </ DataItem >

     

    Eventually the scripts would time out like this:

    Time window start
    12/7/2010 11:57:38 AM

    Time window end
    12/7/2010 12:02:37 PM

    Time First
    12/7/2010 11:57:38 AM

    Time Last
    12/7/2010 11:57:41 AM

    Count
    44

    Context

    Date and Time:
    12/7/2010 11:57:41 AM

    Log Name:
    Operations Manager

    Source:
    Health Service Modules

    Event Number:
    22411

    Level:
    1

    Logging Computer:
    X

    User:
    N/A

    Description:

    The PowerShell script will be dropped because the it has been waiting in the queue for more than 10 minutes. Script Name: DiscoverInterfaceName.ps1 One or more workflows were affected by this. Workflow name: xSNMP.Discovery.InterfaceName Instance name: GigabitEthernet2/21 Instance ID: {X} Management group: X

    Event Data:

    < DataItem type =" System.XmlData " time =" 2010-12-07T11:57:41.4054873-05:00 " sourceHealthServiceId =" X" >

    < EventData >

    < Data > X </ Data >

    < Data > xSNMP.Discovery.InterfaceName </ Data >

    < Data > GigabitEthernet2/21 </ Data >

    < Data > {X} </ Data >

    < Data > DiscoverInterfaceName.ps1 </ Data >

    < Data > 300 </ Data >

    < Data > 10 </ Data >

    < Data />

    </ EventData >

    </ DataItem >

     

    The issue turned out to be  a permissions issue with the health service since it was trying to run these powershell scripts as local system. This article is the one that finally jogged my memory appropriately. Thank you to the author!