Showing posts with label sql. Show all posts
Showing posts with label sql. Show all posts

Thursday, March 1, 2018

[Powershell + OMS HTTP Data Collection API] - Get & Send data from a MS-SQLServer to OMS Log Analytics

Earlier in my post about using the OMS Data Collector API, i promissed that i would post something about it but in a Powershell way of doing it - today is the day.
Somewhere in the past i had the need to put some data stored in a on-prem database to OMS so we could use it's own Analysis engine.
So, obviously, from where you run this, you need to have internet connection or configured HTTP proxy - just sayin'! :)

In my case i needed to have a timestamp stored for the last time i queried data so i don't have duplicated data in OMS, so analysis could be more accurate - this Powershell script version doesn't contain that logical part of code, it's more raw so you could understand how it's done and after it, you could obvisouly change it's logic as i did, or add some other logic.

Before we go to the script itself, let me just explain the part of how we store data and then convert it to JSON.

As you might see i store data from MSSQL in a 'System.Data.DataTable' (variable $table).
So the right to way to change it to JSON is making - $table | select $table.Columns.columnname | ConvertTo-Json
As so, you'll have data in the above and correct format :

{
"property1": "value1",
"property2": "value2"
"property3": "value3",
"property4": "value4"
}

Where "property1", "property2" and so on is your table column names. This ways you only have to concern about the query fields you want/like.

So, this is the powershell i came up with - it's been great to have something like this to do some magic.


# Database
$SQLServer = "YOUR_MSSQL_SERVER"
$SQLDBName = "YOUR_DATABASE"
$SqlQuery = "YOUR_QUERY"

#OMS
$customerId = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
$SharedKey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
$dataType = "YOUR_OMS_CUSTOM_LOG_FILE_NAME"
# Function to create the authorization signature
Function New-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
  $xHeaders = 'x-ms-date:' + $date
  $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource
  $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
  $keyBytes = [Convert]::FromBase64String($sharedKey)
  $sha256 = New-Object -TypeName System.Security.Cryptography.HMACSHA256
  $sha256.Key = $keyBytes
  $calculatedHash = $sha256.ComputeHash($bytesToHash)
  $encodedHash = [Convert]::ToBase64String($calculatedHash)
  $authorization = 'SharedKey {0}:{1}' -f $customerId, $encodedHash
  return $authorization
}
#Send data to OMS
Function Send-OMSData($customerId, $sharedKey, $body, $logType)
{
 $method = 'POST'
  $contentType = 'application/json'
  $resource = '/api/logs'
  $rfc1123date = [DateTime]::UtcNow.ToString('r')
  $contentLength = $body.Length
  $signature = New-Signature `
  -customerId $customerId `
  -sharedKey $sharedKey `
  -date $rfc1123date `
  -contentLength $contentLength `
  -fileName $fileName `
  -method $method `
  -contentType $contentType `
  -resource $resource
  $uri = 'https://' + $customerId + '.ods.opinsights.azure.com' + $resource + '?api-version=2016-04-01'
  $headers = @{
    'Authorization'      = $signature
    'Log-Type'           = $logType
    'x-ms-date'          = $rfc1123date
  }

  $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
  return $response.StatusCode
}

Try {
    $connString = "Data Source=$SQLServer;Initial Catalog=$SQLDBName;user=YOUR_SQL_SERVER_USERNAME;password=YOUR_SQL_SERVER_PASSWORD" 
    $connection = New-Object System.Data.SqlClient.SqlConnection($connString)
    $connection.Open()
    $sqlcmd = $connection.CreateCommand()
    $sqlcmd.CommandText = $SqlQuery
    $results = $sqlcmd.ExecuteReader()
    $table = new-object “System.Data.DataTable”
    $table.Load($results)
    $connection.Close()
}

Catch [Exception] {
Write-Host $_.exception.message
    exit
}
$payload = $table | select $table.Columns.columnname | ConvertTo-Json
#send Data to OMS
Send-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($payload)) -logType $dataType


Hope you all enjoy and give it some use.

Cheers,

Thursday, July 14, 2016

SCCM (ConfigMgr) - IIS Inventor with VBScript and WMI Classes

Recently we had this need for a customer of ours - Make an IIS Inventory report in SCCM with all the sites, related application pools, bindings and IIS versions.

It seems easy, but in a few moments it turned into a great nightmare, still a great challange - that i accepted gladly!

The first thing i've banged into was the fact of having multiple IIS versions and Operating Systems (IIS 6, 7, 8 - and 2003, 2008, 2012).
Why ?
Because for IIS version 6 you get information from "ROOT\MicrosoftIISv2" namespace, and for IIS>7 you have "ROOT\WebAdministration" namespace.
The problem wasn't having multiple data sources from where you could collect data from - i'll explain it further!
There's a solution, a really easy one to overcome this issue - install IIS WMI 6 Compatability role on your IIS>7 - this will make/create the "ROOT\MicrosoftIISv2" namespace even on IIS>7 machines - this way you'll only have a datasource to 'drink' data from.
But, there're some security issues, that some sysadmins don't like about this role, so i couldn't go there!

But, right before i set the classes i wanted, i created a simple collection with this WQL (All devices with IIS installed) :

 select SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System inner join SMS_G_System_SERVICE on SMS_G_System_SERVICE.ResourceID = SMS_R_System.ResourceId where SMS_G_System_SERVICE.Name = 'W3SVC'  

And right after, a custom agent setting where only there i would enable the further classes and the inventory of "C:\Windows\System32\InetSrv" folder so i could inventory inetmgr.exe file so i could've exact IIS version foreach machine.

Now, WMI Classes setup!
Ok! So i checked which classes i was going to set up and read information from, and came up with this :

IIS6 or with IIS 6 WMI Compatability :

 Namespace : MicrosoftIISv2  
 Class : IIsWebVirtualDirSetting  
 Query : SELECT * FROM IIsWebVirtualDirSetting


 Namespace : MicrosoftIISv2  
 Class : IIsWebServerSetting  
 Query : SELECT * FROM IIsWebServerSetting  

For IIS>7 :

 Namespace : WebAdministration  
 Class : Application  
 Query : SELECT * FROM Application

 Namespace : WebAdministration  
 Class : Site  
 Query : SELECT * FROM Site      

But right there, feeling really lucky about how it was going, and banged into my first issue!
At (ROOT\WebAdministration) Application class! You can't enable it because there's already a SCCM built-in class with this name.
So, after some googling i've learned that i could make a UNION class that "mirrors" all the information from a source into this new class in a namespace i wanted - and came up with this code :

 #pragma namespace("\\\\.\\root\\cimv2")  
 [Union,ViewSources{"select ApplicationPool,EnabledProtocols,ServiceAutoStartEnabled,ServiceAutoStartProvider,Path,PreloadEnabled,SiteName from Application"},ViewSpaces{"\\\\.\\root\\webadministration"},dynamic,Provider("MS_VIEW_INSTANCE_PROVIDER")]  
 class IIS_Application  
 {  
     [PropertySources{"ApplicationPool"}] string ApplicationPool;  
     [PropertySources{"EnabledProtocols"}] string EnabledProtocols;  
     [PropertySources{"ServiceAutoStartEnabled"}] boolean ServiceAutoStartEnabled;  
     [PropertySources{"ServiceAutoStartProvider"}] string ServiceAutoStartProvider;  
     [PropertySources{"Path"},key] string Path;  
     [PropertySources{"PreloadEnabled"}] boolean PreloadEnabled;  
     [PropertySources{"SiteName"},key] string SiteName;  
 };  

A Free-Tip : You must have all the KEY propreties mapped in your new class, meaning that if your source class has 3 key properties, your new custom class must also have those 3 key properties.

Basically it's kind of a view of "ROOT\WebAdministration\Application" class that i created in "ROOT\CIMV2" and named it "IIS_Application".

There was another problem i faced - SCCM hardware inventory can't read WMI Classes proprieties when they are objects, meaning that i've tried to set "ROOT\WebAdministration\Site" for having sites information with binding association and i couldn't.
So, again, i needed to make a brand new class :

 #pragma namespace ("\\\\.\\root\\cimv2")  
 class IIS_Bindings  
 {  
 [key]  
 STRING SiteName;   
 STRING Bindings;  
 UInt32 SiteId;  
 };  

But there's a problem - this is not a view! This is a simple new class - with no data, just empty! And the only way (i know) to populate this, was with this script :

 strComputer = "."  
     Set objWMIService = GetObject("winmgmts:{authenticationLevel=pktPrivacy}\\" & strComputer & "\root\cimv2")  
     Set colBindingsItems = objWMIService.ExecQuery("Select * from IIS_Bindings")  
     For Each objItem in colBindingsItems  
             objItem.Delete_()  
     Next  
     Set objWMIService = GetObject("winmgmts:{authenticationLevel=pktPrivacy}\\" & strComputer & "\root\WebAdministration")  
     Set colItems = objWMIService.ExecQuery("Select * from Site")  
     Set oWMI = GetObject("winmgmts:root\cimv2")  
     Set oData = oWMI.Get("IIS_Bindings")  
     Set oInstance = oData.SpawnInstance_  
     For Each objItem in colItems  
             oInstance.SiteName = objItem.Name  
             oInstance.SiteId = objItem.Id  
             Bindings = ""  
             For i = 0 to Ubound(objItem.Bindings)  
         Bindings = Bindings + objItem.Bindings(i).BindingInformation + "|"  
             Next  
             Bindings = LEFT(Bindings, (Len(Bindings) -1))  
             oInstance.Bindings = Bindings  
             oInstance.Put_()  
     Next  

Yes, you need to do a new based script application and deploy it into your collection - it will take a wile so you get data back into your CM database.

So now, that we've done all the setup we need to have all IIS information - in my case "Machine;Site;ApplicationPool;Path;Bindings;IISVersion" - it's time to query!
For every class you enable, SCCM will create a table like "MY_CLASSNAME_DATA", for example : IIS_Application_DATA
And it's here where all the magic can be done.

So, finally, i came up with this query (It needs to be modified to return better results, specially when it comes to the bindings ... i'll do it later someday!)

 SELECT DISTINCT  
     CSYS.Name0 as [Server Name],  
     IISApp.ApplicationPool00 as [AppPools],    
     IISApp.SiteName00 as [Site Name],  
     IISBind.Bindings00 as [Bindings],  
     SUBSTRING(SF.FileVersion, 1,14) AS [IIS Version]  
 FROM   
     [V_R_system] SYS with (nolock)  
     JOIN [v_GS_COMPUTER_SYSTEM] CSYS on CSYS.ResourceID = sys.ResourceID  
     JOIN [v_FullCollectionMembership] FCM on FCM.ResourceID = CSYS.ResourceID  
     JOIN [v_GS_SoftwareFile] SF on SF.ResourceID = SYS.ResourceID  
     FULL JOIN [IIS_Application_DATA] IISApp on IISApp.MachineID = SYS.ResourceID  
     FULL JOIN [IIS_Bindings_DATA] IISBind on IISBind.MachineID = SYS.ResourceID  
 WHERE  
     FCM.CollectionID like 'YOU_Collection_ID_Goes_Here'  
 AND   
     IISApp.SiteName00 = IISBind.SiteName00  
 AND   
     SF.FileName like '%inetmgr%'  

NOTE: This is the first version of the project, a POC if you want, just to show you how to get data, perhpaps it has some flaws - but it's a way of showing how to get data specially if you don't have it in the first place.



Hope this might be useful to you in someway.

Cheers,

Monday, May 23, 2016

OpsMgr (SCOM) - Solve the "An Item With The Same Key Has Already Been Added" error


For some reasons, like, having a bunch of procedures that include Unix/Linux servers into SCOM, you might get the 'An Item With The Same Key Has Already Been Added' error when you go to Administration - Unix/Linux Agents.

That means that you've duplicate unix/linux servers in your SCOM installation.

The only way to solve this is to find the duplicate entry, and delete them both.

To do that, you can't use the Get-SCXAgent, this will output unique values only - the only way to solve this is going to your OpsDB execute a querry (bellow), and foreach value you get, you need to run the 'get-scxagent "duplicate_value" | Remove-ScxAgent' cmdlet.

So, first things first.

The query :

 DECLARE @ClassName NVARCHAR(256)   
 DECLARE @CManagedTypeId UNIQUEIDENTIFIER   
 SET @ClassName = 'Microsoft.Unix.OperatingSystem'  
 SET @CManagedTypeId = (   
     SELECT ManagedTypeId  
     FROM ManagedType   
     WHERE TypeName = @ClassName )   
 SELECT  
     [ManagedEntityGenericView].[Id],   
     [ManagedEntityGenericView].[Name],   
     [ManagedEntityGenericView].[Path],   
     [ManagedEntityGenericView].[FullName],   
     [ManagedEntityGenericView].[LastModified],   
     [ManagedEntityGenericView].[TypedManagedEntityId],   
     NULL AS SourceEntityId   
 FROM  
     dbo.ManagedEntityGenericView   
 INNER JOIN (      
     SELECT DISTINCT [BaseManagedEntityId]   
     FROM dbo.[TypedManagedEntity] TME WITH(NOLOCK)   
     JOIN [dbo].[DerivedManagedTypes] DT ON DT.[DerivedTypeId] = TME.[ManagedTypeId]   
     WHERE  
         DT.[BaseTypeId] = @CManagedTypeId  
         AND TME.IsDeleted = 0 )  
 AS ManagedTypeIdForManagedEntitiesByManagedTypeAndDerived   
 ON ManagedTypeIdForManagedEntitiesByManagedTypeAndDerived.[BaseManagedEntityId] = [Id]   
 WHERE  
     [IsDeleted] = 0 AND  
     [TypedMonitoringObjectIsDeleted] = 0 AND  
     [ManagedEntityGenericView].[Path] IN (   
                                             SELECT [BaseManagedEntity].[Path]   
                                             FROM [BaseManagedEntity]   
                                             GROUP BY [BaseManagedEntity].[Path]   
                                             HAVING COUNT([BaseManagedEntity].[Path]) > 1   
                                          )  
 GROUP BY [ManagedEntityGenericView].[Id],   
     [ManagedEntityGenericView].[Name],   
     [ManagedEntityGenericView].[Path],   
     [ManagedEntityGenericView].[FullName],   
     [ManagedEntityGenericView].[LastModified],   
     [ManagedEntityGenericView].[TypedManagedEntityId]  
 HAVING COUNT([ManagedEntityGenericView].[Path]) > 1  

Now that you've the duplicate values to delete, just open a powershell prompt and run the follow cmdlet foreach duplicate value you've.

 get-scxagent "Your_Server" | Remove-ScxAgent  

And, your problem is solved.

Cheers,

OpsMgr (SCOM) - BUILTIN\Administrators

Well, imagine that for some reason you delete the "BUILTIN\Administrators" group in OpsMgr Administrator Role, you might have some issues if you don't have all the profiles and roles correctly assinged.

There's a workaround, not supported by MSFT, but, well ... :)

Into OpsDB, run this query.

 insert into AzMan_Role_SIDMember ([RoleID],[MemberSID])  
 VALUES (1, 0x01020000000000052000000020020000) -- This is the hex value for BUILTIN\Administrators  

And off you go!

Cheers,

Friday, May 20, 2016

OpsMgr (SCOM) - Ghost Agents

They show up in Monitoring views, but not in Administration ?
Well ... no problem!

At OperationsManager Database :

 -- #1  
 SELECT * FROM dbo.[BaseManagedEntity] where FullName Like '%Windows.Computer%' and Name = 'your_host_here'  
 -- #2  
 UPDATE dbo.[BaseManagedEntity]  
 SET IsDeleted = 1   
 WHERE FullName Like '%Windows.Computer%' and Name = 'your_host_here'  

Hope it helps you out.

Cheers,