Thursday, March 1, 2018

[Powershell + OMS HTTP Data Collection API] - Get & Send data from a MS-SQLServer to OMS Log Analytics

Earlier in my post about using the OMS Data Collector API, i promissed that i would post something about it but in a Powershell way of doing it - today is the day.
Somewhere in the past i had the need to put some data stored in a on-prem database to OMS so we could use it's own Analysis engine.
So, obviously, from where you run this, you need to have internet connection or configured HTTP proxy - just sayin'! :)

In my case i needed to have a timestamp stored for the last time i queried data so i don't have duplicated data in OMS, so analysis could be more accurate - this Powershell script version doesn't contain that logical part of code, it's more raw so you could understand how it's done and after it, you could obvisouly change it's logic as i did, or add some other logic.

Before we go to the script itself, let me just explain the part of how we store data and then convert it to JSON.

As you might see i store data from MSSQL in a 'System.Data.DataTable' (variable $table).
So the right to way to change it to JSON is making - $table | select $table.Columns.columnname | ConvertTo-Json
As so, you'll have data in the above and correct format :

{
"property1": "value1",
"property2": "value2"
"property3": "value3",
"property4": "value4"
}

Where "property1", "property2" and so on is your table column names. This ways you only have to concern about the query fields you want/like.

So, this is the powershell i came up with - it's been great to have something like this to do some magic.


# Database
$SQLServer = "YOUR_MSSQL_SERVER"
$SQLDBName = "YOUR_DATABASE"
$SqlQuery = "YOUR_QUERY"

#OMS
$customerId = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
$SharedKey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
$dataType = "YOUR_OMS_CUSTOM_LOG_FILE_NAME"
# Function to create the authorization signature
Function New-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
  $xHeaders = 'x-ms-date:' + $date
  $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource
  $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
  $keyBytes = [Convert]::FromBase64String($sharedKey)
  $sha256 = New-Object -TypeName System.Security.Cryptography.HMACSHA256
  $sha256.Key = $keyBytes
  $calculatedHash = $sha256.ComputeHash($bytesToHash)
  $encodedHash = [Convert]::ToBase64String($calculatedHash)
  $authorization = 'SharedKey {0}:{1}' -f $customerId, $encodedHash
  return $authorization
}
#Send data to OMS
Function Send-OMSData($customerId, $sharedKey, $body, $logType)
{
 $method = 'POST'
  $contentType = 'application/json'
  $resource = '/api/logs'
  $rfc1123date = [DateTime]::UtcNow.ToString('r')
  $contentLength = $body.Length
  $signature = New-Signature `
  -customerId $customerId `
  -sharedKey $sharedKey `
  -date $rfc1123date `
  -contentLength $contentLength `
  -fileName $fileName `
  -method $method `
  -contentType $contentType `
  -resource $resource
  $uri = 'https://' + $customerId + '.ods.opinsights.azure.com' + $resource + '?api-version=2016-04-01'
  $headers = @{
    'Authorization'      = $signature
    'Log-Type'           = $logType
    'x-ms-date'          = $rfc1123date
  }

  $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
  return $response.StatusCode
}

Try {
    $connString = "Data Source=$SQLServer;Initial Catalog=$SQLDBName;user=YOUR_SQL_SERVER_USERNAME;password=YOUR_SQL_SERVER_PASSWORD" 
    $connection = New-Object System.Data.SqlClient.SqlConnection($connString)
    $connection.Open()
    $sqlcmd = $connection.CreateCommand()
    $sqlcmd.CommandText = $SqlQuery
    $results = $sqlcmd.ExecuteReader()
    $table = new-object “System.Data.DataTable”
    $table.Load($results)
    $connection.Close()
}

Catch [Exception] {
Write-Host $_.exception.message
    exit
}
$payload = $table | select $table.Columns.columnname | ConvertTo-Json
#send Data to OMS
Send-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($payload)) -logType $dataType


Hope you all enjoy and give it some use.

Cheers,

[Check_MK + OMS Log Analytics] - Using Log Analytics Data Collector API

So, it's been a while since i wrote something here.
Not that i've lots of visualizations, but as i said before, i use this not only as a knowlodge share 'platform' and an hobbie, but also as my notebook :)

I've been self challanged to send or integrate if you like, some Check_MK alerts with OMS Log Analytics.

So, i've decided to give it a shot.

The scenario was to send all Check_MK alerts to OMS Log Analytics and later make some analysis about that, and other data that i'm already sending there (i'll make a post about it later - something about Powershell + MSSQL data -> Azure) - Perhaps tomorow i'll post it here.

So - Check_MK has a notifications system almost like Operations Manager.
It's very flexible about what you want to notificate and whom.

You might want to read it a little more here:

http://mathias-kettner.com/checkmk_flexible_notifications.html

After googling a bit, a came across with an article about posting data using the HTTP Data Collector API, where it has a bunch of examples about how to post data to OMS - check it here :

https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api

Resuming a little bit, basically your input data needs to be like this :

{
"property1": "value1",
" property 2": "value2"
" property 3": "value3",
" property 4": "value4"
}

So, after knowing this, this is the Check_MK notification script i came up with (Python):



#!/usr/bin/env python
import os, sys, time, subprocess, json, requests, datetime, hashlib, hmac, base64

# HTTP/HTTPS Proxies - If you need some ... #

os.environ["HTTP_PROXY"] = 'http://:/'
os.environ["HTTPS_PROXY"] = 'https://:/'

# OMS Related Variables #
customer_id = 'xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
shared_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
log_type = 'Your_Custom_Log_Name'

# OMS Related Functions #

# Build the API signature
def build_signature(customer_id, shared_key, date, content_length, method, content_type, resource):
    x_headers = 'x-ms-date:' + date
    string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
    bytes_to_hash = bytes(string_to_hash).encode('utf-8')
    decoded_key = base64.b64decode(shared_key)
    encoded_hash = base64.b64encode(hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest())
    authorization = "SharedKey {}:{}".format(customer_id,encoded_hash)
    return authorization

# Build and send a request to the POST API
def post_data(customer_id, shared_key, body, log_type):
    method = 'POST'
    content_type = 'application/json'
    resource = '/api/logs'
    rfc1123date = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
    content_length = len(body)
    signature = build_signature(customer_id, shared_key, rfc1123date, content_length, method, content_type, resource)
    uri = 'https://' + customer_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'

    headers = {
        'content-type': content_type,
        'Authorization': signature,
        'Log-Type': log_type,
        'x-ms-date': rfc1123date
    }

    response = requests.post(uri,data=body, headers=headers)
    if (response.status_code >= 200 and response.status_code <= 299):
        print 'Accepted'
    else:
        print "Response code: {}".format(response.status_code)

# MK Alert Handling #

MK_HOSTNAME = os.environ['NOTIFY_HOSTNAME']
MK_DESCRIPTION = os.environ['NOTIFY_SERVICEOUTPUT']
MK_PLUGIN_OUTPUT = os.environ['NOTIFY_SERVICEDESC']
MK_LAST_STATE_CHANGE = time.strftime("%Y-%m-%d %H:%M:%S")
MK_SERVICESTATE = os.environ['NOTIFY_SERVICESTATE']

MK_HOST_IP = NEW_INFO.split(",")[0]
MK_DATACENTER = NEW_INFO.split(",")[1]
MK_TEAM = NEW_INFO.split(",")[2]
MK_VENDOR = NEW_INFO.split(",")[3].replace('\n','')

MK_JSON_OUTPUT = json.dumps([{'mk_hostname': MK_HOSTNAME, 'mk_description': MK_DESCRIPTION, 'mk_plugin_output': MK_PLUGIN_OUTPUT, 'mk_last_state_chage': MK_LAST_STATE_CHANGE, 'mk_servicestate': MK_SERVICESTATE }])

post_data(customer_id, shared_key, MK_JSON_OUTPUT, log_type)

After yout get this as an active notification method on your Check_MK site, you'll start to see data getting in OMS like :



This is really nice to have - the OMS analytics engine is awesome and gives you a lot of possibilities.

Obiously, you might want to use the above example on anything you want to, so, your imagination could simply adapt this to your own case scenario.

Hope you all like this thread, and i promise to post things here more frequently.

Cheers,

Wednesday, May 24, 2017

SCCM (ConfigMrg) - WannaCry Ransomware Compliance

WannaCry Ransomware made some damages worldwide, and still lots of doubts about how to check if your infrastructure is safe.
This is been my days lately.

Check if my whole servers are patched, giving management teams compliance reports, and so on.
Lots of online examples, this, yes, is just another one.

Disclaimer: Modify the T-SQL query and VBScript for the specific HotFixID's - in my case are for 2003-2016 windows servers.

So, SCCM query to check out the servers missing the Ransomware fix :


SELECT dbo.v_R_System.Name0 AS 'Computername', v_R_System.Full_Domain_Name0, dbo.v_UpdateInfo.Title AS 'Updatename', dbo.v_StateNames.StateName, dbo.v_Update_ComplianceStatusAll.LastStatusCheckTime, dbo.v_UpdateInfo.DateLastModified, dbo.v_UpdateInfo.IsDeployed, dbo.v_UpdateInfo.IsSuperseded,   
      dbo.v_UpdateInfo.IsExpired, dbo.v_UpdateInfo.BulletinID, dbo.v_UpdateInfo.ArticleID, dbo.v_UpdateInfo.DateRevised,   
      catinfo.CategoryInstanceName as 'Vendor',   
      catinfo2.CategoryInstanceName as 'UpdateClassification'   
      FROM dbo.v_StateNames   
      INNER JOIN dbo.v_Update_ComplianceStatusAll   
      INNER JOIN dbo.v_R_System ON dbo.v_R_System.ResourceID = dbo.v_Update_ComplianceStatusAll.ResourceID   
      INNER JOIN dbo.v_UpdateInfo ON dbo.v_UpdateInfo.CI_ID = dbo.v_Update_ComplianceStatusAll.CI_ID ON dbo.v_StateNames.StateID = dbo.v_Update_ComplianceStatusAll.Status   
      INNER JOIN v_CICategories_All catall on catall.CI_ID = dbo.v_UpdateInfo.CI_ID   
      INNER JOIN v_CategoryInfo catinfo on catall.CategoryInstance_UniqueID = catinfo.CategoryInstance_UniqueID and catinfo.CategoryTypeName='Company'   
      INNER JOIN v_CICategories_All catall2 on catall2.CI_ID=dbo.v_UpdateInfo.CI_ID   
      INNER JOIN v_CategoryInfo catinfo2 on catall2.CategoryInstance_UniqueID = catinfo2.CategoryInstance_UniqueID and catinfo2.CategoryTypeName='UpdateClassification'   
      INNER JOIN v_CH_ClientSummary on v_CH_ClientSummary.ResourceID = v_R_System.ResourceID  
      WHERE (dbo.v_StateNames.TopicType = 500)   
      AND (dbo.v_StateNames.StateName = 'Update is required')   
      AND (dbo.v_R_System.Name0 IN (  
                                     SELECT TOP (100) PERCENT SD.Name0 AS 'Machine Name'   
                                     FROM dbo.v_R_System AS SD INNER JOIN   
                                     dbo.v_FullCollectionMembership AS FCM ON SD.ResourceID = FCM.ResourceID INNER JOIN   
                                     dbo.v_Collection AS COL ON FCM.CollectionID = COL.CollectionID LEFT OUTER JOIN   
                                     dbo.v_R_User AS USR ON SD.User_Name0 = USR.User_Name0 INNER JOIN   
                                     dbo.v_GS_PC_BIOS AS PCB ON SD.ResourceID = PCB.ResourceID INNER JOIN   
                                     dbo.v_GS_COMPUTER_SYSTEM AS CS ON SD.ResourceID = CS.ResourceID INNER JOIN   
                                     dbo.v_RA_System_SMSAssignedSites AS SAS ON SD.ResourceID = SAS.ResourceID   
                                     WHERE (COL.Name like 'All Windows Servers')  
                                   )  
          )   
      AND ((catinfo2.CategoryInstanceName like 'Critical%' ) OR (catinfo2.CategoryInstanceName like 'Security%' ))   
      AND dbo.v_UpdateInfo.ArticleID in ('4012214','4012212','4012213','4012598')  
      AND v_CH_ClientSummary.ClientActiveStatus = 1  
      ORDER BY dbo.v_R_System.Name0  

And, a SCCM configuration item VBScript (some servers don't have powershell...! yes, there're a few ...!) :

 strComputer = "."  
 Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")  
 Set colItems = objWMIService.ExecQuery("Select * from Win32_QuickFixEngineering",,48)  
 Set objFSO=CreateObject("Scripting.FileSystemObject")  
 Set wshNetwork = WScript.CreateObject( "WScript.Network" )  
 strComputerName = wshNetwork.ComputerName  
 For Each objItem in colItems   
 If objItem.HotfixID = "KB4012214" or objItem.HotfixID = "KB4012213" or objItem.HotfixID = "KB4012212" or objItem.HotfixID = "KB4012598" then  
 wscript.echo "Compliant"  
 End If  
 Next  

So, now just make a configuration baseline or add this configuration item to your existing configuration baseline.

Hope this helps you guys out.

Thursday, November 17, 2016

OpsMgr (SCOM) - Maintenance Web-Portal (Beta version is here!)

As mentioned here I've decided to develop a web based interface to manage scheduled maintenance mode for SCOM Objects.

And finally, i came up with a first beta version of it.
I promise that as soon as possible I'll be releasing it here - i'm just cleaning up code and giving a more exhaust testing before i give this to you all.
I hope until next week I'm making this available.

So far, the specifications/functionalities of it are:

Search objects of classes Windows and Unix/Linux servers;
Set two types of maintenance mode :
- From now until 'x' minutes;
- From a specific date until 'x' minutes;
Check for the latest maintenance history of searched objects (agents);

1 ) Searching for agents/objects

It shows you the result and give you the possibility to Manage (Set-up maintenance mode for searched object) and to check on maintenance history (History button).



2 ) Manage button to setup Maintenance

Here you can either setup a 'right-now' maintenance or scheduled it in the future.
I'm using this Orchestrator Runbook solution I've developed a few months ago to handle future maintenance.



This is the the way you select your future start-time



Fill the information and Submit it!



You will get different outputs on success or failure!



And ... the result :




As you can see ...



3) History button

Since you click it you'll be given a pop-up with the latest maintenance information like bellow



And this is it! I hope you enjoy the idea and what came out of it, hope you could give me any suggestions and feedback about it.

Now what i would like to do in the future of it.

Further specifications/functionalities i would like to develop:

Windows authentication instead of SDK user in web.config file;
- At this point it's much easier to work like this, but it's of course more reliable to only have access to the objects you manage;
Edit maintenance mode:
- Able to stop current maintenance mode;
- Able to increase the end time of current maintenance mode;
Give the end-user the ability to choose specific classes of objects (SQL Databases, Clusters ...);
On main page give some stats about SCOM and SCOMDW performance in graphs;
(Any other feature I'm missing and you could give me the idea instead!)

Cheers!

Tuesday, November 15, 2016

[Update II] OpsMgr (SCOM) - Operations Manager Maintenance Web-Portal

Keeping this up to date.

Started to do this :


From now on, i'll be developing the scheduled part of the solution - as mentioned before, this will integrate with my previous solution (Orchestrator Runbook and a simple SQL Database)

I've already studied on how i'm passing the windows authentication and will be on my further to-do list.

Cheers,

[Update] OpsMgr (SCOM) - Operations Manager Maintenance Web-Portal

Since last post about the Operations Manager Maintenance Web-Portal, I've been editing code so i could publish the solution, and getting new functions to it.

This is getting a beta version! :)

So far:
- Added the SDK user credentials to Web.Config file;
- Created the Object Maintenance Mode History functionality;
- You can now search objects of classes instead of agents - still figuring out the best way to give user the possibility to choose or to limit class scope;

History button :



Details view



To do :
- User scope - still didn't think about it!
- Future Maintenance (I'll use this Orchestrator Runbook to do the job later!) - the layout is done, code is missing!


Friday, November 11, 2016

OpsMgr (SCOM) - Operations Manager Maintenance Web-Portal

I'll be introducing this post with a disclaimer - I'm not a programmer! Still, i'm an enthusiast about learning and have crazy ideas to occupy my time! :)

I've decided to develop a web-portal (asp/c#) so we could use a web page to put our servers into maintenance mode without having the need to open Operations Manager console - I'll be enhancing this project with other "tasks" that could fit in it - any suggestions, feel free to share!

So ... I came up with this :

1 ) Searching for agent



2 ) Click Manage to setup Maintenance



The future is already designed (missing back code to handle!)



3 ) Settings and Submit!



4) You will get different outputs on success or failure!



And ... the result :




As you can see ...





Further things to do on this project :
- Have a login page so we could have different server scopes - different users see/manage their own servers;
- Future Maintenance (I'll use this Orchestrator Runbook to do the job later!) - the layout is done, code is missing!
- Search for other classes, instead of agents (I'm evaluating the best way to do it!)
- I'll provide the solution to everyone, just finishing to edit and put the code better!

This is a Alpha version, but, as soon as possible i'll be releasing a beta version so you could test on your own environment - until that any suggestions are very welcome!

Hope you enjoy!