Thursday, March 1, 2018

[Powershell + OMS HTTP Data Collection API] - Get & Send data from a MS-SQLServer to OMS Log Analytics

Earlier in my post about using the OMS Data Collector API, i promissed that i would post something about it but in a Powershell way of doing it - today is the day.
Somewhere in the past i had the need to put some data stored in a on-prem database to OMS so we could use it's own Analysis engine.
So, obviously, from where you run this, you need to have internet connection or configured HTTP proxy - just sayin'! :)

In my case i needed to have a timestamp stored for the last time i queried data so i don't have duplicated data in OMS, so analysis could be more accurate - this Powershell script version doesn't contain that logical part of code, it's more raw so you could understand how it's done and after it, you could obvisouly change it's logic as i did, or add some other logic.

Before we go to the script itself, let me just explain the part of how we store data and then convert it to JSON.

As you might see i store data from MSSQL in a 'System.Data.DataTable' (variable $table).
So the right to way to change it to JSON is making - $table | select $table.Columns.columnname | ConvertTo-Json
As so, you'll have data in the above and correct format :

{
"property1": "value1",
"property2": "value2"
"property3": "value3",
"property4": "value4"
}

Where "property1", "property2" and so on is your table column names. This ways you only have to concern about the query fields you want/like.

So, this is the powershell i came up with - it's been great to have something like this to do some magic.


# Database
$SQLServer = "YOUR_MSSQL_SERVER"
$SQLDBName = "YOUR_DATABASE"
$SqlQuery = "YOUR_QUERY"

#OMS
$customerId = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
$SharedKey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
$dataType = "YOUR_OMS_CUSTOM_LOG_FILE_NAME"
# Function to create the authorization signature
Function New-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
  $xHeaders = 'x-ms-date:' + $date
  $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource
  $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
  $keyBytes = [Convert]::FromBase64String($sharedKey)
  $sha256 = New-Object -TypeName System.Security.Cryptography.HMACSHA256
  $sha256.Key = $keyBytes
  $calculatedHash = $sha256.ComputeHash($bytesToHash)
  $encodedHash = [Convert]::ToBase64String($calculatedHash)
  $authorization = 'SharedKey {0}:{1}' -f $customerId, $encodedHash
  return $authorization
}
#Send data to OMS
Function Send-OMSData($customerId, $sharedKey, $body, $logType)
{
 $method = 'POST'
  $contentType = 'application/json'
  $resource = '/api/logs'
  $rfc1123date = [DateTime]::UtcNow.ToString('r')
  $contentLength = $body.Length
  $signature = New-Signature `
  -customerId $customerId `
  -sharedKey $sharedKey `
  -date $rfc1123date `
  -contentLength $contentLength `
  -fileName $fileName `
  -method $method `
  -contentType $contentType `
  -resource $resource
  $uri = 'https://' + $customerId + '.ods.opinsights.azure.com' + $resource + '?api-version=2016-04-01'
  $headers = @{
    'Authorization'      = $signature
    'Log-Type'           = $logType
    'x-ms-date'          = $rfc1123date
  }

  $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
  return $response.StatusCode
}

Try {
    $connString = "Data Source=$SQLServer;Initial Catalog=$SQLDBName;user=YOUR_SQL_SERVER_USERNAME;password=YOUR_SQL_SERVER_PASSWORD" 
    $connection = New-Object System.Data.SqlClient.SqlConnection($connString)
    $connection.Open()
    $sqlcmd = $connection.CreateCommand()
    $sqlcmd.CommandText = $SqlQuery
    $results = $sqlcmd.ExecuteReader()
    $table = new-object “System.Data.DataTable”
    $table.Load($results)
    $connection.Close()
}

Catch [Exception] {
Write-Host $_.exception.message
    exit
}
$payload = $table | select $table.Columns.columnname | ConvertTo-Json
#send Data to OMS
Send-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($payload)) -logType $dataType


Hope you all enjoy and give it some use.

Cheers,

[Check_MK + OMS Log Analytics] - Using Log Analytics Data Collector API

So, it's been a while since i wrote something here.
Not that i've lots of visualizations, but as i said before, i use this not only as a knowlodge share 'platform' and an hobbie, but also as my notebook :)

I've been self challanged to send or integrate if you like, some Check_MK alerts with OMS Log Analytics.

So, i've decided to give it a shot.

The scenario was to send all Check_MK alerts to OMS Log Analytics and later make some analysis about that, and other data that i'm already sending there (i'll make a post about it later - something about Powershell + MSSQL data -> Azure) - Perhaps tomorow i'll post it here.

So - Check_MK has a notifications system almost like Operations Manager.
It's very flexible about what you want to notificate and whom.

You might want to read it a little more here:

http://mathias-kettner.com/checkmk_flexible_notifications.html

After googling a bit, a came across with an article about posting data using the HTTP Data Collector API, where it has a bunch of examples about how to post data to OMS - check it here :

https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api

Resuming a little bit, basically your input data needs to be like this :

{
"property1": "value1",
" property 2": "value2"
" property 3": "value3",
" property 4": "value4"
}

So, after knowing this, this is the Check_MK notification script i came up with (Python):



#!/usr/bin/env python
import os, sys, time, subprocess, json, requests, datetime, hashlib, hmac, base64

# HTTP/HTTPS Proxies - If you need some ... #

os.environ["HTTP_PROXY"] = 'http://:/'
os.environ["HTTPS_PROXY"] = 'https://:/'

# OMS Related Variables #
customer_id = 'xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
shared_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
log_type = 'Your_Custom_Log_Name'

# OMS Related Functions #

# Build the API signature
def build_signature(customer_id, shared_key, date, content_length, method, content_type, resource):
    x_headers = 'x-ms-date:' + date
    string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
    bytes_to_hash = bytes(string_to_hash).encode('utf-8')
    decoded_key = base64.b64decode(shared_key)
    encoded_hash = base64.b64encode(hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest())
    authorization = "SharedKey {}:{}".format(customer_id,encoded_hash)
    return authorization

# Build and send a request to the POST API
def post_data(customer_id, shared_key, body, log_type):
    method = 'POST'
    content_type = 'application/json'
    resource = '/api/logs'
    rfc1123date = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
    content_length = len(body)
    signature = build_signature(customer_id, shared_key, rfc1123date, content_length, method, content_type, resource)
    uri = 'https://' + customer_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'

    headers = {
        'content-type': content_type,
        'Authorization': signature,
        'Log-Type': log_type,
        'x-ms-date': rfc1123date
    }

    response = requests.post(uri,data=body, headers=headers)
    if (response.status_code >= 200 and response.status_code <= 299):
        print 'Accepted'
    else:
        print "Response code: {}".format(response.status_code)

# MK Alert Handling #

MK_HOSTNAME = os.environ['NOTIFY_HOSTNAME']
MK_DESCRIPTION = os.environ['NOTIFY_SERVICEOUTPUT']
MK_PLUGIN_OUTPUT = os.environ['NOTIFY_SERVICEDESC']
MK_LAST_STATE_CHANGE = time.strftime("%Y-%m-%d %H:%M:%S")
MK_SERVICESTATE = os.environ['NOTIFY_SERVICESTATE']

MK_HOST_IP = NEW_INFO.split(",")[0]
MK_DATACENTER = NEW_INFO.split(",")[1]
MK_TEAM = NEW_INFO.split(",")[2]
MK_VENDOR = NEW_INFO.split(",")[3].replace('\n','')

MK_JSON_OUTPUT = json.dumps([{'mk_hostname': MK_HOSTNAME, 'mk_description': MK_DESCRIPTION, 'mk_plugin_output': MK_PLUGIN_OUTPUT, 'mk_last_state_chage': MK_LAST_STATE_CHANGE, 'mk_servicestate': MK_SERVICESTATE }])

post_data(customer_id, shared_key, MK_JSON_OUTPUT, log_type)

After yout get this as an active notification method on your Check_MK site, you'll start to see data getting in OMS like :



This is really nice to have - the OMS analytics engine is awesome and gives you a lot of possibilities.

Obiously, you might want to use the above example on anything you want to, so, your imagination could simply adapt this to your own case scenario.

Hope you all like this thread, and i promise to post things here more frequently.

Cheers,