TOP for Perforce

Sometimes catching small transient activities such as fstat locking is really difficult. Here is a concept script written in Powershell that gives you a dynamic real-time view of Perforce activity.

Dot source the script, then when logged into your Perforce server with super privileges, run the p4top command. You should get a screen looking something like this. (Sorry for the black bars, I have to protect the innocent.)

Windows PowerShell  (07252012 182103)_2013-08-13_13-47-32

The interesting bit is this content flickers up on the screen very quickly and disappears just as fast. However, when watching for a while you can get a cadence for how your service is responding to user requests. It’s helpful for diagnosing transient hangs and other performance issues


Download a zip archive here: p4top-noDependencies


function p4get
{param( [Parameter(Mandatory=$true,ValueFromRemainingArguments=$true)][string[]]$Command )

	$result = @()
	$LastErrorActionPreference = $ErrorActionPreference
	$ErrorActionPreference = "Stop"
	$cmd = $Command -join " "
	$cmd = "p4 -ztag " + $cmd 
		$p4Results = iex $cmd 2>&1
		write-error $error[0]
	#funny thing about $p4Results is that it is an array of strings, we want one big long one, so lets join them
	$p4ResultsFixed = $p4Results -join "`n"

	# ztag returns data in strings like this:
	#... User zjones
	#... Update 1173264248
	#... Access 1289961618
	#... FullName Zach Jones
	#... Email
	#... Password enabled
	#Each block is seperated by an empty newline, each entry may or may not have all the attributes, each entry starts with ...

	# this regex finds all the 'blocks' like the above, returns them as a match collection
	$ms =[regex]::matches( $p4ResultsFixed, '(?m)((?:^\.\.\. \w+ .+\n)+)')

	foreach( $m in $ms )
		# we will put the attributes into a hash to make a powershell object out of

		# split each group into lines		
		$lines = $m.groups[0].value -split "`n"

		# Roll through each line

		foreach( $line in $lines )
			# Remove the beginning ...
			$line = $line -replace "^\.\.\. ",""
			# match the attribute name and attribute value
			if( $line -match "^(\w+) (.*)$" )
				# stick them in the hash
				$h[$matches[1]] = $matches[2]

		# if we found something, emit a powershell object with the appropriately named attributes and values
		if( $h )
			New-Object psobject -property $h
} #end p4get

function p4get-monitor
	$result = p4get "monitor show -sRTPI -a -e -l"
	foreach ($line in $result)
		$line.time = [timespan]$line.time

function unchunk-PSObject
{param([Parameter(Mandatory = $true)]$PSobject)
	$intermediateprop = @() #create an intermediate array to store our properties that we will comb out from objectchunks.
	foreach ($chunk in $PSobject) #for each of the two chunks (or more), parse each property in the chunk. (
		foreach ($property in ${$intermediateprop += $property} #insert each property into an intermediate table.
	$viewobjects = @()
	$unchunkedobject = new-object pscustomobject #instantiate a new $object for return from the base function.
	foreach ($property in $intermediateprop) #pull out the properties we care about and insert them with add-member into the new object.
		add-member -inputobject $unchunkedobject -membertype $property.membertype -name $ -value $property.value -force

function p4top
{param([string]$sortBy = "time",[switch]$nice)

	function PutCursor
	{param([int]$x = 0,[int]$y = 0)
		$pos = New-Object -TypeName System.Management.Automation.Host.Coordinates #create a new object to set screen cursor coordinates
		$pos.X = $x #cursor from far left
		$pos.Y = $y #cursor from far top
		$Host.ui.RawUI.CursorPosition = $pos #set the character coordinates
	clear-host #clear screen first thing so we have a nice presentation surface
	$whitespace = "                                                                                                                                              " #terrible hacky way of making the text appear cleaner. 100 characters of space bar.
	$consoleCharCount = 0 #init the console character count. This is used to clean the screen of debris occasionally.
	$maxOpsTime = 3 #maximum ops time in seconds before an operation is flagged as being "long" and operation is flagged with a solid color.
	$bg = $host.ui.rawui.backgroundcolor #save the original background color
	$fg = $host.ui.rawui.foregroundcolor #save the original foreground color
	$i=0 #init elapsed time counter
	$averageQuery=0 #init the average query counter
	$consoleLastCharCount = 0 #init the console character count. This is used to clear the screen occasionally of debris.
	$swLoop = [Diagnostics.Stopwatch]::StartNew() #start the average query stopwatch
	$swTotal = [Diagnostics.Stopwatch]::StartNew() #start the total elapsed time run stopwatch
	$p4ServerInfo = unchunk-psobject (p4get info) #get information from perforce session. This is used to identify what user is running the monitor query (and remove from the list) and identify the server in the display ui.
	$columnsArray = @() #init the display array for the data columns. We display as an array so the total character count can be stored.
	$columnsArray += @{time="Time";status="S";command="Command";user="User";id="ID";client="Client";prog="Program"} #hash array displaying the names of the commands
	$columnsArray += @{time="----";status="-";command="-------";user="----";id="--";client="------";prog="-------"} #hash array referenced the same to make underscores.
	while ($true) #infinite loop of the main program.
		if ($Host.UI.RawUI.KeyAvailable -and ("q" -eq $Host.UI.RawUI.ReadKey("IncludeKeyUp,NoEcho").Character)) #how we exit from infinite loop. Q exits.
			Write-Host "Stopping." -Background DarkRed #note that we are stopping
			$host.ui.rawui.foregroundcolor = $fg #reset foreground to original
			$host.ui.rawui.backgroundcolor = $bg #reset background to original
			break; #break from infiniloop.
		if ((($consoleCharCount - $consoleLastCharCount)/100) -gt 90000) #die if for some reason the console character count math is out of bounds.
			throw "Console character count out of bounds!"
		$wipe = 0 
		While ($wipe -le (($consoleCharCount - $consoleLastCharCount)/10))
		$consoleLastCharCount = $consoleCharCount #clear the console last char count.
		if ($nice) #if we select nicing the process, we only query one time a second.
			sleep 1
		$headerArray = @() #init the header array (the part about the warning and the server, qps, elapsed, etc)
		$contentArray = @() #init the content array
		$result = $null #clear result after every loop. Result is the monitor results from p4.
		PutCursor #put the cursor to 0,0. 

		$result = p4get-monitor | ?{$_.user -ne $p4ServerInfo.userName} #query the perforce server for the process monitor. Remove the monitor entry from ourselves. We still want to see others monitoring so we strip out the username. Subject to change without notice. :) 
		$perfMetrics = @{Server = ("Server:[" + $p4ServerInfo.serverAddress + "]");Sorting = ("Sorting:[" + $sortBy + "]");QPS = ("QueriesPerSecond:["+ $avergeQuery +"]");ET = ("ElapsedTime:[" + ("{0:N2}" -f $swTotal.Elapsed.TotalSeconds) + "]")}
		$host.ui.rawui.foregroundcolor = "white" #set foreground to white.
		$headerArray += "WARNING: This tool consumes resources on the targeted Perforce Server.                  Press Q exit."
		$headerArray += $perfMetrics | %{ '{0,15} {1,20} {2,19} {3,48}' -f $_.Sorting,$_.QPS,$_.ET,$_.Server }
		$headerArray += $columnsArray | %{ '{0,5} {1,1} {2,10} {3,20} {4,7} {5,30} {6,25}' -f $_.time,$_.status,$_.command,$_.user,$,$_.client,$_.prog } #format the column display text and store in header.
		$host.ui.rawui.foregroundcolor = "yellow"
		$host.ui.rawui.backgroundcolor = "darkred"
		$headerArray[0]  #print the warning label
		$host.ui.rawui.foregroundcolor = $fg 
		$host.ui.rawui.backgroundcolor = $bg
		$headerArray[2]  #print the status header row
		$headerArray[3]  #print the status header row lines
		foreach ($process in ($result|sort-object -descending -property $sortBy)) #iterate through each monitor object returned from p4.
			#trim some long strings so it fits in the grid.
			if ($process.prog.length -gt 24)
				$process.prog = $process.prog.Remove(24) + "+"
			if ($process.client.length -gt 29)
				$process.client = $process.client.Remove(29) + "+"
			if ($process.user.length -gt 19)
				$process.user = $process.user.Remove(19) + "+"
			#case statement coloring
			switch ($process.command) #case switch for objects. This is used to perform console color coding on each type of operation.
				fstat	{
							$host.ui.rawui.foregroundcolor = "red"
							if ($process.time.TotalSeconds -gt $maxOpsTime) #if the maxopstime is exceeded, highlight this process.
								$host.ui.rawui.foregroundcolor = "black"
								$host.ui.rawui.backgroundcolor = "darkRed"
				sync	{
							$host.ui.rawui.foregroundcolor = "blue"
							if ($process.time.TotalSeconds -gt $maxOpsTime) #if the maxopstime is exceeded, highlight this process.
								$host.ui.rawui.foregroundcolor = "black"
								$host.ui.rawui.backgroundcolor = "blue"
				where	{
							$host.ui.rawui.foregroundcolor = "green"
							if ($process.time.TotalSeconds -gt $maxOpsTime) #if the maxopstime is exceeded, highlight this process.
								$host.ui.rawui.foregroundcolor = "black"
								$host.ui.rawui.backgroundcolor = "green"
				revert	{
							$host.ui.rawui.foregroundcolor = "yellow"
							if ($process.time.TotalSeconds -gt $maxOpsTime) #if the maxopstime is exceeded, highlight this process.
								$host.ui.rawui.foregroundcolor = "black"
								$host.ui.rawui.backgroundcolor = "yellow"
				edit	{
							$host.ui.rawui.foregroundcolor = "magenta"
							if ($process.time.TotalSeconds -gt $maxOpsTime) #if the maxopstime is exceeded, highlight this process.
								$host.ui.rawui.foregroundcolor = "black"
								$host.ui.rawui.backgroundcolor = "magenta"
				submit	{
							$host.ui.rawui.foregroundcolor = "cyan"
							if ($process.time.TotalSeconds -gt $maxOpsTime) #if the maxopstime is exceeded, highlight this process.
								$host.ui.rawui.foregroundcolor = "black"
								$host.ui.rawui.backgroundcolor = "cyan"
				dm-CommitSubmit	{
							$host.ui.rawui.foregroundcolor = "cyan"
							if ($process.time.TotalSeconds -gt $maxOpsTime) #if the maxopstime is exceeded, highlight this process.
								$host.ui.rawui.foregroundcolor = "black"
								$host.ui.rawui.backgroundcolor = "cyan"
			$process | %{ '{0,5} {1,1} {2,10} {3,20} {4,7} {5,30} {6,25}' -f $_.time.TotalSeconds,$_.status,$_.command,$_.user,$,$_.client,$_.prog } #print the process with the nice formatting. (format-table has various issues so we have to do it manually)
			$contentArray += $process | %{ '{0,5} {1,1} {2,10} {3,20} {4,7} {5,30} {6,25}' -f $_.time.TotalSeconds,$_.status,$_.command,$_.user,$,$_.client,$_.prog } #add to counting array. Note that this array isn't actually displayed, it's just for counting up to perform console cleaning.
			$host.ui.rawui.foregroundcolor = $fg #set the color up for the next screen paint
			$host.ui.rawui.backgroundcolor = $bg #set the color up for the next screen paint
			[int]$consoleCharCount = ($headerArray | out-string).length + ($contentArray | out-string).length #do some math to figure out how many characters were written to the screen.
			if ($swLoop.elapsed.totalSeconds -ge 1) #poll results from the average query stopwatch. we are counting queries per second.
				$avergeQuery=$i #averagequery is then stored for the next pass in the statistics header.
				$swLoop.Restart() #reset the stopwatch back to zero.
				$i=0 #reset the query counter back to zero, and we'll wait for the next full second to elapse before doing this again.




Relocating fruit trees in spring.

Due to a property line clarification, I had to move our non-fruiting cherry tree from one side of the yard to the other. It saddens me to move it this time of year, because the likelihood of survival isn’t great.

The xylem structure in the root nodes gets damaged, and then the roots can’t transfer enough nutrients and moisture to maintain a healthy tree. Stresses induced into the homeowner from such activities? Well, that’s another post for another day.

Automating Service Desk Plus with Powershell

Assuming you’re using PowerShell 3.0 (you better be!) you can use the new method tool for REST APIs called “Invoke-RestMethod”.

It’s quite nice to have a REST parser instead of trying to write some sort of custom thing specific to each REST application I’ve needed to interact.

This little piece of demonstration code below demonstrates a way to create a new helpdesk request with an XML package, submitted through the ServiceDesk Plus URI.

In a future post, I may get around to writing additional examples that demonstrate updating existing requests. This should be pretty straightforward, as the object returned from Invoke-RestMethod returns the calls from the REST server. In this SDP implementation, you get the ID of the request returned, which you can utilize later to modify and update all aspects of an existing request.

function new-helprequest
			[string]$requester="Lastname, Firstname",
			[string]$impact="Impacts User",

	#this is an xml template to generate new SDP requests

	$url = "" #url to your helpdesk server
	$api = "/sdpapi/request/?" #api path, which is pretty much the same for SDP 8+
	$operation = "ADD_REQUEST" #the operation
	$apiKey = "541240AE9-197D-4F91-BDBC-7C614CA980D7" #the api key you generated from one of your technician accounts.

	#Configure the parameters for the xml with the content from the command parameters
	$requestXML.operation.details.requester = $requester
	$requestXML.operation.details.subject = $subject
	$requestXML.operation.details.description = $description
	$requestXML.operation.details.category = $category
	$requestXML.operation.details.subcategory = $subcategory
	$requestXML.operation.details.impact = $impact
	$requestXML.operation.details.urgency = $urgency
	$ = $site

	$uri = $url + $api + "OPERATION_NAME=" + $operation + "&TECHNICIAN_KEY=" + $apiKey + "&INPUT_DATA=" + $requestXML.InnerXml #assemble a URI. SDP expects a paramatized URI method to generate requests.
	Invoke-RestMethod -Method post -Uri $uri

Arduino TFT Shield

I’ve been working (playing) with an Arduino Mega2650, TFT shield and prototype board and I have to say it’s a lot of fun. One of the issues I ran into is the TFT library available doesn’t reference the right arduino core library in it’s various headers and references, and won’t compile out of the box. If you’re playing with the red TFT screen and mega shield like you see in the picture, try using this library to get going. I modified the include defs so it should work with newer development libraries. The link to a zip file with the modified library after the code example.

#if defined(ARDUINO) && ARDUINO >= 100
#include "Arduino.h"
#include "WProgram.h"