I had a machine misbehaving. Nothing dramatic, just the kind of thing that ruins an afternoon: it would slow to a crawl, stay there for a while, and be perfectly fine again by the time I was looking at it. Task Manager tells you what is happening right now. It tells you nothing about what was happening at 03:40 while you were asleep.

What I actually needed was narrow. One box. A record of overall CPU and RAM, plus which processes were responsible, sampled every fifteen seconds, in a format I could open in Excel and sort. Short-term. Then delete.

So I wrote ServerMonitorScript, and it turned out to be a much better exercise than I expected. Not because the tool is clever. Because the problem is sneakier than it looks, and because of how the build went.

Let me be upfront about that second part: I built this with a coding agent, and I am going to talk about that honestly, including the bugs it left behind and the ones it introduced while fixing those. That turned out to be the most useful thing in the whole exercise.

First: why not just use Prometheus?

This is the first question any DevOps person asks, and it deserves a straight answer rather than a defensive one.

If you are monitoring a fleet, use windows_exporter with Prometheus and Grafana. If you are in Azure, use Azure Monitor and VM insights. Those are the right tools, they are better than anything I am going to write in an afternoon, and I have no interesting argument against them.

What I had was one machine, a question with a short shelf life, and a preference for not standing up an exporter, a scrape target, a time-series database and a dashboard to answer it. I wanted a CSV. The tool I built is a diagnostic, not infrastructure, and the README says so in a section called “When not to use this.”

I mention this because I think a lot of “I built my own X” posts quietly skip it, and the skipping is what makes readers suspicious. The honest version is more defensible: I knew the proper tooling existed and chose not to use it for a specific, small reason.

The part I thought would be easy

Getting CPU percentage per process. I assumed this was a property you read. It is not. It is a number you compute, and the computation has a few traps.

Get-Process gives you a .CPU property, but it is cumulative CPU-seconds since the process started, not a percentage. A process that has been up for three days and used 400 seconds of CPU tells you nothing about whether it is busy now. To get a rate, you diff two samples:

powershell

$delta = $CurrentCpuSeconds - $PreviousCpuSeconds
[math]::Round(($delta / $ElapsedSeconds / $ProcessorCount) * 100, 2)

Three things fall out of that, and each one cost me something.

You need two samples before you can report anything. The first sample after startup has no baseline to diff against, so every process reports 0%. This is not a bug, but if you do not know it you will spend a while wondering why your monitor thinks an idle box and a burning box look identical for the first fifteen seconds. I documented it in the README because I knew I would forget.

You have to divide by core count. A single process saturating one core on an eight-core machine has consumed one CPU-second per wall-clock second. Without the $ProcessorCount division you report that as 100%, which is wrong, and it will not match what Task Manager shows the user next to you. Task Manager normalises across all cores. If your tool disagrees with Task Manager, people trust Task Manager, and they are right to.

PIDs get recycled. This one is genuinely nasty, and I did not think of it until later. If you track previous CPU-seconds keyed on process ID alone, and Windows reuses a PID between two samples, you diff a brand new process’s CPU-seconds against a completely unrelated dead process’s baseline. The new process has used almost no CPU, the old one had used plenty, so the delta comes out negative. You clamp negatives to zero and it looks fine.

Until it goes the other way. A short-lived process that gets a recycled PID from something that had barely run produces a large positive delta, and you get a phantom spike in your data: some process apparently pinning the CPU for one sample, then vanishing. On a box where you are hunting an intermittent problem, a phantom spike is genuinely harmful. You will chase it.

The fix is to key on process identity, not process ID:

powershell

function Get-ProcessSampleKey {
    param(
        [Parameter(Mandatory)][int]$ProcessId,
        [AllowNull()][Nullable[long]]$StartTimeTicks
    )

    if ($null -eq $StartTimeTicks) { return "$ProcessId|unknown" }
    return "$ProcessId|$StartTimeTicks"
}

A PID plus a start time identifies a process instance. Note the null path: Idle, System, and various protected processes will not give you a StartTime at all, and reading it throws. So you catch, mark the key unknown, and skip CPU-delta tracking for those rather than letting an exception take down your sampling loop.

Two CPU numbers that do not agree

The first working version had a subtler problem. System-wide CPU came from Win32_PerfFormattedData_PerfOS_Processor, which hands you a ready-made percentage. Per-process CPU came from the delta calculation above.

Both numbers are defensible. They are also computed over different windows by different mechanisms, so they do not reconcile. You would see system CPU at 30% while the process rows summed to something quite different, and there is no way to explain that in a CSV. Anyone using the data would assume one of the numbers was broken.

The fix was to compute both the same way, from raw counters rather than pre-formatted ones:

powershell

$deltaIdle = $CurrentIdle - $PreviousIdle
$pct = 100 - (($deltaIdle / $deltaTimestamp) * 100)

Win32_PerfRawData_PerfOS_Processor gives you cumulative idle time and a timestamp in the same 100-nanosecond units. Diff both, work out what fraction of the interval was idle, subtract from 100. Now the system row and the process rows are both interval-averaged over the same window, and they behave consistently. Both report 0 on the first sample, for the same reason, which is itself a small sign that the two paths are doing the same thing.

There is a nice side effect: because the raw counters are absolute, the maths survives a skipped cycle. If one sample fails, the next one covers a longer window and is still correct, rather than silently reporting a rate over the wrong interval.

A smaller one: your sleep is lying to you

The naive loop is:

powershell

while ($true) {
    # do the work
    Start-Sleep -Seconds 15
}

The actual interval is fifteen seconds plus however long the work took. If a cycle takes 800ms, your samples land every 15.8 seconds, and the error accumulates all day. By evening your timestamps are meaningfully out of step with the wall clock, which matters if you are correlating against an application log.

Time the cycle and sleep the remainder:

powershell

$elapsed = $stopwatch.Elapsed.TotalSeconds
$remaining = $intervalSeconds - $elapsed
Start-Sleep -Milliseconds ([int]($remaining * 1000))

Use -Milliseconds, not -Seconds. On Windows PowerShell 5.1 the -Seconds parameter is typed [int], so your carefully computed 14.2-second remainder gets rounded straight back to a whole second and you have undone half the fix.

How this actually got built

I specced the tool and had a coding agent implement it. That first version was good. Clean functions, sensible error handling, correct CSV escaping, and the CPU-delta approach was right from the start.

It was also wrong in ways that took a careful read to find.

The scheduled task ran at logon, as the interactive user. For a server that is the wrong model twice over. It requires somebody to be logged in, so a rebooted box silently stops monitoring. And Get-Process cannot read .CPU for processes owned by other users unless you are elevated, so on a multi-user or RDP box you silently undercount. Not an error, not a warning. Just quietly wrong numbers. It now runs at startup as SYSTEM, with a -RunAsUser switch for desktop use.

The CSVs had no retention, so a monitoring tool would eventually fill the disk it was monitoring.

There was no alerting, which meant the thing was not really monitoring at all. It was logging.

So I wrote a spec: fifteen items, prioritised, each with the reasoning attached. The agent implemented all fifteen, and did a few of them better than I had asked. Where PSScriptAnalyzer flagged a rule it could not satisfy cleanly, it excluded the rule and left a comment explaining why the alternative was worse, which is a judgment call I would have been happy to see from a person.

Then I reviewed it properly, and found four bugs.

The four bugs, and the pattern in them

The log file did not capture the messages that most needed capturing. The scheduled task runs with a hidden window, so console output goes nowhere. That is the entire reason for writing warnings to a file. But the log path was resolved partway through the first sample cycle, and three of the most important messages fired before that: “another instance is already running”, the startup banner, and the first cycle’s config warnings. All three fell through a null check and vanished. The one feature added specifically to make failures visible was invisible for exactly the failures that mattered.

Retention never ran on a fresh start. The sweep was gated on the log date having changed since the last check, and on startup there was no last check, so the guard was false and nothing got purged. Retention only ever fired if the process survived past midnight. On a box that reboots nightly, the feature did nothing at all.

The new log file had no retention of its own. Retention filtered for dated CSV files. The warnings log is a single undated file, so it was never matched, and it grew forever. In normal use it is tiny. But picture an intermittent WMI fault that logs a warning every cycle without ever hitting the consecutive-failure limit: roughly 5,800 lines a day, indefinitely. The disk-filling problem I had just fixed, reintroduced through the fix.

Mutex creation was unguarded. The single-instance lock used a Global\ mutex, created outside any try/catch, before any logging existed. Creating objects in that namespace needs the “Create global objects” privilege, which SYSTEM has and an ordinary interactive user may not, which is precisely the desktop path. Any failure there kills the script with a raw .NET stack trace and nothing in the log.

Here is the thing I actually took away. I had pulled all the pure logic into a separate file and covered it with Pester tests: CSV escaping, settings validation, the CPU maths, the process key, the retention filter. Those tests all passed, and every one of them was correct.

All four bugs were outside that file. They were in startup ordering, lifecycle, and the seams between components. Not one of them was a wrong calculation. They were all “the right calculation, wired up wrong”.

That is not a coincidence, and it is not really about AI. Pure functions are easy to test because they are easy to think about, and they are easy to think about because they have no state and no ordering. The bugs go and live in the part you did not extract, for exactly the same reason you did not extract it.

Round two

I specced the fixes. All four came back correct, and the mutex fallback was better than what I asked for: rather than just failing loudly, it now falls back to a session-local mutex and logs the downgrade, so desktop users get a working tool with a slightly weaker guarantee instead of a stack trace.

That round introduced three new issues. All of them were at the boot sequence, which is precisely where the previous round’s fixes had gone in. A stray Logs folder created before the config is read. Boot-time config warnings written to the default log path and then duplicated to the real one. A null return under -WhatIf that the next line dereferences.

Smaller than the first four, and the same shape: state and ordering, at the seam.

One more observation, and it is the one I keep turning over. The retention fix extracted this:

powershell

function Test-ShouldSweepLogRetention {
    param(
        [AllowNull()][Nullable[datetime]]$CurrentLogDate,
        [Parameter(Mandatory)][datetime]$Today
    )

    return $CurrentLogDate -ne $Today
}

Three tests cover it. They pass. But look at the body: it is a single comparison. The bug was never in that comparison. The bug was a nested conditional at the call site, and what fixed it was deleting that conditional, not writing this function.

So the tests are close to tautological. What they actually do is document the invariant, and put a name and a comment somewhere future-me will read. The structure is what enforces correctness. The tests just explain it. I think that distinction matters more than test count, and I think test count gets used as a proxy for correctness far more often than it deserves.

What I would tell you if you are doing the same thing

An agent is very fast at implementing a spec and very good at the parts with clear inputs and outputs. The judgment lives in the spec, and the risk lives in the seams: startup order, lifecycle, what happens on the first iteration, what happens when the thing restarts, what happens when a component below you is unavailable. Review those first. They are where I found every single bug across two rounds, and I doubt that was luck.

And write the “when not to use this” section. Mine points at windows_exporter and Azure Monitor, because for almost everything larger than the problem I had, those are the right answer. Being clear about the boundary of a tool is not an admission of weakness. It is most of what makes the tool trustworthy inside that boundary.

The code is on GitHub, MIT licensed. It lints and tests in CI. It monitors exactly one machine, and it is quite good at that.