Sunday, March 17, 2013

Some times you have to do it yourself.

A Quest for Inexpensive Power Control


When I started working on designing a teaching lab for system administration I wanted to include as many of the remote control and monitoring elements as I could.  One of the elements I wanted was remote power control.  I was discouraged to find that commercial grade switched PDUs are hard to find for less than $500US.  I've used hobbyist home automation with powerline signalling and found it to be cumbersome and unreliable.  A lucky search query lead me to PowerUSB.  It seemed to be exactly what I wanted. It's relatively inexpensive, uses standard USB communications and the vendor provides software for Linux.

It is important to note that these are not commercial grade power control.  The three switched sockets cannot carry a full 15A at 120VAC.  One of them is rated at 6A, controlled with a relay and the other two at 4A with solid state power switch.  You can't use these to control high powered rack-mount machines.  They'll do find for ordinary desktop units and are ideal for the low-power ARM systems I'm working with.

I ordered two of the Basic units to try in my first PiLab pod.

Almost, But Not Quite


A few days later they arrived, packed in nicely designed boxes. It didn't take me long to find that things weren't as bright as they looked.

The software on the included CD is a GUI application for Windows.  I want to be able to control the strips from Linux.

There are downloads for Linux on the PowerUSB web site.  Cool.  But they are binaries and shared libraries for ix86 and x86_64 only.  The aren't packaged and require a number of manual steps (placing udev rules) requiring root access to make them work.

Tweaks?


Alright.  I write to the email address indicated on the site and several days later I get an email which includes access to zip files containing the Linux source files.

It turns out that the Linux code seems to be a bit of an afterthought, or possibly the folks producing it aren't versed in best practices for Linux.  It uses a publicly available USB HID library which is licensed to allow both open source and proprietary use (hidapi).  The code that actually communicates with the power strip is fairly clear and consistent, but the CLI command that wraps it doesn't take advantage of current CLI argument patterns.  The output isn't really designed for either humans or computers to use easily.

Fair enough.  It could be that headless Linux servers are not their target market and so they haven't devoted too much time to it. They were very nice to give me what they did.  I can ask for more, or I can do it myself.

I started by writing a set of Makefiles and RPM packaging boilerplate so that I could automatically generate software for both RPM and DEB based distributions. (I work at Red Hat and I run Raspbian on my RPis at the moment).  That was OK as far as it went.  I could package and re-produce what I'd been given.  I sent that back to the folks at PowerUSB so they can share and use it if they wish.

There was something unsatisfying in the result.  So I dug some more.

What do I REALLY Want?


What I really wanted was something that would work on any distribution without a lot of compilation dependencies.  I like Python so I looked at what was available.

I found PyUSB and PyUDEV.  libhid has Python bindings.  In the end I found it easiest to use PyUSB to port just enough of the original hidapi to Python to do what I need.

It's not finished, but it does enough now for my purposes. It only manages the Basic features.  I'll need to get one each of the IO, Watchdog and Smart units to finish development and testing.

Usage Sample

So here's what I want to be able to do:

powerusb
0:0, Basic    , FWVer: 3.1, Curr(mA)  10.0, Power(KWh): 3.92, off,  on,  on
0:1, Basic    , FWVer: 3.1, Curr(mA)  10.0, Power(KWh): 8.70,  on, off, off

powerusb --xml 0:1 status 

powerusb 0:1:2 on

powerusb --syslog 0:1:1

Eventually I'd like to add a couple more things, like labeling power strips and/or sockets so that you can call them out by name.  I'd also like to be able to extend the library and CLI program to be able to manage the IO, Watchdog and Smart units.

Revision Control and Project Planning

The source code (such as it is) is on Github at https://github.com/markllama/powerusb

I've made just the slightest stab at trying Agile project management even when it's just me. I have this project on a free project management site, Trello: Mine is the powerusb-cli project.  (I just set it public, let me know if you want to join the project).

Please note that this is ALPHA! ALPHA! ALPHA! software. It will be changing as I get time to add things.

Hardware Wish List

There are a couple of things I'd like to see in the power strips themselves to make managing them easier.

Unique USB identifier

USB devices are identified by a Vendor and Product ID.  The Vendor ID is acquired from the Universal Serial Bus Implementer's Forum (USB-IF).  Currently the PowerUSB devices identify themselves by the chip manufacturer and device which PowerUSB used.  I see some indications that PowerUSB is not the only device manufacturer who has used those chips without modifying the Vendor or Product ID.  This could lead to mis-identfication of those devices if someone connected a different one that used the same chip. 

Unique Serial Number


USB Devices also report a serial number when the are connected.  It appears that the serial number for the two devices I have are identical.  The only way currently to distinguish two devices is by the bus and port number to which they are connected.  Adding devices, removing them, switching ports or adding or changing a hub could result in a device appearing to move.  A unique serial number on each device could resolve any ambiguity.

Is it an HID really?


This one is a nit-pick on my part, I admit. 

PowerUSB took advantage of the simplicity of the HID (Human Interface Device) specification and the plug-and-play nature of the HID connection protocol to make connecting PowerUSB strips simpler for them.  But HID devices are supposed to be things that humans use to communicate with the computer: Keyboards, Mice, Tablets, Microphones, Cameras etc. With a little more work PowerUSB could have implemented this as a regular USB device.  I'm going to be looking to see if I can manage this as a pure USB device.  The fragment of the hidapi library which I ported essentially lays the HID interface over plain ol' USB.

Just One Part

I have to remind myself that this is just one part of the real goal: The Pi Lab.  I've got most of the parts of the first pod.  The other hitch I've hit is that I specified a router that's been discontinued.  I'd also specified CeroWRT (bufferbloat.org) which only runs (and is being developed) on that discontinued router.  I think I'll have to pick a new router and switch to the more generally available OpenWRT.  More on that search later.

References



  • hidapi - https://github.com/signal11/hidapi
  • PyUSB - http://sourceforge.net/apps/trac/pyusb/
  • PyUDEV - https://github.com/lunaryorn/pyudev
  • libhid - http://libhid.alioth.debian.org/

Wednesday, January 2, 2013

PiLab - Management and Monitoring Infrastructure

In previous posts I began detailing the components and layout of a lab for teaching system administration to high school or beginning college students.  So far I've focused on the parts that the students will manage and control.  The student gear is formed into pods of four Raspberry Pi units and sufficient additional gear to power and control them. The pods are modular.  The infrastructure defined below can support one to four pods. On top of the student work spaces the instructor will need to set up,  manage and monitor the pod operations. In addition I want to have a boundary between the lab networks and the exterior net to keep the world out and the lab in.

Each of the pods has four cables running from it:


  • Head Node Serial Console (USB B, Male)
  • Head Node Network (Cat 5e UTP, RJ-45)
  • Lab (Pod) Network (Cat5e UTP, RJ-45)
  • Power (110VAC, 15A NEMA 5-15P)

I want to aggregate the head node consoles on a master node.  I'm not sure yet if it matters but I think I'd like to keep the head node network separate from the lab nets.  Right now the pod power will come from a fixed power strip.  I think the pod power draw is low enough that they might be run from a switched socket, but for now I think I'm going to leave that alone.

For network isolation I need a router.  Another Netgear WNDR3800 will do nicely.  A pair of Netgear 5 port unmanaged switches will fan out the lab and head node networks.  Another 7 port USB hub will pull together the serial consoles from the pod head nodes as well as the router above.  There are two ports free for controlling power in on the infrastructure devices, though cost may preclude that and it may not be needed.

The master node will have attached storage.  This will be used to provide automated re-installation of the lab Pi units.

PiLab infrastructure Manifest

This infrastructure can control and manage four PiLab pods for a total of 16 student lab nodes.  The pods can form 4 separate service systems.

Here's the parts list for the infrastructure:

  • 1 Netgear WNDR3800 600N router, $110
  • 1 Raspberry Pi Model B, $35
  • 1 Adafruit Raspberry Pi enclosure, $15
  • 2 Adafruit TTL - USB serial cable, $10
  • 1 8GB SD card, $10
  • 1 D-Link DUB-7H 7 port USB hub, $26
  • 2 Netgear FS105 5 port unmanaged switch, $22
  • 2 PowerUSB Basic switched PDU, $70
  • 1 Seagate 320GB USB 2.0 disk drive, 7200RPM, $80
Total: $504
 
I didn't include the cables on this one but they should be incidental.



I haven't laid out the power cable runs.  There are 6 devices in the infrastructure which require power (aside from the pods).  At least the router, the master node and the disk drive will require fixed power.  I can save ~$120US by using all fixed power strips.  I'm assuming the maximum cost for now so that I know the upper limits.

All of the monitoring and visualization will reside on the master node, as will the basic network services.  It will have serial access to all of the pod head nodes.  Each of those have access to the pod Pi units.  Now that the hardware's complete,it's time to start work on the software tools and the lessons.