Sunday, April 13, 2014

Initializing an OpenLDAP database with the LDIF configuration

Pretty much all host and network services have traditionally been configured using flat files in /etc.  Several also have databases which are stored in flat files, and sometimes even intermingled with the configuration proper.  ISC DNS and DHCP are two significant ones.  This has the advantage of making the configuration and data easy to edit and update manually.  The disadvantage is that it must be edited and updated manually and any change means either restarting the daemon or signalling it to reload the database.

The most common solution to the editing problem is to create templates and scripts to make changes and re-generate the config/database files.  This still requires kicking the daemon for each change.   The data is often stored in a back-end database which the scripts read to generate the new config files.

What many people don't know is that both ISC DNS and DHCP can use an LDAP database directly as the back-end.  Using the LDAP database, changes can be made programatically, using standard protocols and standard APIs that implement them.

In the next couple of posts I plan to show how to create an LDAP backed DHCP service, but I need a working LDAP service first.  This post will show how initialize the LDAP service on a Linux server using OpenLDAP.  I'm going to do most of the work on Fedora 20, but it should all translate simply to either Red Hat Linux or to Debian based Linux distributions.  Where I am aware of it I'll make notes on the differences for those.

Ingredients


  • LDAP database top level distinguished name (DN): dc=example,dc=com
    A domain object for DNS domain example.com
  • LDAP admin user: cn=Manager,dc=example,dc=com
  • Initial admin user password: make one up.

LDAP terminology 101


LDAP is actually not nearly as complicated as it has been made to seem.  It does have some rather arcane terminology and it helps to get that out of the way before starting.

LDAP is a hierarchical key/value database.  This means that each value has a unique name (the key) and that each key is composed of two parts.  The first part is the local name and the remaining part is the name of the "parent" object.  At the top is the "root object" which has is special in that it has no parent. The root object can have direct values and it can have children, other objects which have their own values.

In some ways you can think of an LDAP database in the same way as you think of a filesystem. There is a root path to the top directory.  Each directory can contain files and subdirectories which in turn can have their own subdirectories.  Unlike a filesystem each object (directory) has one or more "objectClass" definitions which define the set of acceptable values and types of children.

Unlike a filesystem you can't easily browse the directory tree.  You need to know the name of the value you want, though you can make queries using type and value patterns.

Here are the most important terms you need to know to get started with an LDAP database:
  • LDAP service
    The process which answers LDAP queries.  May contain more than one database
  • LDAP database
    A unit of related data contained within an LDAP service.  Each database has a "Base DN"
  • LDAP schema
    The definition of sets of related data objects.  The schema defines both the attributes of the objects and their relationships (if any)
  • LDAP Data Interchange Format (LDIF)
    A serialized text format which describes both the contents of a database and certain operations on the contents (add/modify/delete)
  • Distinguished Name (dn)
    A unique name for a data object within the database.  A DN is usually composed by prepending a Common Name onto the object's parents DN. 
  • Base DN
    The root of the data hierarchy within an LDAP database.
  • Common Name (cn)
    A potentially non-unique name for a data object.
  • Object Class (objectClass)
    An attribute of a data object which defines which other attributes and relationships the object can have.  An object may have multiple object classes.
  • Domain Component (dc)
    This indicates one part of a DNS domain name.  The parts normally separated by dots (.) This is only called out specially here because DNS domains are commonly used as the conventional RootDN for corporate LDAP databases.

Required Packages


The first step is to install the OpenLDAP software packages.

I work with two main Linux distribution families. I differentiate them by the packaging mechanism since that's the practical difference that I have to deal with.  It's not nearly the only difference.

Since I work at Red Hat (actually long before I worked and Red Hat) I use RPM based distributions like Fedora and Red Hat Enterprise Linux (RHEL).  The other major distribution family is the Debian based distributions which also include Ubuntu and its variants.  Each family tends to contain forks of one of the two "parent" distributions so that the locations and names of packages and the files they contain tend to fall into one of those two groups.

I'm going to refer primarily to the locations of files in the RPM based distributions. I'll call out the variations for Debian distributions when it matters.

If you're installing a new OpenLDAP service then the first thing you need to do is install the required packages.

RPM based systems (Fedora, RHEL)


  • openldap-servers
  • openldap-clients

Debian based systems (Debian, Ubuntu...)


  • slapd
  • ldap-utils

Debian systems in their misguided (though sometimes effective) attempt to make things easier for sysadmins attempts to configure and start new services when the packages are installed.  When you install the slapd package you will be prompted for the initial admin password for your LDAP service.  Have your initial password ready before you begin package installation. When the package finishes installing you will have a running, but not yet properly configured LDAP service. You will be able to skip several of the steps below.  Watch for the notes.

Initialize the LDAP server


The code samples below are from a Fedora 20 system.  You'll need to adjust file locations for the schema and configuration files if you're running on a Debian based system.

Once the OpenLDAP packages are installed it''s time to begin setting up the contents of the LDAP database.  If you're working on a Debian based system you can skip the next step as it is done for you when it starts the service.

Copy default DB_CONFIG (Fedora)


If you installed on Debian and it set the initial password and started the service for you, you can skip down to the next section.

OpenLDAP typically defaults to using one of two varieties of the Berkeley DB storage format.  The standard Berkeley DB format is indicated by "bdb".  A more recent version tuned for hierarchical databases like LDAP is known as "hdb".  When I looked recently both Debian and Fedora create an initial database with the hdb format.

The BDB derivatives are very tunable to a level  to which most people will not be interested.  The tuning it set in a file called DB_CONFIG which resides in the same directory as the database files (/var/lib/ldap). Both Debian and Fedora offer a default tuning file and I generally use it unchanged.

  • /usr/share/openldap-servers/DB_CONFIG.example

cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG

At this point I can start and enable the slapdservice on RPM based systems.

sudo systemctl start slapd
sudo systemctl enable slapd

Communicating with OpenLDAP (local)


The default initial configuration of OpenLDAP allows the root user to view and manage the database configuration using the LDAP client tools and commands expressed in the LDIF... format (yes, it's redundant, but colloquial).  The database will accept queries and changes from the system root user (UID=0,GID=0).  Since I'm a fan of doing things as a non-root user, you'll see most calls to LDAP client commands via sudo.

There's a special incantation to authenticate this way.  It has three parts and looks like this:


I'll show how this works in the next section.  For ldapsearch commands I'm also going to add -LLL.  This suppresses some formatting and comments that you probably want to see, but which is more verbose than is useful in a blog post.  You can safely leave it out of your queries if you want to see the complete output.

Loading the standard schema


An LDAP service is a database in one traditional sense.  Each of the data objects is defined in a schema which describes the attributes of the object.    The schema must be loaded into the configuration database before the objects they define can be used in the user database.

In the Fedora and Debian software packages, the standard schema are provided as LDIF files which can be loaded using the ldapadd command.  The  call is similar to the ldapsearch command above:

ldapadd -Q -Y -H ldapi:/// -f <filename>>

One Fedora systems, the stock schema files are located in /etc/openldap/schema.  Each one is offered in both the original LDAP schema form and in LDIF.  Most LDAP databases will use three standard schema to start:

  • core
  • cosine
  • inetorgperson

These three define the basic objects and attributes needed to describe a typical organization: people, groups, rooms etc. Loading these three would look like this.

ldapadd -Q -Y -H ldapi:/// -f /etc/openldap/schema/core.ldif
ldapadd -Q -Y -H ldapi:/// -f /etc/openldap/schema/cosine.ldif
ldapadd -Q -Y -H ldapi:/// -f /etc/openldap/schema/inetorgperson.ldif

Finding the database configuration object


As noted above, the LDAP service can contain multiple databases.  In fact, it must because on of the databases is the configuration database itself. Like all LDAP databases, the configuration database has a DN which defines the root of the database for queries.  The DN of the configuration database is cn=config. That is: Common Name = "config".

We can query and modify the OpenLDAP configuration using the ldapsearch, ldapadd and ldapmodify commands (or any other client mechanism which can use SASL external authentication). That is: we can configure LDAP using LDAP.

Now we won't want to store our data in the configuration database.  Each distribution includes a default database configuration object for a user database. Database configuration objects have the objectClass: olcDatabase. The user databases are indicated by the data storage back end (bdb|hdb). This means we can query for the list of databases and then find which one is the user database by looking at the DN.

sudo ldapsearch -Q -Y EXTERNAL -H ldapi:/// -LLL -b cn=config olcDatabase=\* dn
dn: olcDatabase={-1}frontend,cn=config

dn: olcDatabase={0}config,cn=config

dn: olcDatabase={1}monitor,cn=config

dn: olcDatabase={2}hdb,cn=config

Each LDAP search query has two parts.  The first is a filter which selects which records to report.  The second (optional) is a selector for which fields to report for each record.

The query above indicates to search within the base DN (-b) cn=config and search for all records with a key named 'olcDatabase' regardless of the value (olcDatabase=\*) and report the dn field.

The result shows that the LDAP service has four databases. The numbers {0} are essentially LDAP array indices.  The part after the index indicates the database back end.  We're only concerned with two of these right now.

We're working with the config database {0}config,cn=config. The database we want to configure is the hdb back end.  The DN for that is olcDatabase={2}hdb,cn=config. We'll base the rest of our search and change queries on that.  Now we can query the current database configuration object.

(In Debian systems you will likely not see the monitor database, and the index of the hdb database will be 1. Adjust accordingly)

sudo ldapsearch -Q -Y EXTERNAL -H ldapi:/// -LLL -b cn=config 'olcDatabase={2}hdb' 
dn: olcDatabase={2}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {2}hdb
olcDbDirectory: /var/lib/ldap
olcRootDN: cn=Manager,dc=my-domain,dc=com
olcDbIndex: objectClass eq,pres
olcDbIndex: ou,cn,mail,surname,givenname eq,pres,sub
olcSuffix: dc=my-domain,dc=com


The olc prefix on the class and attribute names indicates that they are part of the OpenLDAP configuration schema.

The interesting values right now are the olcSuffix and olcRootDN attributes (as well as the absence of an olcRootPW). The default database on Fedora, seen here starts with a suffix of dc=my-domain,dc=com and the root user (aka RootDN) is cn=Manager,dc=my-domain,dc=com. These are perfectly valid but useless values. For a real database we want to define our own DB suffix and root user.

Set the Database Suffix


By loose convention the LDAP database suffix for corporate LDAP services is based on the DNS domain of the organization.  This also defines the top level object in the database which we will add later.

I'm going to replace one useless default convention with another because, well using real DNS names might mess people up if they cut-n-pasted stuff from this blog. I'm going to create a database for the mythical Example Company, Inc. Of course their domain name is example.com. Now I have to translate that into an LDAP dn:

dc=example,dc=com

A domain name is composed of a list of Domain Components.  See how that works?  So we want to replace the existing olcSuffix value with our new one. This will be the first change to the default database.

Changes made using ldapadd or ldapmodify are defined using LDIF in the same way that the output of ldapsearch is expressed in LDIF.  We have to craft a change query for the olcSuffix of olcDatabase={2}hdb,cn=config and replace the existing value with our new one. Here's what that looks like:

sudo ldapmodify -Q -Y EXTERNAL -H ldapi:/// <<EOF
dn: olcDatabase={2}hdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=example,dc=com

EOF
modifying entry "olcDatabase={2}hdb,cn=config"

The ldapadd and ldapmodify commands expect a stream of LDIF on stdin unless an input file is indicated with the -f option. I provided the update stream as a shell HERE document indicated by the EOF markers.

If you run the ldapsearch query from the previous section you can verify that the olcSuffix value has been changed.

Set the Root DN


Now that we've set the suffix for our database we need to update the DN of the user who will be able to make changes (who is not the root user on the LDAP server host).

User names in LDAP are Distinguished Names of objects stored within the database, the same as any other record. We might want to keep the (common) name "Manager" but we need to place it within the proper hierarchy for our database. Since our database is now dc=example,dc=com then the manager really must be cn=Manager,dc=example,dc=com. We'll update that in the same way that we did the suffix.

sudo ldapmodify -Q -Y EXTERNAL -H ldapi:/// <<EOF
dn: olcDatabase={2}hdb,cn=config
changetype: modify
replace: olcRootDN
olcRootDN: cn=Manager,dc=example,dc=com

EOF

modifying entry "olcDatabase={2}hdb,cn=config"

Set the root password


The final step of the stock LDAP service set up is to create a database user password which can be used to make queries and changes without requiring the system root user to do it. The attribute for this password is olcRootPW. (it goes with the olcRootDN set above).   If the RootPW is unset then the RootDN cannot log in.  When you add this attribute, you are opening up access to the database a bit, but securing the system by allowing the DB admin to work without needing system root access.

The OpenLDAP service can store passwords in clear text (BAD) or using one of several one-way hash algorithms. You can create a new password hash using the slappasswd command. The default hash is currently SHA1, which is better than all of the others but still could be improved.

slappasswd
New password: 
Re-enter new password: 
{SSHA}nottherealhashstringthiswontworkuseyourown

At the prompts, enter the password you want and confirm it. The last line is the hashed result. This will be placed as the value of the olcRootPW attribute. See the tricky thing I did to prevent you from cut-n-pasting that last bit and using a bad password?

sudo ldapmodify -Q -Y EXTERNAL -H ldapi:/// <<EOF
dn: olcDatabase={2}hdb,cn=config
changetype: modify
add: olcRootPW
olcRootPW: {SSHA}nottherealhashstringthiswontworkuseyourown

EOF
modifying entry "olcDatabase={2}hdb,cn=config"

Create the top object in the database


Since LDAP is a hierarchical database, each object must have a parent.  Because it can't be "Turtles All The Way Up", there must be one special object which has no parent, but which is the parent of all of the other objects in the database. That's the object who's DN is the value of the database configuration olcSuffix.

Most organizations use their domain name as the pattern for the top DN and use an LDAP "organization" object for that top object. An organization object is a container. It is meant to have children of arbitrary types. This allows for the creation of any desired structure for the database. Because the suffix is a domain name, The object must also be a Domain Component object. Domain Components are not top level or container objects. They must have a parent. By combining the organization and domain component classes we create a top level object that can have the name we want.

We're going to create a very minimal organization object at the top of the database to contain the DHCP server (machine) objects and the DHCP service (content) objects.

Organization objects have only one required attribute. the o value is a string which is the organization's name. It may also have a description attribute.

ldapadd -x -w secret -D cn=Manager,dc=example,dc=com  -H ldapi:/// <<EOF
dn: dc=example,dc=com
objectClass: domain
dc: example
description: The Example Company of America

EOF

Summary

At this point I have a running LDAP service with a minimal database. The configuration database contains the minimal schema needed for a typical LDAP service. A single user database has been defined. It contains only the top object named with the shortest DN possible in the database: dc=example,dc=com. An administrative user account has been defined and a password set for it.

The database is ready to be populated and used.

References


  • OpenLDAP - http://www.openldap.org/
  • Configuring slapd; http://www.openldap.org/doc/admin24/slapdconf2.html
  • Another Config Guilde: http://www.zytrax.com/books/ldap/ch6/slapd-config.html

Sunday, March 17, 2013

Some times you have to do it yourself.

A Quest for Inexpensive Power Control


When I started working on designing a teaching lab for system administration I wanted to include as many of the remote control and monitoring elements as I could.  One of the elements I wanted was remote power control.  I was discouraged to find that commercial grade switched PDUs are hard to find for less than $500US.  I've used hobbyist home automation with powerline signalling and found it to be cumbersome and unreliable.  A lucky search query lead me to PowerUSB.  It seemed to be exactly what I wanted. It's relatively inexpensive, uses standard USB communications and the vendor provides software for Linux.

It is important to note that these are not commercial grade power control.  The three switched sockets cannot carry a full 15A at 120VAC.  One of them is rated at 6A, controlled with a relay and the other two at 4A with solid state power switch.  You can't use these to control high powered rack-mount machines.  They'll do find for ordinary desktop units and are ideal for the low-power ARM systems I'm working with.

I ordered two of the Basic units to try in my first PiLab pod.

Almost, But Not Quite


A few days later they arrived, packed in nicely designed boxes. It didn't take me long to find that things weren't as bright as they looked.

The software on the included CD is a GUI application for Windows.  I want to be able to control the strips from Linux.

There are downloads for Linux on the PowerUSB web site.  Cool.  But they are binaries and shared libraries for ix86 and x86_64 only.  The aren't packaged and require a number of manual steps (placing udev rules) requiring root access to make them work.

Tweaks?


Alright.  I write to the email address indicated on the site and several days later I get an email which includes access to zip files containing the Linux source files.

It turns out that the Linux code seems to be a bit of an afterthought, or possibly the folks producing it aren't versed in best practices for Linux.  It uses a publicly available USB HID library which is licensed to allow both open source and proprietary use (hidapi).  The code that actually communicates with the power strip is fairly clear and consistent, but the CLI command that wraps it doesn't take advantage of current CLI argument patterns.  The output isn't really designed for either humans or computers to use easily.

Fair enough.  It could be that headless Linux servers are not their target market and so they haven't devoted too much time to it. They were very nice to give me what they did.  I can ask for more, or I can do it myself.

I started by writing a set of Makefiles and RPM packaging boilerplate so that I could automatically generate software for both RPM and DEB based distributions. (I work at Red Hat and I run Raspbian on my RPis at the moment).  That was OK as far as it went.  I could package and re-produce what I'd been given.  I sent that back to the folks at PowerUSB so they can share and use it if they wish.

There was something unsatisfying in the result.  So I dug some more.

What do I REALLY Want?


What I really wanted was something that would work on any distribution without a lot of compilation dependencies.  I like Python so I looked at what was available.

I found PyUSB and PyUDEV.  libhid has Python bindings.  In the end I found it easiest to use PyUSB to port just enough of the original hidapi to Python to do what I need.

It's not finished, but it does enough now for my purposes. It only manages the Basic features.  I'll need to get one each of the IO, Watchdog and Smart units to finish development and testing.

Usage Sample

So here's what I want to be able to do:

powerusb
0:0, Basic    , FWVer: 3.1, Curr(mA)  10.0, Power(KWh): 3.92, off,  on,  on
0:1, Basic    , FWVer: 3.1, Curr(mA)  10.0, Power(KWh): 8.70,  on, off, off

powerusb --xml 0:1 status 

powerusb 0:1:2 on

powerusb --syslog 0:1:1

Eventually I'd like to add a couple more things, like labeling power strips and/or sockets so that you can call them out by name.  I'd also like to be able to extend the library and CLI program to be able to manage the IO, Watchdog and Smart units.

Revision Control and Project Planning

The source code (such as it is) is on Github at https://github.com/markllama/powerusb

I've made just the slightest stab at trying Agile project management even when it's just me. I have this project on a free project management site, Trello: Mine is the powerusb-cli project.  (I just set it public, let me know if you want to join the project).

Please note that this is ALPHA! ALPHA! ALPHA! software. It will be changing as I get time to add things.

Hardware Wish List

There are a couple of things I'd like to see in the power strips themselves to make managing them easier.

Unique USB identifier

USB devices are identified by a Vendor and Product ID.  The Vendor ID is acquired from the Universal Serial Bus Implementer's Forum (USB-IF).  Currently the PowerUSB devices identify themselves by the chip manufacturer and device which PowerUSB used.  I see some indications that PowerUSB is not the only device manufacturer who has used those chips without modifying the Vendor or Product ID.  This could lead to mis-identfication of those devices if someone connected a different one that used the same chip. 

Unique Serial Number


USB Devices also report a serial number when the are connected.  It appears that the serial number for the two devices I have are identical.  The only way currently to distinguish two devices is by the bus and port number to which they are connected.  Adding devices, removing them, switching ports or adding or changing a hub could result in a device appearing to move.  A unique serial number on each device could resolve any ambiguity.

Is it an HID really?


This one is a nit-pick on my part, I admit. 

PowerUSB took advantage of the simplicity of the HID (Human Interface Device) specification and the plug-and-play nature of the HID connection protocol to make connecting PowerUSB strips simpler for them.  But HID devices are supposed to be things that humans use to communicate with the computer: Keyboards, Mice, Tablets, Microphones, Cameras etc. With a little more work PowerUSB could have implemented this as a regular USB device.  I'm going to be looking to see if I can manage this as a pure USB device.  The fragment of the hidapi library which I ported essentially lays the HID interface over plain ol' USB.

Just One Part

I have to remind myself that this is just one part of the real goal: The Pi Lab.  I've got most of the parts of the first pod.  The other hitch I've hit is that I specified a router that's been discontinued.  I'd also specified CeroWRT (bufferbloat.org) which only runs (and is being developed) on that discontinued router.  I think I'll have to pick a new router and switch to the more generally available OpenWRT.  More on that search later.

References



  • hidapi - https://github.com/signal11/hidapi
  • PyUSB - http://sourceforge.net/apps/trac/pyusb/
  • PyUDEV - https://github.com/lunaryorn/pyudev
  • libhid - http://libhid.alioth.debian.org/

Wednesday, January 2, 2013

PiLab - Management and Monitoring Infrastructure

In previous posts I began detailing the components and layout of a lab for teaching system administration to high school or beginning college students.  So far I've focused on the parts that the students will manage and control.  The student gear is formed into pods of four Raspberry Pi units and sufficient additional gear to power and control them. The pods are modular.  The infrastructure defined below can support one to four pods. On top of the student work spaces the instructor will need to set up,  manage and monitor the pod operations. In addition I want to have a boundary between the lab networks and the exterior net to keep the world out and the lab in.

Each of the pods has four cables running from it:


  • Head Node Serial Console (USB B, Male)
  • Head Node Network (Cat 5e UTP, RJ-45)
  • Lab (Pod) Network (Cat5e UTP, RJ-45)
  • Power (110VAC, 15A NEMA 5-15P)

I want to aggregate the head node consoles on a master node.  I'm not sure yet if it matters but I think I'd like to keep the head node network separate from the lab nets.  Right now the pod power will come from a fixed power strip.  I think the pod power draw is low enough that they might be run from a switched socket, but for now I think I'm going to leave that alone.

For network isolation I need a router.  Another Netgear WNDR3800 will do nicely.  A pair of Netgear 5 port unmanaged switches will fan out the lab and head node networks.  Another 7 port USB hub will pull together the serial consoles from the pod head nodes as well as the router above.  There are two ports free for controlling power in on the infrastructure devices, though cost may preclude that and it may not be needed.

The master node will have attached storage.  This will be used to provide automated re-installation of the lab Pi units.

PiLab infrastructure Manifest

This infrastructure can control and manage four PiLab pods for a total of 16 student lab nodes.  The pods can form 4 separate service systems.

Here's the parts list for the infrastructure:

  • 1 Netgear WNDR3800 600N router, $110
  • 1 Raspberry Pi Model B, $35
  • 1 Adafruit Raspberry Pi enclosure, $15
  • 2 Adafruit TTL - USB serial cable, $10
  • 1 8GB SD card, $10
  • 1 D-Link DUB-7H 7 port USB hub, $26
  • 2 Netgear FS105 5 port unmanaged switch, $22
  • 2 PowerUSB Basic switched PDU, $70
  • 1 Seagate 320GB USB 2.0 disk drive, 7200RPM, $80
Total: $504
 
I didn't include the cables on this one but they should be incidental.



I haven't laid out the power cable runs.  There are 6 devices in the infrastructure which require power (aside from the pods).  At least the router, the master node and the disk drive will require fixed power.  I can save ~$120US by using all fixed power strips.  I'm assuming the maximum cost for now so that I know the upper limits.

All of the monitoring and visualization will reside on the master node, as will the basic network services.  It will have serial access to all of the pod head nodes.  Each of those have access to the pod Pi units.  Now that the hardware's complete,it's time to start work on the software tools and the lessons.

Sunday, December 30, 2012

Pi Lab - Physical Layout of a Pod

In the previous two posts I outlined an idea for a relatively low cost hardware environment for teaching system administration to high-school and early college level students.  In the first post I created a parts list for the parts of the lab.  In the second I create a manifest for each student's gear and for a working unit I called a Pod

Providing a Grounding for Remote Access



In thinking about the PiLab and what I would like to be able to do with it, I realized that I had made a mistake right at the start. I was focused on emphasizing the remote access non-graphical (non visual) aspect of system administration. I was aware still that I needed some visualization elements to give the students a means to see the effects of what they were doing. I was thinking about creating a set of monitors that work at multiple levels and which could be seen to change as the students turn their idle disjoint computers into a network of interacting services

I started thinking about how I'd lay out the lessons to introduce the systems to the students and then let them play and I realized I'd missed an opportunity with my focus on avoiding physical access.

What I realized is that, if I design the layout of the pod, the placement of devices and cable routing then I can design a session in which the students assemble their pod and can watch the monitors as each component is correctly placed, cabled and powered up. Then when the students begin work over the network they will have a visual reference to the components and connections with which they are working.


This week I've been working to make the idea a bit more concrete.  In this post I'm presenting a physical layout for a single pod. The pods are the units of student control.  Each pod will support 4 students.  Each student gets her own Raspberry Pi.  The pod provides power control and access to the console of each Pi.  It also provides power control and console access to the uplink router for the pod.

Physical Layout


The diagram below shows one possible physical layout for a pod.


The lab include 5 Raspberry Pi's.  The fifth unit is the Head Node.  This node is powered on when the pod has power.  It is the end point for the consoles for the 4 student Pi units and for the lab router. The head node also controls power to second power strip which powers the USB hub and the first three Pi units.

The pod is designed to be build on a plywood backing board approximately 42x56cm (17x22in).  All of the hardware except the Pi units have mounting holes on the back.  The Pi units can be affixed with velcro straps tacked to the board.  Additional velcro straps can be tacked along the cable runs to dress the cables.

Power Control


The PowerUSB Basic units have four sockets, but one is always on.  These will power the head node (from the first power strip) and the USB hub (from the second).  Two of the PowerUSB sockets are controlled by mechanical relays and can provide 4A.  The second power strip is plugged into one of these.  The router and the remaining 4 Pi units are plugged into the switched sockets.

In this configuration, every component except the head node can be powered by the head node itself.  The head node exposes its network port and console serial line which can in turn be used to control the pod as a whole.

Student Assembly


The Pi lab is designed to allow the students to assemble and cable the pod from components and cables.  They will get a chance to handle the parts.  The cables should be cut to length where possible and labeled.  Where the cable lengths are fixed, the pod back board should provide a means to dress the cables with slack loops.

While one of the goals of the course is to show the students how to work remotely using the network and command line, the exercise of assembling the pod and (hopefully) observing as it comes to life for the first time will give the students a sense of the hardware that is on the other end of the network.

Next: The management infrastructure


The next job is to design the infrastructure layer.  The goal there is to provide the next level network and host installation services for the lab Pi units.  It would be possible to add USB storage to the PiLab head node but I think it's preferable to provide the OS and monitoring from outside.  The monitoring must include both passive and active probes for the nodes.  It must provide near real-time graphical feedback of the Pod state to the students.  Back to work!

Wednesday, December 26, 2012

PiLab - A network lab for teaching System Administrators.

I'm getting fired up about the idea of creating an inexpensive computer lab aimed at teaching System Administration.  The creation of the Raspberry Pi and the availability of some inexpensive but rootable home routing hardware and USB power control make it possible to design and implement an environment for students to learn and break things.

There have been interminable round-and-round discussions in the Sysadmin community formed by the USENIX LISA SIG and LOPSA over the last two decades and so far few have come to any lasting result. (The LOPSA Mentorship Program is one that's excellent and ongoing and I encourage you to look at it both as a guide and a learner).  I have no idea if this will be any more successful or enduring, but at least it's different. I'm willing to throw it against the wall and see if it sticks.

So I've created a Github project to collect and track ideas.  At first they'll be my ideas but I know I can't do this myself so I'd really love contributions of ideas and work.  I'll need sysadmins of all stripes as well as writers editors and educators to help.

With lots of hope and trepidation I'd like to introduce...


- Mark

References

Monday, December 24, 2012

A Modest Pi-Posal?

Cool Geek Toys for Education!

Ever since I heard about the Raspberry Pi project I've been trying to figure out how best to use them for their intended purpose: Teaching. 

I've always despised what our secondary schools typically call "Computer Education".  My daughters came home complaining about hours learning how to embed images in office documents.  For a whole term.  There are obvious exceptions, but in general it seems you need to have an inspired and empowered teacher who offers something outside class time to get much else.

I've also thought a bit about how system administration as a profession is (not) taught in higher education.  There have been the Certification Wars.  The Coders vs Non-coders and even the Anti-Degreers.  One thing that's occurred to me is that the elements I use daily in my work aren't really beyond the capabilities of someone with a reasonable grasp of Algebra and Logic.  At least superficially your typical motivated 9th grader could understand it.

Now I realize I'm a bit of an odd bird in the teaching area.  I'm a professional system administrator.  My first thought isn't about soldering or motors or cool screens.  It's not about building web sites with Rails or JBoss.  It's not about Android games or HTML5 Canvas and Javascript.  My first thought is "they should get a chance to run through the steam tunnels!"

I have taught some.  I worked at a community theater teaching children's theater on and off for 20 years.  I have two girls.  I know that the lure of a blinking cursor can't, by itself, compete with all of the audio and video opportunities out there on all scales.  The Big, the Flashy, and the Colorful all start with an advantage even when there's little content that's not obscured by the noise.  I know I have to draw students in somehow, but that way isn't available.  I'll have to be sneaky.  I'll have to make sure they think there's something cool (or scary) just around the next corner. Which there is.   That's what kept me busy for all these years.

Now there are elements of systems and networks that can be visualized.  The filesystem tree and the network itself can be mapped. Network traffic can be graphed, sorted, filtered and color coded. If I can find a way so that the student work has effects which can be seen I'll have some means of giving the students more visceral feedback.  There's nothing inherently graphical about computers or networks.  That's one of the messages I mean to bring to the students.  My job in designing a lab and a course is keeping them interested long enough for them to get it.

I've also read a whole bunch of books on tech topics.  One set that stand out in my mind are the "Think *" series by Allen Downey.  The first book "Think Python" was originally written to teach Java to high school students, and its original title was "How to Think Like a Computer Scientist".  These are neither traditional tech topic books, nor are they traditional classroom textbooks.  They are comprehensive in their topics, but they encourage exploration and self-learning in a way that I have not seen in any other books.  These are the inspiration for what I'd like to do.

A Pi Pod Lab

I have a number of goals to meet to create a teaching environment for System Administration and several points in which it will be different from lab for teaching graphics or hardware hacking.

  • The lab must be inexpensive.
    If it's going to be implemented in schools or clubs resources will be limited.
  • The lab environment must be modular.
    The lab environment should scale to accommodate small and large groups.
  • Student dues are less than $75 for materials.
    The student goes home with a Pi model B and enough additional hardware to make it run.
  • No commercial software should be needed.
    The students must be able to use existing workstations with minimal changes.
  • Remote Access and Control
    Physical access to the lab must not be required for students.
    The instructor should be able to restore the lab to initial state without physical access.
  • Physical Composition
    The lab components should be suitable for student access without danger of injury or damage.
    Ideally the students should be able to assemble and disassemble the lab from components.
  • CLI Control Interface
    The lab components will be controlled by various CLI interfaces through the head node.
    The Pi nodes will NOT have dedicated graphical displays.
  • Visualization
    The head node will provide some graphical visualization of the network and host status.
That's a short list. There are more that will come up.

Selecting the Components

Since it's already decided that a Raspberry Pi will be the central component of the lab, it's time to look at how to tie several of them together.  I'd really like for the students to be able to manage the power and console remotely.  I'd also like for them to be able to manage the first hop switch/router if possible. Ideally they could re-install the operating system from scratch without physical access.  This would require some kind of PXE based boot service on the student network.

It is required that the lab be accessible from the school network, but it should be possible to restrict outbound traffic from the lab to protect the hosting network.  Ideally the lab is also accessible directly from the internet, but it may be necessary for safety and security to require some authenticated access to reach it from outside.

In my experience I've worked mostly with medium and large scale enterprise network hardware. So of course my first thought for a router is a Cisco Catalyst gear.  Surely I could find some used gear on the web that would be suitable. Even when you whittle it down to the smallest managed components though, used and refurb hardware  runs into hundreds of dollars.  I thought that there would be APC switched PDUs on the used market, but no. A simple 8 port 15A switched PDU runs many hundreds of dollars.  This clearly isn't going to work.

Return to the Hacker Market

I really didn't want to build the lab from totally hacked up hardware.  I've worked with X10 power controls, but I don't find managing them to be very easy.  The units are bulky and trivial noise on the line can make them unreliable.  I could use another Pi for a router, but that would require another stage of configuration and management that I'm not prepared for.  I really wanted something in the middle.  The hardware should be commercially packaged, but I should be able to replace the OS with something designed for the job I have in mind.

I have heard of several home router hacks and I recently heard a talk CeroWRT (an update of OpenWRT with the Codel anti-bufferbloat algorithm implemented in the network stack).  I'd bought a Netgear WNDR 3800 router and re-flashed it with CeroWRT.  It only has 4 downlink ports, but it has an accessible USB port and it has a set of pins installed to provide a TTL serial console.  With the addition of a TTL to USB cable I could control and monitor the router out of band.  This would let me give the students access without fear of losing connectivity if they damaged it.  It's also $100US online.

I still needed power control.  On a whim I searched for "USB managed power" and came up with a hit. A company aptly named Power USB makes four port USB controlled 15A strips and they go for $70US retail. Who knew? With those two components I have the makings of a 4 Pi unit I'm calling a pod.

The remaining components are easier.  I need a powered USB hub to aggregate the Pi consoles, the router console and the power controller. A 7 port hub will do. I need some small high quality 5V 1A USB power converters but those are easy to come by. I need a TTL to USB serial cable for each Pi and for the router.  Those are now available pre-built by AdaFruit on the Newark/Element 14 site. 

I have every thing I need.  Now to lay it out, create a manifest and price the whole thing.

A "Pod" Component Map

This below is the map I came up with.  It includes 4 Raspberry Pi for the lab units, one Netgear router as the first hop for the lab, a 7 port powered USB 2.0 hub,  one PowerUSB Basic 4 port PDU and a 5th Pi as the "Head Node" to control the whole thing.  The diagram also includes the lab gateway router which will be required to isolate the lab components while providing connectivity.



Manifest

Student Gear

This is the hardware that will be given to the student at the end of the session and replaced at the beginning of the next.  The cost of this hardware is covered under the student dues.


Not the $75US I was hoping for and it doesn't account for shipping, etc, but it's close.
For the lab I may also want shorter USB power cables to keep them neat.

Pi Pod Gear

This section lists the stuff that's needed to make one pod, suitable to contain four student Pis.  Prices are approximate.  Network cables should be custom made to fit when the enclosure design is complete.


All that should go in an enclosure of uncertain dimensions and design.  I should probably add a set of Velcro ties to keep the cables neat and I'd really love to label all the cables with a Bradey cable labeller. That will come with the enclosure design.

Infrastructure

In addition to the Pis and the pods, a certain amount of infrastructure will be needed to provide connectivity, security and control of the lab pods themselves.  This includes the gateway router, and network service system which can provide PXE boot and install files for the pod hosts. The infrastructure I envision would support up to four pods and could be extended fairly easily in units of four more pods.


This places the starting cost for a single pod at $650 in reusable parts and $320 in student gear. Each new pod will cost and additional $380US. Hopefully I can design an enclosure that I can build for under $100.  If so that's actually on-par with Lego Mindstorms NXT for a comparable sized group.

Any mech eng people out there want to try building an enclosure for the pod?

I need to build one of these up and start looking at how to use it.  Ideas?

 References

Hardware

  • Raspberry Pi Model B
    http://www.raspberrypi.org
  • PowerUSB Basic
    http://www.pwrusb.com/powerUSB-basic.html
  • Netgear WNDR3800
    http://www.netgear.com/home/products/wirelessrouters/high-performance/wndr3800.aspx
  • Adafruit TTL-USB serial cable
    http://adafruit.com/products/954
  • Newark/Element 14
    http://www.newark.com

Linux Distributions

  • Raspbian
    http://www.raspbian.org
  • Arch Linux ARM, Raspberry Pi
    http://archlinuxarm.org/platforms/armv6/raspberry-pi
  • Fedora Remix for Raspberry Pi
    https://fedoraproject.org/wiki/Raspberry_Pi
  • CeroWRT (Router)
    http://www.bufferbloat.net/projects/cerowrt