# Introduction to Cereal - the serial line management tool

Many people view serial ports as antiquated, out-dated connectors taking up space on their computers. However, serial ports still offer one of the best ways to communicate reliably and simply with a machine. For example, a serial port can be configured to act as a full-featured system console. This article describes how you can use cereal to monitor, log, and control access to serial lines connected to the consoles of other computers.

A computer today has several subsystems that should be run over the serial port to make it a full-featured system console:

- most modern operating systems (including the GNU/Linux utilities favored by Debian), are able to provide login terminals on serial ports,
- most modern kernels (including Linux) are capable of directing their console I/O to a serial line,
- most bootloaders (including stalwarts like GRUB, the SYSLINUX family, and LILO) can communicate via serial ports,
- and many BIOSs, including most modern server BIOSs, can be redirected over the serial port.

Using a null-modem cable, you can connect the primary serial port on one computer to a remote computer's spare serial port. The remote computer can have full access to the first machine's console using only a terminal emulator like GNU screen or minicom. Not only do you not need yet another cumbersome monitor and keyboard, but you get a digital text stream to manipulate directly. You can log, timestamp, copy, paste, diff, etc the entire data stream flowing over the console, including boot-time spew, kernel crash output, etc.

Using serial ports as system consoles is especially useful in high-density computing environments, such as racks in data centers. Clutter like monitors, vga cables, video cards, keyboards, keyboard cables, and kvm switches can all be thrown out in favor of serial cables and a single computer that we will refer to as a serial console server. A serial console server is a machine with many serial ports, each connected to the serial console of another machine. An administrator accesses the serial console server over the network (e.g. via ssh) and from there can connect to the console of any of the connected machines. In all examples below, `scs` is the hostname of a serial console server.

This is where `cereal` comes in. `cereal` is a management framework designed to monitor, log, and control access to serial lines. It is installed on the serial console server, and makes the task of maintaining such a machine simpler. It can also be used in simpler configurations, such as two machines which each monitor the other's console. This article describes how to setup and use `cereal`. We assume that the computers you are connecting to are already configured to send their consoles over the serial line. If that's not the case yet, read The Remote Serial Console HOWTO and get 'em configured first!

The `cereal` utility is not yet available in Debian proper, but it is packaged and available from the CMRG APT repository. Once you've added this repository to your `sources.list` (you may want to update your `apt` keys if you care to about verifying packages from the repo), you can pull it with `apt-get` or `aptitude`:

    scs:0:~# echo deb http://cmrg.fifthhorseman.net/debian unstable cereal >> /etc/apt/sources.list.d/cereal.list
    scs:0:~# aptitude update
    scs:0:~# aptitude install cereal

## Administering `cereal` sessions: `/usr/sbin/cereal-admin`

Serial line management with `cereal` is broken out into sessions. A `cereal` session is a single runit service controlling a special `screen` session attached to a single serial line. Creation and control of these sessions is done with the `cereal-admin` utility:

    scs:0:~# cereal-admin help
    Usage: cereal-admin  [options] [args]
    Cereal session management program.

    subcommands:
      create (c) SESSION TTY BAUD USER LOGGROUP    create cereal session
      start (s) [options] SESSION [SESSION]...     start cereal session(s)
        -a (--all)                                   start all sessions
      stop (k) [options] SESSION [SESSION]...      stop cereal session(s)
        -a (--all)                                   stop all sessions
      restart (r) [options] SESSION [SESSION]...   restart cereal session(s)
        -a (--all)                                   restart all sessions
        -r (--running)                               restart any currently running
                                                      sessions
      destroy (d) [options] SESSION [SESSION]...   destroy cereal session(s)
        -a (--all)                                   destroy all sessions
      list (l) [SESSION]...                        list session(s)
      help (h,?)                                   this help

    scs:0:~#

### Creating a `cereal` session

Let's assume the first computer whose serial console you want to monitor with cereal is called `lemur`. Let's assume `lemur`'s serial console is configured with baud rate 115200, and is connected to the serial console server on `/dev/ttyS1` (`/dev/ttyS0` on `scs` is of course reserved for the serial console server's own serial console!).

You can give access to `cereal` sessions, and the ability to read the session logs, to existing users. However, it is often a good idea to create new users and groups for these purposes. We begin by creating a new group that will be allowed to view ('follow') `lemur`'s serial console logs, followed by creating a new user that will be able to control ('attach' to) lemur's cereal session.

    scs:0:~# addgroup lemurviewers
    Adding group `lemurviewers' (GID 1001) ...
    Done.

    scs:0:~# adduser --ingroup lemurviewers --gecos 'lemur cereal session admin,,,' lemuradmin
    Adding user `lemuradmin' ...
    Adding new user `lemuradmin' (1001) with group `lemurviewers' ...
    Creating home directory `/home/lemuradmin' ...
    Copying files from `/etc/skel' ...

    scs:0:~#

You are now ready to create `lemur`'s `cereal` session with `cereal-admin`:

    scs:0:~# cereal-admin create lemur /dev/ttyS1 115200 lemuradmin lemurviewers
    Created session 'lemur':
    --f lemur /dev/ttyS1 115200 lemuradmin lemurviewers
    scs:0:~#

The first three characters of the session status line are the session flags that indicate the state of the cereal session. In order, they are:

state
  The first flag indicates whether the session is running (+) or not (-), or in some unusual state (!).
attach
  The second flag indicates whether the user can attach to the session (a) or not (-).
follow
  The third flag indicates whether the user can follow the session (f) or not (-).

Once a `cereal` session is created, you can begin monitoring of it with the `cereal-admin start` command:

    scs:0:~# cereal-admin list lemur
    --f lemur /dev/ttyS1 115200 lemuradmin lemurviewers
    scs:0:~# cereal-admin start lemur
    Started session 'lemur'.
    scs:0:~# cereal-admin list lemur
    +-f lemur /dev/ttyS1 115200 lemuradmin lemurviewers
    scs:0:~#

## Accessing `cereal` sessions: `/usr/bin/cereal`

Now that you have a cereal session up and running, you can test the connection. Log into `scs` as the non-privileged `lemuradmin` user and use the `cereal` tool to attach to or follow the session:

    lemuradmin@scs:~$ cereal help
    Usage: cereal  [options] [args]
    Cereal client program.

    subcommands:
      attach (a) SESSION                      attach to session
      follow (f) [options] SESSION            follow session log
        -c (--cat)                              cat log instead of follow
        -p (--path)                             output just log file path
      list (l) [SESSION]...                   list session(s)
      help (h,?)                              this help

    lemuradmin@scs:~$ cereal list lemur
    +af lemur /dev/ttyS1 115200 lemuradmin lemurviewers
    lemuradmin@scs:~$

### Attaching to a `cereal` session

Using the attach function will put the the user into a `cereal` console, provided by a command-restricted `screen` session. To detach from an attached session, use "C-\ d". For help on other available commands, use "C-\ ?".

    lemuradmin@scs:~$ cereal attach lemur

    Debian GNU/Linux 4.0 lemur ttyS0

    lemur login: root
    Password:
    Last login: Sat Oct  6 10:16:26 2007 on ttyS0
    Linux lemur 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64

    The programs included with the Debian GNU/Linux system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.

    Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
    permitted by applicable law.
    lemur:0:~# exit

    cereal console: lemur              2007-10-06 10:17              "C-\ d" to detach

    lemuradmin@scs:~$ cereal attach lemur
    [detached]
    lemuradmin@scs:~$

### Following a `cereal` session

You can use the follow command to easily view the logs of a cereal session.

    lemuradmin@scs:~$ cereal follow --cat lemur | cat -A
    ############################################################
    ### cereal log: lemur
    ### +af lemur /dev/ttyS1 115200 lemuradmin lemurviewers
    ### log file: /var/lib/cereal/sessions/lemur/log/main/current
    ############################################################
    2007-10-06_14:39:44.56895
    2007-10-06_14:39:44.56900 cereal: session 'lemur' started.
    2007-10-06_14:39:44.56900 starting screen session...
    2007-10-06_14:39:58.86707
    2007-10-06_14:39:58.86708 cereal: user 'lemuradmin' attaching to session...
    2007-10-06_14:40:02.57346
    2007-10-06_14:40:02.57348 Debian GNU/Linux 4.0 lemur ttyS0
    2007-10-06_14:40:02.57348
    2007-10-06_14:40:02.57349 lemur login: root
    2007-10-06_14:40:13.58131 Password:
    2007-10-06_14:40:14.58129 Last login: Sat Oct  6 10:16:48 2007 on ttyS0
    2007-10-06_14:40:14.58129 Linux lemur 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64
    2007-10-06_14:40:14.58130
    2007-10-06_14:40:14.58131 The programs included with the Debian GNU/Linux system are free software;
    2007-10-06_14:40:14.58131 the exact distribution terms for each program are described in the
    2007-10-06_14:40:14.58132 individual files in /usr/share/doc/*/copyright.
    2007-10-06_14:40:14.58132
    2007-10-06_14:40:14.58133 Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
    2007-10-06_14:40:14.58134 permitted by applicable law.
    2007-10-06_14:40:14.58135 Last was Sat 06 Oct 2007 10:33:15 AM EDT on ttyS0.
    2007-10-06_14:40:14.58135 ESC[31;1mlemur:0:~#ESC[0m exit
    2007-10-06_14:40:17.58525 logout
    2007-10-06_14:40:17.58525 ESC[HESC[J
    2007-10-06_14:40:18.58523 Debian GNU/Linux 4.0 lemur ttyS0
    2007-10-06_14:40:18.58523
    2007-10-06_14:40:18.58524 lemur login:
    2007-10-06_14:40:19.62101 cereal: user 'lemuradmin' detached from session.
    lemuradmin@scs:~$

## Conclusion

Now that you've created your first cereal session, you're ready to create serial sessions for all the rest of the attached serial consoles:

    scs:0:~# cereal-admin list
    +-f lemur /dev/ttyS1 115200 lemuradmin lemurviewers
    +-f chimp /dev/ttyS2 115200 chimpadmin chimpviewers
    +-f douc /dev/ttyS3 115200 doucadmin doucviewers
    +-f galago /dev/ttyS4 115200 galagoadmin galagoviewers
    +-f baboon /dev/ttyS5 115200 baboonadmin baboonviewers
    ...
    scs:0:~#

Cereal is developed and maintained by CMRG. Hopefully the package will be included in Debian soon, at which point bug tracking will be handled in the Debian BTS. Until then, please send all bug reports, questions, and suggestions to cereal AT fifthhorseman.net.

Last modified: Mon Oct 15 13:27:18 EDT 2007