The Computer Corner Take II (#36) by Bill Kibler

To see more Computer Corner articles look here: CCII page or check out the Home Page .

Programming for the BOBZ DACS

Some History and Info

I recently sold and shipped a number of old CP/M systems, which got me reviewing how we did things way back then. Seems like just the other day, not the almost forty years, that I was figuring out how to stuff a centronic print driver into 16 bytes of ROM. The computer industry has come a long way in those years and yet I still find myself doing things not much different than before.

My first experience with micro like systems, was while I worked at KBHK in San Francisco. I was assigned to install a new type of remote control system for the just installed transmitters at Sutro Tower. The controller was all 7400 TTL chips that later I realized was simply a CPU in chip form. Being a two foot square mess of wire wrapped circuitry it was no surprise it had problems, which took me and the design engineer many days to fix. What was in those two square feet of components, now fits inside a 2MM square chip, similar to what runs the BOBZ DACS. In fact, I would suspect that the BOBZ DACS could actually be used to control a TV transmitter with some I/O left over.

The BOBZ group, is two Bob's, that design, program, and build small systems based on using the SiLabs 8051 line of enhanced controllers. These small controllers have software that Bob Nash and Charley Shattuck developed - a small forth macro compiler system to enable fast and simple programs. They call it MyForth since it was intended to be something only they used to do their programming. It is still MyForth that allows BOBZ to produce very small and highly efficient systems. In the case of the DACS, and unlike other similar products, the DACS's MyForth uses plain text over serial to control and report data to the user.

This enhanced product features a number of inputs and outputs to enable analog and digital data collection. To make it more useful, they added four relay driver chips that can be used to control up to 48V relays. The real bonus however is the simple text string over serial command and reporting system. This makes using the system platform independent and accessible to any programing language. The manual shows how using a simple terminal program can have you using the unit within minutes of setting it up. A very simple setup in fact could be just a terminal program connected to the DACS, where you read and change outputs by using the built in command structure. However, what I intended to cover here is slightly more complex than possible with a terminal.

Programmer Startup

There are three ways you can use the DACS in a programming environment. Probably the most common and simple of tasks for the DACS is recording data. In this case you have several sensors connected and you record their state at some regular interval. You later can plot, chart, put the data in HTML tables, or look for changes to figure out when something failed. I will cover this use first, by showing how to setup regular capturing of the data using crontab in Linux and parsing into a simple HTML table.

The second use for the DACS is real time programming and operations. Here we need to monitor several inputs and turn on or off devices based on input states. This is a "state" engine as such and may or may not have logging enabled. I will show how to create a simple temperature system to keep water within some fixed range by turning on and off cooling or heating. I brew beer and need to make a water bath to hold the fermenting beer within a two degree range using a thermal pile. Should be a fun project that we can all drink to.

The last use of the DACS is a combination of the two previous, without the real time events. By this I mean use of logging in which one or more programs check the last entry in the log looking for action states. The "action state" is some event that requires some other event to happen, just not in real time. A simple case might be mail delivery, and all we want to know is that it happened and it is time to go get the mail. The issue here is low key events that we check once each half hour looking for state changes, we toggle some light somewhere to tell us to go and check our mail. Another example might be doors at a business that we want to know if they are open after the store closes and by counting the number of times they opened and closed during the day, we get an idea of traffic into and out of the store.

I will cover creating programs for all of the above, give examples, explain how and why I did what I did, and hopefully pass along some things to consider when you do your programs to use the DACS.

Choosing Perl

I plan on using perl with some shell scripting thrown in. My reasoning goes something like this, I have used perl for over 20 years now and have many programs done back then that are still running. I know it works and works well. In using perl all those many years, I also developed a programming style that enabled others to pickup and maintain what I did long after being gone. Since perl has been main stream for so many years, I feel the support is very good, and the number of library modules is extremely large. There are several e-books, lots of tutorials, and tons of how-to for almost any aspect of perl you can think of.

Why not use python? The simplest of reasons would be that everyone is using it now and thus I want to cover something that is just as good but not getting the attention it should be getting. Keep in mind that python and perl are somewhat similar and once you master one, learning the other is rather simple. Why not just use shell scripting? You can do almost everything with shell scripts, however it takes about two to three time the effort with shell scripts as it does with perl. "perl" was created to make life easier over using shell scripts, and as such, you will find many features that just can't be matched sticking to true shell commands. Don't forget that you can call perl from within a shell script as well as calling system commands from inside perl. For people with some scripting knowledge, using perl will seem like a logical step forward and makes life simpler.

Before we get into the actual programming, there are a few concepts we need to cover in some detail. The first one has to do with understanding clearly what we hope our program will do for us. Chuck Moore who created Forth once said "that if you can't solve your problem in 1000 bytes, you don't understand the problem." Over the years I have seen how true that statement is. Coders that clearly understood what was needed, wrote very concise and short programs. Those programs that ran on for pages and seemed to do nothing, were clearly from programmers who had no grasp of what needed to be solved. You will also find code out there from people who have yet to master their program language of choice. And don't be fooled by over complex samples of code that show how complex it can be made that no one, even the original programmer, can figure out what it does. Good coding should be well structured, easy to read and truly modularized.

Back in the early 70's when I first was learning to program, modularization was all the rage. It has since been replaced by object oriented programming, which to me is just a more formal approach to modules. No matter what you call it, the concept is still the same, a module should stand by itself, do one thing well, not know and not care about anything else. That last bit is really important, as a standard failing of new programmers is to add bits of logic into sections of code that are too specific to the overall task and thus should be somewhere else.

I did a project some years back in perl that read an SQL database and then performed and scheduled tasks based on the returned data. I was a contractor and the lead programmer was fresh from college. We had many discussions where the newbie thought that this or that specific task should be added. I stuck fast to my modular design and it worked well. So well in fact that several years later I had a chance meeting in a hallway with the now experienced programmer who informed me that my code was still working and that only one bug had ever been found. The bug was in the SQL code he wrote and not in my code. He thanked me for showing him good programing technique, as lots of changes had happened over the years, but due to my style of programming, it was never necessary to change any of my code. All changes were handled by adjusting the SQL code. Well organized, modular coding, produces simpler and cleaner coding which requires less maintenance as usage matures over time.

Some Simple Logging

Now that we have a few programming points settled and are looking at the DACS to capture some data, what do we do next. As I said above, we need to clearly understand the project and it's goals. My personal test setup for the DACS consists of the DACS connected to a BeagleBone by USB cable. The analog input is connected to a battery that is charged by a Harbor Freight Solar Array. I have two relays that toggle back and forth from commands and who's outputs trigger four digital inputs. When the states of all signals is printed out, you can see all the voltages and transitions from the setup. It is a good way to check that your code and the DACS are working together. Thus this first coding step is great for just checking out the DACS operations in general.

As I said before, we need to know and understand our problems, before we can solve them. What problems does the DACS present. First off we are dealing with a serial device and must know how to talk to it. When we used a terminal program to check it out, that was platform independent. However from now on, I will be talking about Linux and primarily the BeagleBone. There is little difference for what we will be doing between the BeagleBone and RasPi, or for that fact even a full size desktop running Linux. My comment to Windows user is get a real operating system, as you have no idea what your missing. I must state that most of what we will be doing is perl based and there are versions of perl that will run on Windows. It is just things like shell scripts and crontab that are native to Linux that will need to be found and added to Windows(try cygwin).

BOBZ is shipping the DACS with a serial to USB converter and as such you can just connect the converter to the USB port and Linux will see the new device and should assign it a device name. The typical device name, assuming you don't have any other USB serial devices will be "/dev/ttyUSB0". This is a Linux file name and works much like any Linux "device" file name. However, and this a big "however", as it must be treated differently than regular files. You can sort of pipe things to serial ports, but to really do things properly, you will need a special program to talk to serial devices. In our case there are several things happening, you must issue a command, get a response, accept characters until you get a ">" prompt, that means "all done and ready for next entry". All normal programs on Linux will expect a newline prompt, not the ">", to mean end of data sent. So a special program or utility is needed.

So let me get this out of way now by saying you can but shouldn't try to do this using shell scripting. I recently checked out several forums and they presented one or two ways of using shell scripts to get data from serial ports, and in all cases said, it was a bad idea, just use perl or a "C" program. In the forums, the requester always said they didn't want to learn a new language, as if shell scripting isn't a programming language. Shell scripting is a programming language, albeit a bit simpler than most, but still a programming language. Note the "a bit simpler" as it is important to remember that many of the functions provided in shells are similar but less full than you get using something like perl. To do the same in a shell may require hundreds of lines of doing this and calling that, while the same task in perl will be one or two lines of function calls. I personally found learning perl easier than learning shell scripting, if for now other reason than everything was in one book. That is due to shell scripting calling many programs that each have their own man pages.

I could provide you with a C program to read the DACS, but it would require you to compile the program on each version and platform of Linux it gets used on. However, I can provide the same functional results, by providing one perl script that does the same job and will run on every platform. Do you need to know perl to use the program? Absolutely not! Consider this the same as running cat, only it returns data from the DACS and not a file. The only problem with perl is using the library "Device", which is not normally loaded with most distributions. Here is the code and note the "apt-get" line, you may need to do it before running the script:

#!/usr/bin/perl
#
# util to pass command to dacs board
# > ./get_dacs.pl /dev/ttyUSB0 h
#
# may need to do "sudo apt-get install libdevice-serialport-perl"
use Device::SerialPort;

# get commands to use
($device,$cmmd, $other) = @ARGV;
if($device eq "") {
  print "usage: get_dacs.pl device command ( /dev/tty??? for dacs )\n";
  exit(0);
}
if($cmmd eq "") {
  print "usage: get_dacs.pl device command ( command to send to dacs )\n";
  exit(0);
}

my $port = Device::SerialPort->new("$device");
$port->user_msg(ON);
$port->databits(8);
$port->baudrate(19200);
$port->parity("none");
$port->stopbits(1);
$port->write_settings;

# needed to clear buffer
$port->lookclear;

# do send command - get ?
$port->write("$cmmd\n");

# get results and print it
do_print () ;

exit;

###### subs ########
sub do_print {
  while(1) {
    my $byte=$port->read(1);
    if ( $byte eq ">") {
       last;
    }
    if ( $byte eq "\r") {
       $byte="";
    }
    print "$byte";
  }

}

Data Logger

In the above sections we covered needing to create our own perl script to access the DACS serial data. The "get_dacs.pl" perl script sends a command to the DACS and prints what is returned. You can run it from your command line and try all the commands covered in the manual. It basically is a one command terminal program designed to match the way the DACS reads and responds to commands. What we want to do next for this article is turn the DACS into a data logger. I started by hooking up some voltages to the analog inputs, connecting two relays to the outputs, with their contacts connected to four of the digital inputs. The idea was simply to test all the ins and outs and make sure they changed regularly. To get them to "toggle", I added an extra command that would not be a normal "logger" command.

Let me talk about what to record when logging data. If I only had one or two items connected, clearly I would only want those in a log. With the DACS, we have the "s" command which prints the entire set of values. Using the DACS "s" command, it is possible to get all the readings on one line, and in my case that works perfectly. I often suggest to clients that they get as much data recorded as possible, since you never know when something may fail and thus you need all the data available. The same stands for removing data after say one week or so. Data storage is cheap and the more you have the better it will be to figure out what happened a long time ago. I have found that failures usually creep up on you and having a long trend of data will show just what really happened. So save as much as you can for as long as you can. I have some logs that go back 5 or more years and are only a few megabytes in size.

There are two ways we can capture our data into a file and one is using the get_dacs.pl to read each input we are using. The other option is changing the perl script to do what we need. What do we need? For starters we will need the line to start with the date and time, so we know when this specific line was captured. Then we will want the data items, and fortunately the way the DACS works, we get an echo of the command before we get the data. Lastly we want each line to end with a newline character, so all our data is on one line. That means any newline or carriage returns need to be removed before we add our data to the log file. We will put all these into a shell script that is called by the crontab for the user reviewing the data log. An example is below:

# get_dacs.sh
# tr is used to remove or replace the newline
date +'%Y-%m-%d %H:%M:%S' | tr "\n" " "

get_dacs.pl /dev/ttyUSB0 v0 | tr -d "\n"
get_dacs.pl /dev/ttyUSB0 i0 | tr -d "\n"
get_dacs.pl /dev/ttyUSB0 tf

2014-05-31 16:28:19 v0 1.379  i0 1  tf 84.5 

02,32 * * * * /home/kibler/bin/get_dacs.sh >> /home/kibler/logs/get_dacs.log

NOTE: You could use shell scripts to do the above, but there are several points to be made as to why you should not use shell scripting. One reason is that shell scripting is coarse while perl is finer grained. Shell works best on larger structures, like capturing lots of data. For the DACS and by using perl, we have finer control over the serial transfer and changing or removing "bad" characters. Normal testing using perl scripting will turn up minor problems that usually can be fixed with very little code change. I have had cases with shell scripting where a minor error required a complete re-write. You will find it is pretty hard to control the serial communications with shell scripting, while perl gives you just the level of control to be sure your getting all the data you need.

We can as well, alter the basic perl script to provide the date, removing newlines, and any other problems we might have found using the get_dacs.pl program. Let us assume that most of the I/O is being used and therefore we can use the "s" command to get all our data points - used or not. The changes are minor and the results is get_dacs2.pl:

#!/usr/bin/perl
#
# util to pass s command to dacs board
# > ./get_dacs2.pl /dev/ttyUSB0
#
# may need to do "sudo apt-get install libdevice-serialport-perl"
use Device::SerialPort;

# get device dir to use
($device, $other) = @ARGV;
if($device eq "") {
  print "usage: get_dacs2.pl device ( /dev/tty??? for dacs )\n";
  exit(0);
}

# do first - if port down - will exit and do nothing...
my $port = Device::SerialPort->new("$device");
$port->user_msg(ON);
$port->databits(8);
$port->baudrate(19200);
$port->parity("none");
$port->stopbits(1);
$port->write_settings;

# get start time of run
$date = `date +'%Y-%m-%d|%H:%M:%S'`;
chomp $date;
print "$date|";

# needed to clear buffer
$port->lookclear; 

# do send command - get all values   
$port->write("s\n");

# get results and print it
do_print () ;

# do toggle of output
$port->write("o~\n");
do_print () ;

print "\n";
exit;

###### subs ########
sub do_print {
  while(1) {
    my $byte=$port->read(1);
    if ( $byte eq ">") {
       last;
    }
    if ( $byte eq "\r") {
       $byte="";
    }
    if ( $byte eq "\n") {
       $byte="";
    }
    print "$byte";
  }
}

2014-05-31|20:02:02|s  | 1 1 1 0 1 0 | 0 0 0 0 | 1.277 1.277 1.277 1.277 | 29.3 84.8 |   o~   
2014-05-31|20:32:02|s  | 1 1 0 1 0 1 | 1 1 1 1 | 1.268 1.268 1.268 1.268 | 29.7 85.5 |   o~   
2014-05-31|21:02:02|s  | 1 1 1 0 1 0 | 0 0 0 0 | 1.273 1.273 1.273 1.273 | 29.0 84.3 |   o~

Creating a web page

Using perl makes it pretty easy to create a web page from the data log. I have been doing this for about 20 years now and as such don't use any of the perl libraries for creating web pages. You can use my example, or you can use one of the many perl modules to do it for you. Simply put, the perl script uses a system command to get the last 48 lines of data from our log file. It then parses each line and places each field in the appropriate table location. The output is a HTML file with a simple header, a table of data points, and nothing more. You can use the example as is, or adjust it in many ways to create a much more complex page. You can even create charts as an image and add one print statement to include that image, as I have found that it helps the image if the data used to make it is also included on the page. Here is the code:

#!/usr/bin/perl
#
# get list of DACS values from get_dacs2 log file
# and put into html table for inclusion into page...
#

# get commands to use
($source,$destination, $other) = @ARGV;
if($source eq "") {
  print "usage: dacs_tbl.pl source destination ( source is path and file name of log file)\n";
  exit(0);
}
if($destination eq "") {
  print "usage: dacs_tbl.pl source destination ( destination is path and file name of html file )\n";
  exit(0);
}

# get start time of run
$date = `date`;

@TMPLIST=(`tail -n 48 $source`);
# debug option
#print "@TMPLIST";

open ( FINDEX, ">$destination");
print FINDEX "<html><head><title>DACS Report</title></head>\n";
print FINDEX "<body><h2>DACS Report $date</h2>\n<p>\n";
print FINDEX "DACS report using Summary DATA \n<p>\n";
print FINDEX "<br>\n";
print FINDEX "<br> <b>DACS</b> = Data Acquisition And Control Board by BOBZ\n";
print FINDEX "<br> <b>NOTE:</b> data obtained by using the command s\n";
print FINDEX "<br> <b>NOTE:</b> not shown is command o~ which toggles outputs each reading\n";
print FINDEX "<br> <b>Volts In:</b> is battery voltage from Harbor Freight solar array.\n";
print FINDEX "<br><p>\n";
print FINDEX "<table border=0 cellpadding=4 cellspacing=4>\n";
print FINDEX "<tr><th>Date</th><th>Time</th>";
print FINDEX "<th>Inputs</th><th>Outputs</th><th>Volts In</th><th>Temperature</th></tr>\n";

foreach (sort @TMPLIST)
{
        chomp;
        ($tdate, $ttime, $tt1, $tt2, $tt3, $tt4, $tt5, $junk) = split(/\|/, $_);
        print FINDEX "<tr><td>$tdate</td><td>$ttime</td><td>$tt2</td>";
        print FINDEX "<td>$tt3</td><td>$tt4</td><td>$tt5</td>";
        print FINDEX "</tr>\n";

}

print FINDEX "</table>\n";
print FINDEX "<br>\n<p>\n";
print FINDEX "</body>\n</html>\n";
close FINDEX; 

exit(0);

02,32 * * * * /home/kibler/bin/get_dacs2.pl /dev/ttyUSB0 >> /home/kibler/logs/get_dacs.log
03,33 * * * * sudo /home/kibler/bin/dacs_tbl.pl /home/kibler/logs/get_dacs.log /var/www/dacs_tbl.html

DACS Report Sun Jun 1 14:03:01 PDT 2014

DACS report using Summary DATA

DACS = Data Acquisition And Control Board by BOBZ
NOTE: data obtained by using the command s
NOTE: not shown is command o~ which toggles outputs each reading
Volts In: is battery voltage from Harbor Freight solar array.

Date            Time            Inputs          Outputs	        Volts In                       Temperature
2014-05-31	14:32:02	1 1 0 1 0 1 	1 1 1 1 	1.325 1.326 1.326 1.326 	27.2 80.9
2014-05-31	15:02:02	1 1 1 0 1 0 	0 0 0 0 	1.322 1.322 1.322 1.322 	27.2 80.9
...
...
2014-06-01	13:32:02	1 1 0 1 0 1 	1 1 1 1 	1.331 1.331 1.331 1.331 	28.0 82.5
2014-06-01	14:02:02	1 1 1 0 1 0 	0 0 0 0 	1.325 1.324 1.324 1.324 	28.0 82.5

That pretty much covers the first case of using the DACS. The next article will cover using the DACS in a state engine, followed by another article that shows how to use log files to send emails and toggle alarm relays. Please keep in mind that the three perl scripts can be used just like any Linux program. You don't need to know how perl works or even any perl syntax, just use them like "cat" or "cut". And like the normal Linux tools, should there be a problem, just email me, the maintainer, and we will see what went wrong.

Links

The basic perl script get_dacs.pl
Enhanced perl script using the "s" command
Produce a HTML table web page from log data