This article is a continuation of the previous (ccii_13) and is being done to provide the needed information or knowledge to create a web page server for collected data at a remote site. In ccii_13 we used a Sylvania $99 netbook and converted it from the WinCE OS to Debian. When we left the article that spent most of the time discussing how to put debian on the 4Watt system, it had a minimal Debian operating system. Clearly more programs are needed to make it a web server, as well as a data collector.
The main focus of this article is the netbook and simulation of a remote site, such as a solar powered ham radio repeater on a mountain top. The information and steps are the same for any minimal debian setup and data collecting system. Very little of what we talk about and show is so specific to our project that you will not be able to transfer the information.
The equiptment is the Sylvania Netbook - SYNET07526-R - but could be any laptop or small system that you have sitting around. Two items are used that I have been using for some time, the Phidgets and Temperature systems. The two data collectors are specific to this project, but one cost $88 and the other $30, and thus keeping your expense reasonable. The fact that one is USB and the other serial but USB connected will be rather generic and similar to any type of data collection you might chose.
We will make one assumption to start with - you have a minimal system up and running. In ccii_13 we left the article after doing the initial install and little more. We assume your logging in to the system from the local keyboard and seeing results on the screen. If it is the netbook, your rapidly finding the keyboard way too small and the screen less than desirable. So we have a few items we need to do first.
At this point you want to make sure you have the latest versions of the Debian OS, since any number of upgrades could have happened between the creation of the installation image and now. We do that by running "apt-get update", followed by "apt-get upgrade". You might get an error message about a missing logging directory and if so, make it before moving on to the next step. We have assumed as well that the networking is properly setup, if not you will have to do that first - check the links below for a good Debian Networking setup site.
The next step is fixing the overall work situation and getting you doing things remotly. We will use the standard for remote access these days - SSH - secure shell. We do that by adding it to your set of tools - "apt-get install ssh". It will take a few minutes to download and be installed. Next we want to make sure it works and move the rest of our tasks to a full size workstations. On your workstation, do "ssh root@your.ip.of.machine". In setting up the netbook, you will have put a static IP address into the system and from now on, that will be the IP used to connect to it. If you followed the minimum instructions correctly it will ask for your root password after asking you to accept the new connection.
For some of you this will be a completely new experience - remotly accessing a server. This however is how I do my work everyday. For windows users, you will need something like "puddy" to do the ssh connection, while linux users will do the work in a terminal session. There is no point and clicking as in windows, although you can move data from one terminal session to another by using the mouse much as you do in normal windows mode. For beginners it will take time to master it, but you will find it a very fast and powerful way to work.
The next step is setting up the web server - in this case Apache2. Like before, we run the udate tool - "apt-get install apache2 perl" - and load both apache2 and associated perl modules. I like using perl for creating web pages on the fly. Most of the tools I will use, I developed some time ago and are in perl. You might consider tcl or java. I like tcl as it is small and as good as perl, while I would not recommend java due to the size of libraries it might load on your system. Use of java scripting within your web pages, however, does not require it on the server as long as you do not make any server side java requests.
After installing apache2, you should be able to point your web browser at the IP of the netbook and see a "it works" web page. We can find the web page at "/var/www/index.html". You might edit this page using "vi" the normal editor for unix/linux systems. The changes you make will be seen in the browser after you save them and refresh the browser. I enhanced mine by cutting and pasting the output of some commands into the web page.
I might point out at this point that you do not need apache to get data from a remote site. I manage several git repos using gitolite and have learned all about using ssh. Gitolite uses ssh with public and private keys that are linked through ssh to a given program. When a defined user ssh's into the system, ssh finds the matching key in the keytable and runs the program that is asociated with that key. You could use this feature of ssh to return a string of data back to the user and disconnect after the transfer. Thus the collecting system or web server would use a cron job to trigger a ssh connection, say every half hour, and take the returned data and update the database or web page. This is a clean, safe, and protected method for getting data back that doesn't expose your site to exploits.
You now have a running Web Server and are ready to move on. Since the objective is data collecting we need to install and assemble all the parts and hardware to make it work.
This whole project is based on using USB devices to collect data and using the stock debian linux for ARM to talk to these devices. At this point in my testing, I quickly and without much effort, got debian running on the netbook. Next step was to checkout the USB access, which for some reason didn't work at all. I will do more testing later, the short of it was no USB action at all. I loaded more utilities - you will need "usbutils" to get "lsusb" - which only showed the internal hub and nothing more. So I tried upgrade, dist-upgrade, and lastly changed the "/etc/apt/sources.list" to "testing" and after a dist-upgrade, on reboot crashed the system.
At this point I need to re-do the entire install again and start over, but as I read more about changing the netbook to debian, I read instructions on doing "SD+USB". In the SD+USB method, the SD device holds the kernel and start script, while the ext2 files system is on a USB flash drive. The more I considered this option, the more I like it, especially for the initial testing and setup phase of the project. I have a number of long term design concepts and it would seem using a USB drive for startup and testing makes more sense.
The advantages for me using one of my old 40GB USB hard drives over the internal or external flash are many. Easy to backup, no concern about space limits, not worried about burning the flash up, can move from netbook to workstation very easily, can test many design concepts all on one device. So when I followed the instuctions and booted up using the SD and my 40GB USB hard drive, it didn't work. I checked the instructions and saw the wait time was "0" not "10" as instructed. So I changed the "cmd" file, ran the build script to make a new scriptcmd and that worked just fine. I did apt-get install for usbutils and "lsusb" showed all my USB devices properly.
I believe the issue must be different kernel images in the fatpart but not sure. In either case the next issue is getting kernel modules for my devices loaded, as you only get the needed modules for screen and wifi. One advantage of using a large USB drive, is plenty of space to load and build a new kernel from souce on the actual machine. You will need to be able to build, not only the kernel, but other programs to get the USB device data collecting going. So the next step is adding gcc, make, and other tools for building programs.
I got the kernel source from the bento-linux links and loaded it on my workstation, which I then sftp'd it to a /work directory of the hard drive. Making the /work space, allows me to have several copies of everything in one place for later backing up. I often tar up my work space and sftp the tar file to one or more workstations - thus saving off my work at various points in time. You will need to "apt-get bz2 gcc-4.4 make" to do anything with the source. At this point you should be able to use vi pretty well, but with the hard drive, adding X or more tools that are more to your liking is OK, as they will not be part of the final install.
Some time has passed since I wrote the above and I have been working both on and off on the project. I have been succeding and getting things to work, but had many failures and changes in process. I tried rebuilding the tarred version of the source code without much luck. I found the git repo of the source and used it with some success. Overall I feel I am getting close to the objectives. Let me say here and now, that most of the steps will require a bit more than just a passing shot at the needed process. I am succeding because it is somewhat normal work for me. New users or those with limited developer skills will find lots of stumblings blocks that require more skills than you might posses.
Here is an example of the issues you will be faced with - the netbook git version is 1.5 and I wanted to use the kernel source from github. I had no problem cloning the repo on my workstation, but the netbook version is too old to work properly. I then upgraded the netbook version and recompiled it for the ARM, only to run out of memeory when attempting to do a clone. I resloved the problem by tarring the repo up on the workstation after a good build and using that source tree to compile the needed modules.
To achive my goal of using the netbook to collect USB provided data, you need the kernel modules for each of the devices. The normal kernel does not have very many devices of any type available for loading - remember you can load kernel modules on demand if needed - they do not need to be pre-compiled into the kernel to work. To get the needed modules you will need to recompile the kernel source tree after "turning on" the appropriate setting in the config file.
I was having some issues creating the modules and getting them to load - I found it was necessary to use the "modprobe -f module_name" as the modules kernel string name was not the same as the running kernel. I dug around and re-read the kernel "README" file on the github site, to discover a few missed commands. Using those I then got a correct module loading - so re-reading the docs may solve some of your problems as well. Here are the correct compile options and notes:
GIT repo of WM8505 Kernel Source - the README.
(use this command to change the modules currently being created) make menuconfig ARCH=arm (there are some pre-compiled objects needed, do this to copy the binary blob .os in to place) make via_obj (when on a x86 machine with the "embedian" toolset installed - this builds the uzImage.bin) (the kernelrelease value is need to create modules that will load properly) make ubin modules ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- KERNELRELEASE=2.6.29-00236-g4f8dbbb-dirty (to do a non-cross compile on the netbook use this command) make ubin modules ARCH=arm CROSS_COMPILE= KERNELRELEASE=2.6.29-00236-g4f8dbbb-dirty (To install the modules somewhere) make modules_install ARCH=arm INSTALL_MOD_PATH=/some/staging/path
I would suggest you do most of your building on a workstation and copy the files over. The workstations took a few minutes, while the netbook will take an hour or so. When using the copy of the workstation build source tree, most operations on the netbook are short, as it is only rebuilding the changes that you made in the config setup. I did look in "/usr/bin" on the netbook and noticed the "arm-linux-gnueabi-*" tools, which means it might build ok using the cross_compile settings without "nulling" them out.
There are two files you should edit, the "/etc/modules" file to have your new modules loaded at boot time, and "/lib/modules/2.6.29-00236-g4f8dbbb-dirty/modules.dep" if your modules requires other modules loaded first. I did those and my modules now load at boot time correctly. You might check "dmesg" after a reboot to see if all went well. USBserial appears to be compiled into the kernel and doesn't need to be loaded - but then I am using the SD+USB boot options.
At this point you should have a kernel source tree that you can compile and produce needed kernel driver modules. I need the mos7720 dual port serial to USB adapter and after loading the newly compiled module, did a serial test to check operations. What I noticed was baud rate problems, if I went full tilt. I loaded minicom and it only supports three baud rates, 9600, 38400, and 115000. I saw dropped characters and when using the port as a console with "getty" had numerous issues making it unusable. I suspect this is all memory and processor restricted issues that I hope will not be a problem later.
The next phase of testing will require the user side of the data programs. The modules or drivers allow other programs to get data from the USB devices, and it is those other programs we need to buld now. In the case of "phidgets" a complete set of library modules need building and installing. I compiled those libraries and installed without issue. I compiled a variation of the examples "ifkit" utility and it ran, but produced low values. I have since done some checking and have yet to find out why there is a difference from running the same libraies on a x86 machine and running on the netbook. I can only assume some minor changes in system or math libraries that an upgrade might solve. However for now I adjusted the scaling values in the utility that fetches the values and prints them in usable format.
I have several other tools that I normal would use, but a shortage of usb ports limited my selection. I could use a hub, but ran into problems when trying to use one. I feel this particular kernel has issues with hubs and the hard drive it runs on. I will experiment using hubs later, but for now one other usb devices will be fine. I tried the mos7720 based usb to dual serial port adapter, only to find no "ttyUSB?" being made automatically. I had to do a "mknod /dev/ttyUSB0 c 188 0" and "mknod /dev/ttyUSB1 c 188 1" to get anything to work. The dual serial ports are working fine now and all that remains is tweaking the web pages and adding a few final touches.
The overall project has been a lot easier than my first impressions. The inital setup using pre-done debian systems was extremely easy, to the point that most somewhat knowledable linux users could do it. The difficulty is in building a new kernel to get drivers for using USB devices. It took some trial and errors before I understood the kernel building issues and got mine to work (I will be posting the tar file for others to use). Once you have a system working, using apt-get to install new tools is pretty standard, and seems to solve most problems. The standard "configure, make, install" for programs worked fine on the few I used and makes me think you will encounter few if any troubles loading more porgrams.
What remains for me, is figuring out just how I want to use it for real work. Most of what I covered so far has been testing the overall system for issues, of which I found no show stoppers. The next step for me, is figuring out just how and which features I want to use. Since it is using flash for normal storage, do I let anything be written to it normally? Do I add a flash drive for storing data? Use a RAM-Drive for the var file system? How about using NFS mounted drives for all disk accesses? Consider the SSH option, such that nothing is ever written to flash, but every half hour a remote site does an SSH query that spawns a script to collect the data and all output goes to the remote site with nothing being saved locally?
Overall there are lots of options to consider and each one has plenty of pros and cons to ponder. Which way I actually go has yet to be determined and most likely is going to take me some time to settle on. I am not in any hurry, and in fact have lots of little things to do first, so it might be a while before I do the next article. That next article will most likely be all the answers to the above questions, but I do have a IPAD knock off to play with and might just document that saga first.
Till then enjoy!
Links
The SYNET07526-R on-line - see web pages from the netbook.
Bento Linux downloads for linux on wm8505
Great Details for WM8505s - serial output and much more!!!!
Google Group - linux kernel for vt8500/wm8505
debian-administration - good site for help articles.
GIT repo of WM8505 Kernel Source - where to clone the kernel source from.
The wm8505 kernel - built, tarred, and zipped on the netbook.
archives_debs.lst - a list of all the deb packages loaded to date.