RTLinux has become a serious alternative for PC-based real-time control and acquisition systems. Our work explores its performance in embedded controllers for distributed applications. The proposed architecture is based on a network of diskless RTLinux nodes, booting from a Central Workstation. A set of experiments were made to test the feasibility of this approach, using the standard Linux TCP/IP network interface, under different network and computing loads. Alternatives to reduce communication overhead are discussed.
Modern laboratory automation systems are rapidly evolving towards highly distributed configurations. Such topologies provide modularity, reliability and resource sharing at reasonable costs. Traditional approaches to distributed data acquisition systems usually rely on vendor specific real time kernels and proprietary network interfaces. However, the actual tendency to software standardization and the ever-decreasing cost of personal computer (PC) hardware suggest interesting novel approaches to experiment automation.
In the last decade, Linux has emerged as a serious alternative to commercial operating systems. It offers a high quality POSIX environment, with hundreds of drivers and free availability of system sources. Being a general-purpose, time-sharing OS, Linux doesn't perform well for real-time, interrupt based applications, although some real time extensions already exist. Among them, RTLinux provides consistent real time performance, still retaining many Linux attractive features, such as wide connectivity and extensive peripheral support.
This work presents results on the use of RTLinux for embedded distributed data acquisition applications in experimental physics. The proposed architecture consists of a network of diskless RTLinux-based embedded processors, sharing results and configuration parameters with a Central Workstation. The resulting system has many interesting features, regarding development costs, upgrading possibilities, availability of software tools, and flexibility. Not less important, this alternative allows recycling of PC hardware which, being obsolete for desktop applications, is very well suited for embedded control scenarios.
A basic embedded controller node consists of a standard PC motherboard (without keyboard, video or local disks), a standard Ethernet interface, a boot EPROM and the I/O adapters for the specific application. At least four megabytes of RAM are required for the operating system and another four megabytes are used as RAM disk.
This system boots a small RTLinux kernel via tftp, from a bootp LAN server running on the Linux-based Central Workstation, through the mechanisms provided by the Netboot or Etherboot packages. These packages allow booting the PC using a boot EPROM installed in the network adapter. The boot file comprises the kernel image (eventually with NFS support) and a RAM-based root filesystem image. It also includes the RTLinux extension. Standard Unix functionality is provided by Linux, including common networking applications such as telnetd, ftpd, httpd, etc. Figure 1 depicts a block diagram of the proposed architecture. In this application, an ethernet switch separates experiment and global traffic, although this could be avoided if network activity is low.
The resulting embedded controllers can be rack mounted, have no local user interfaces, and are remotely monitored from the Central Workstation's console. Real-time tasks run in a multiuser, interrupt-driven, priority-based scheme, using the facilities provided by the RTLinux kernel. Although RTLinux is not POSIX.4 compliant, it provides a flexible platform to interact with hardware that presents temporal requirements. Its modularity allows the addition of different real-time device drivers and scheduling algorithms, such as Rate Monotonic, EDF, etc. In addition, this framework provides non real-time POSIX.1 functionality via Linux, without degrading local real time performance.
This configuration has many practical advantages. The PC motherboard is reliable, inexpensive, and in many cases is already available as a by-product of desktop machine's upgrades. The general-purpose features of Linux provide standard and powerful compilers and debuggers, network daemons, standard file access, and extensive documentation. The availability of device drivers for standard hardware, and device driver development tools for non-standard hardware is a key feature for the selection of this operating. For instance, each embedded node can run a http server, making experiment results widely available on-line, using a uniform graphical user interface, provided by standard web browsers. All software development can be done in the Central Workstation, and made available through NFS.
Multiple Embedded Controllers can be served by a single Central Workstation. This Linux-based workstation also provides the Data Aquisition's Graphical User Interface (GUI), Internet access, mass storage and backup facility.
Given the POSIX.1 functionality provided by Linux, multiple tools are available for GUIs. In the current application, a Xwindows-based environment was designed, using Tcl/Tk.
It must be noted that real time performance is restricted only to the embedded nodes and not to the whole system, due to the network technology used. Experimental tests on RTLinux performance and interrupt latency for different platforms can be found, for instance, in RTLinux. In those time-critical applications where the RTLinux response is not fast enough, a hierarchical real time structure can be devised, using embedded microcontroller boards attached to the PC bus.
One of the real-time limitations of this proposal is the Ethernet-TCP/IP-based network structure, which is not well suited for real-time applications. Moreover, in those embedded nodes having NFS-mounted file systems, file operations for data storing/retrieval and communication between processors can't be performed in real time, and NFS swapping must be strictly avoided. This situation is partially alleviated providing a small ram disk root file system at boot time, as part of the boot image. The acquisition system has to be designed with these constrains in mind, using the network only for non time-critical operations and monitoring. Still, this arrangement can be useful in a wide spectrum of applications.
In order to study the performance constrains imposed by the TCP/IP network structure, a set of experimental tests were devised over the architecture of Fig.1, using standard Linux for the network tasks and RTLinux for local real-time tasks, under different computing and network loads. Following the philosophy of the experiments reported in RTLinux, where RTLinux was evaluated in a single node, these tests measure the time required for one embedded node (rtserver) to recognize an I/O interrupt on another node (rtst1). Figure 2 shows the experimental setup, while Figure 3 shows the results for the following load conditions:
* No network load, no computing load. The nodes are running only their real-time tasks, and the network is only used to transfer results. However, of course, there exist broadcast messages (ARP, etc.) from the global network thorough the switch. This traffic is present in all tests.
* Apart from the real-time task, mixtures of floating point, file system access and network tasks are running on all nodes. This is an attempt to measure interrupt response under known congestion conditions.
Details on test settings and histograms for each load condition are shown in RTLinux.
All tests were run for 10000 interrupts each. Regarding Figure 3, in all cases the response time is below 100 ms. This result can be considered acceptable in many "soft" real-time scenarios. The main factor degrading performance is local load, and specially filesystem load. As networking is run as a standard Linux (non real-time) task, network behavior is not affected by the presence of RTLinux.
The best way to reduce communication overhead is making networking a real time task itself, avoiding filesystem access. As an illustrative example, the same experiment of figure 3 was run using a simple point-to-point protocol via the PC parallel port, as a real time task under RTLinux. The results, shown in Figure 4, are basically the same as reported in RTLinux for standalone systems. The response time is basically reduced by a factor of 100.
Fig. 3. Communications performance using standard Linux network facilities. Network delay is allways below 100 ms. Top: heavy computing/network/filesystem load. Bottom: only real time tasks.
Fig. 4. Communications performance using a point-to-point protocol under RTLinux. Worst case delay is under 1.0 ms. Top: heavy computing/network/filesystem load. Bottom: only real time tasks.
The proposed architecture is fully functional at present, and is being used with success for physics experiments data acquisition and automation.
Test results show that, although network response is acceptable for many applications, a network interface running as a real time task under RTLinux (not available at present) can provide better time response, opening the door for truly distributed real time applications. Currently, real time performance is restricted only to local nodes. Work in this direction is now undergoing.