Ayers, J. (1992)
Desktop Motion Video for Scientific Image Analysis.
Advanced Imaging 7: 52-55.

Desktop Motion Video for Scientific Image Analysis.

Joseph Ayers
Marine Science Center
Northeastern University
East Point, Nahant, MA 01908

The emergence of the discipline of Neuroethology implies that neuroscientists are willing to tackle the daunting challenge of explaining animal behavior in terms of the properties, connections and especially the activity of motor networks in the central nervous system. Neuroethological experimentation requires the integration of neurophysiological approaches with behavioral approaches. Neurophysiological data typically consists of the measurements of trans-membrane voltages, currents or their resultant extracellular field potentials recorded as a function of time. Behavioral data, however consists of measurements of the kinematics of the motion which results form the action of muscles joints and/or hydrostatic skeletons. Neurophysiological and motion data are therefore different media and their correlated analysis therefore requires multi-media techniques.

This problem is not unique to neuroethology. For example during angiography one injects dye into the cardiac circulation to visualize obstructions of the coronary arteries. The diameters of these arteries is also determined by the state of contraction of the cardiac muscle. Thus the image of the arteries must be obtained at a particular point in the cardiac cycle which is defined by the electrical signal the electrocardiogram. Thus angiographic analysis could be enhanced by multi-media techniques. Another application of these techniques is the analysis of performance in sports medicine.

Neuroethological analysis presents a unique problem since the most interesting behavior is typically spontaneous. In other words, under the most carefully designed and controlled experimental conditions the animal will exercise its own volition. Thus the experimenter can only create conditions which will optimize the occurrence of the behavior in question in the sampled data. As a result, acquisition devices must be capable of acquiring large amounts of image and neuro-physiological data.

Mulitmedia neuroethological experimentation is, at present, a multi-stage process involving acquisition, browsing, cataloging, digitization, analysis and ultimately visualization displays. Videotaped data must first be browsed and interesting epochs identified. To be subject to automated analysis, the interesting epochs must be digitized. The kinematic raw data can be stored as digitized "movies" while analog data is typically stored as binary files. Digital movies must be subjected to frame-by-frame analysis of the relevant kinematic parameters and the electrophysiological data may be subjected signal processing to extract relevant electrophysiological parameters or viewed in raw format. Finally, the results of kinematic analysis and signal processing must be presented in graphical displays which allow visualization of the interrelations between electrophysiology and behavior.

The above discussion was intended to introduce the fundamentals of neuroethological analysis . I will now address current technology and how we expect to to proceed in the future. The optimal device for acquisition is the High-8 video tape transport. Current technology acquires high resolution (>400 lines horizontal resolution) video images and digital or pulse-code modulated stereo audio. During neurophysiological experimentation. one can simultaneously record the kinematic information on the video signal and electrophysiological data on the analog tracks. The analog recording circuits of most consumer grade transports damp transients of less than 40 hz so if one wants to use this technology to record slower phenomena there are two alternatives. One approach is to use voltage to frequency converters to convert the slowly varying signal to a frequency. During playback, a frequency to voltage converter can be utilized to decode the frequency back to a voltage. Several vendors have developed multichannel versions of this technology which can also record multiple analog channels on the video signal of the tape. An alternative technology is the use of digital meters which superimpose a digital version of the voltage level on the video signal.

Browsing and cataloging of videotape-based data can be readily accomplished in the Macintosh environment. Abbate Video Consultants has developed a Hypercard-based videotape logging and assembly system which operates through the Control-L or VISCA interfaces of Sony VCR's and CamCorders and has been extended to a broad variety of transports and, in fact the Windows environment. The VideoToolkit environment allows interactive browsing of videotape data using both mouse and keypad controls to establish a Hypercard database of the interesting epochs within a tape. This system provides a ready solution to the most fundamental problem of neuroethological analysis; i.e. keeping track of sparce results in a sea of data.

As a platform for the analysis of video-based neuroethological data, we extended the NIH Image program to support color acquisition and video tape control. The color segmentation and VCR control capabilities of an earlier version of ColorImage developed in collaboration Garth Fletcher have been described in an earlier article in Advanced Imaging (Nov, 1990). An example of the sort of direct kinematic analysis supported by ColorImage is indicated in Figure. 1. In this experiment we quantified the dyamics of the movement of the digestive apparatus of a larval lobster. The quantified parameter was the length of the pyloric region of the lobsters stomach. ColorImage allowed us to digitize a videotape of lobster feeding behavior, create a digital movie and to determine the length of the pylorus using a click-drag procedure in each of the frames of the movie.. The results of this analyis are presented graphically in Fig. 1.

Figure 1. Analysis of Kinematics from Video

In more recent versions we have added multimedia capabilities which allow correlated acquisition of kinematic data from the video channel of video tapes and the electrophysiological signal from the audio channels. We integrated a rewind and search algorithm into ColorImage which would initiate digitization of "movie" or analog data as the Control-L interface encountered counter transitions or as an A/D converter detected saturated noise bursts on one of the audio channels. These techniques make it possible to provide precise synchronization between the digital movie and the corresponding analog data. The overall accurracy of correlating analog sensor data with kinematic data is limited to the frame rate (±66msec) or in some cases the field rate (±33msec). The use of SIMPTE or RC time codes affords frame accuracy and many frame grabbers allow decomposition to odd and even fields.

Figure 2: Correlated Movement and Thrust measurements

ColorImage supports digitization of analog data using the GW Instruments MacAdiosII card. To acquire correlated analog data we currently employ the same rewind and search algorithm used for acquisiton of digital "movies". ColorImage generates a "chart recorder" image file of the digitized waveforms and two analog files for each of the stereo audio channels.
The analog files which are generated by ColorImage can be read directly by the GW Instruments SuperScope application where they can be processed for filtering, pulse analysis or to extract relevant electrophysiological features. We have developed an ancillary program, SpikeTrain, which uses cluster analysis to decompose two channel recordings from nerves into the activity of individual units. We use this program to relate the acvtivity of giant reticulospinal neurons to the swimming behavior of lamprey.
In addition to electrophysiology, ColorImage supports the digitization of additional sensor data with both electrophysiology and kinematic analysis. Figure 2 shows a digital movie of a swimming lamprey which is tethered to a strain gauge. The swimming thrust of the lamprey is recordd in the video signal using a Colorado Instruments Model 109b Digital Display Generator. We have implemented an OCR reader in ColorImage which can digitize such alphanumeric data and save the numerical result directly to a file or alternatively present a dialog box to allow manual input of the result during digitization of each frame of the movie.

The final stage of neuroethological analysis is the visualization of multimedia data. Figure 3 indicates a graph of three media, two channels of electromyograms digitized from the stereo audio channels, a kinematic analysis of the progress of flexions (here regions of curvature maxima) down the body and the instantaneous swimming thrust derived from a digital meter as in figure 2. ColorImage acquired the video and analog data using the rewind and search mechanism described above. It then automatically traced the shape of the lamprey and entered the thrust from each frame. A second program performed curvature analysis on the lamprey shapes and generated a file of the locus (position) of flexions and the instantaneous thrust in each frame of the movie. A third program read the analog electromyogram files and the locus/thrust file and generated figure 3 as a MacPaint bitmap file.

Figure 3. Multimedia visualization of Electromyograms, Kinematic analysis and Swimming Thrust

As described above, multimedia analysis is somewhat ad hoc. A solution to many of the problems can be achieved throuth ghe QuickTime architecture recently announced by Apple. QuickTime supports digital movies which combine both video and autio tracks.. We are presently incorporating QuickTime into ColorImage as a solution to multimedia neuroethological recording. We plan to utilize the VideoToolkit (Abbate Video Consultants, Norfolk, MA) interface currently in ColorImage to support the generation of QuickTime Movies from videotape using the same rewind and search mechanism described above.

One disadvantage of QuickTime as presently implemented is the constriction of small image sizes and inherent resolution in kinematic measurements. DataTranslation has recently announced a product, the Media 100 non-linear editing syhstem which transcends the disadvantages inherent in small windows allowing full motion video and analog with 640x480 windows at video frame rates. The combination of this product with ColorImage may provide the ultimate solution for multimedia neuroethology.

Joseph Ayers is the Director of the Marine Science Center, Northeastern University, East Point, Nahant, MA 01908, (781) 581-7370. His research in motion analysis is supported by NSF grant DIR 8910-7532 while his research in the neuroethology of larval lobsters is supported by NSF grant NS IBN-9121224