Dynamic Neural Networks

Ph.D. thesis of Peter B.L. Meijer,  ``Neural Network Applications in Device and Subcircuit Modelling for Circuit Simulation'' (1.2MB PDF file, HTML version; ISBN 90-74445-26-8;  https://doi.org/10.6100/IR459139). This thesis generalizes the multilayer perceptron networks and the associated backpropagation algorithm for analogue modeling of continuous and dynamic nonlinear multidimensional systems for simulation, using variable time step discretizations of continuous-time systems of coupled differential equations. A major advantage over conventional discrete-time recurrent neural networks with fixed time steps, as well as Kalman filters and time-delay neural network (TDNN) models with fixed time steps, is that the distribution of time steps is now arbitrary, allowing for smaller time steps during steep signal transitions for much better trade-offs between accuracy and CPU time, while there is also still freedom in the choice of time steps after the neural network model has been generated. In fact, multirate methods for solving differential equations can be readily applied. The use of second order differential equations for each neuron allows for complex oscillatory behaviours even in feedforward networks, while allowing for efficient mappings of differential-algebraic equations (DAEs) to a general neural network formalism. The resulting formalism represents a wide class of nonlinear and dynamic systems, including arbitrary nonlinear static systems, arbitrary quasi-static systems, and arbitrary lumped linear dynamical systems.

Envisioned application areas include the representation and control of nonlinear neural dynamics and its use in neuroengineering, oscillatory brain dynamics, neuromodulation, and computational neuroscience. Other possible application areas include nanoscale device modeling, and modeling of signal propagation, delays and responses in dynamical systems in general. With feedback from output to input, attractor neural networks can be represented for modeling arbitrarily complex brain dynamics (including various forms of chaotic behavior). This is possible because the neural formalism with external feedback can represent any dynamical system described by implicit nonlinear vector equations of the general form f(x,dx/dt,t)=0. Note that the approach may also be applied to non-deterministic and noisy systems that are characterized by differential-algebraic equations for the deterministic statistical models, e.g., dynamic Bayesian networks (DBN).

Since the methods described in this thesis generalize multilayer perception networks, they may similarly be readily extended to incorporate modern  deep learning methods for layer-by-layer pre-training with stacked autoencoders, e.g. using restricted Boltzmann machines (RBM) with contrastive divergence for deep belief networks (DBN).

Keywords: IC design, modeling, artificial neural network (ANN), dynamic neural network (DNN), variable time steps, differential equations, circuit simulation, transient analysis, transient sensitivity, AC analysis, AC sensitivity.

Committee members at the Ph.D. defense at Eindhoven University of Technology were Prof. Dr. J. Jess, Prof. Dr. W.M.G. van Bokhoven, Prof. Dr. P.P.J. van den Bosch, Dr. P.J.M. Cluitmans, Dr. J.T.J. van Eijndhoven, Prof. Dr. A.H.M. van Roermund, Prof. Dr. H. Wallinga and Prof. Dr. L. Spaanenburg. May 2, 1996.

Summary

Basically, the thesis covers the problem of constructing an efficient, accurate and numerically robust model, starting from behavioural data as obtained from measurements and/or simulations. To achieve this goal, the standard backpropagation theory for static feedforward neural networks has been extended to include continuous dynamic effects like, for instance, delays and phase shifts. This is necessary for modelling the high-frequency behaviour of electronic components and circuits. From a mathematical viewpoint, a neural network is now no longer a complicated nonlinear multidimensional function, but a system of nonlinear differential equations, for which one tries to tune the parameters in such a way that a good approximation of some specified behaviour is obtained.

Based on theory and algorithms, an experimental software implementation has been made, which can be used to train neural networks on a combination of time domain and frequency domain data. Subsequently, analogue behavioural models and equivalent electronic circuits can be generated for use in analogue circuit simulators like Pstar (from Philips), SPICE (University of California at Berkeley) and Spectre (from Cadence). The thesis contains a number of real-life examples which demonstrate the practical feasibility and applicability of the new methods.

Click image for LaTeX source.

Curriculum Vitae

From September 1985 until August 2006 he worked as a research scientist at  Philips Research Laboratories in Eindhoven, The Netherlands, initially focussing on black-box modeling techniques for analogue circuit simulation. In May 1996 he received his Ph.D. from Eindhoven University of Technology,  Department of Electrical Engineering, on the subject of dynamic neural networks for device and subcircuit modeling for circuit simulation. From 1999 until 2003 he was cluster leader of the Future Design Technologies cluster within the research group Digital Design & Test at Philips Research, while working on nanotechnology and the simulation and modeling of RF effects in high-speed digital circuits. In October 2006 he left Philips and joined Central R&D of the newly founded  NXP Semiconductors, to work in the field of computer vision research, using a massively parallel SIMD-based hardware platform for real-time low-power video processing ("pixel crunching" with the 320-core Xetal chip). In September 2008 the focus of his work shifted towards display technologies.

In parallel with his regular work in the electronics industry, and in line with his interests in human sensing capabilities, he developed an image-to-sound conversion system known as "The vOICe", aimed at the development of a synthetic vision device (prosthetic vision system) for the totally blind.

His U.S. Patents on neural networks:

No. 5,553,195: Dynamic neural net, September 3, 1996.

No. 5,790,757: Signal generator for modelling dynamical system behaviour, August 4, 1998.

 Extended CV

Some other publications on nonlinear modeling and computer vision:

 ``Multiple View Camera Calibration for Localization'' L. Spaanenburg, M.A. Tehrani, R.P. Kleihorst and P.B.L. Meijer,  ``Behavior Modeling by Neural Networks,'' (450K PDF file) 19th International Conference on Artificial Neural Networks (ICANN 2009), September 14-17, 2009, Limassol, Cyprus. Published in Artificial Neural Networks – ICANN 2009, Lecture Notes in Computer Science, Volume 5768/2009, pp. 439-448, Springer Berlin / Heidelberg.  DOI.

Abstract - Modeling of human and animal behavior is of interest for a number of diagnostic purposes. Convolutional neural networks offer a constructive approach allowing learning on a limited number of examples. Chaotic tendencies make that learning is not always successful. The paper looks into a number of applications to find the reason for this anomaly and identifies the need for behavioral references to provide determinism in the diagnostic model.

Abstract - This paper discusses a method for abnormal motion detection and its real-time implementation on a smart camera. Abnormal motion detection is a surveillance technique that only allows unfamiliar motion patterns to result in alarms. Our approach has two phases. First, normal motion is detected and the motion paths are trained, building up a model of normal behaviour. Feed-forward neural networks are here used for learning. Second, abnormal motion is detected by comparing the current observed motion to the stored model. A complete demonstration system is implemented to detect abnormal paths of persons moving in an indoor space. As platform we used a wireless smart camera system containing an SIMD (Single-Instruction Multiple-Data) processor for real-time detection of moving persons and an 8051 microcontroller for implementing the neural network. The 8051 also functions as camera host to broadcast abnormal events using ZigBee to a main network system.

Abstract - In this paper we discuss the use of the state-space modelling MOESP algorithm to generate precise information about the number of neurons and hidden layers in dynamic neural networks developed for the behavioural modelling of electronic circuits. The Bartels-Stewart algorithm is used to transform the information from the MOESP algorithm to the neural network formalism. The technique is related to the class of model order reduction algorithms that receives much attention in recent years, especially in the electronics industry.

Abstract - This paper presents a self-rectification stereo vision system based on a real-time, low power, and wireless smart camera platform. The proposed self-rectification method is suitable for an embedded parallel stereo system, where the epipolar lines are parallel to the image scan lines. The stereo images are first aligned by applying 1D signature matching. Then the alignment is refined based on the quality of the disparity measurement. The rectification method can be applied both offline and online. The major advantage of this rectification method is that no clean background is needed during the rectification process. After the rectification, the conjugate epipolar line is collinear. The dense matching method is implemented to achieve the depth map. This depth map provides a tool for segmentation. The application runs in an SIMD video-analysis processor, IC3D, at 30 fps and handles disparity up to 37 pixels in CIF (320times240 pixels) mode.

Abstract - The recent development of distributed smart camera networks allows for automated multiple view processing. Quick and easy calibration of uncalibrated multiple camera setups is important for practical uses of such systems by non-experts and in temporary setups. In this paper we discuss options for calibration, illustrated with a basic two-camera setup where each camera is a smart camera mote with a highly parallel SIMD processor and an 8051 microcontroller. In order to accommodate arbitrary (lens) distortion, perspective mapping and transforms for which no analytic inverse is known, we propose the use of neural networks to map projective grid space back to Euclidean space for use in 3D localization and 3D view interpretation.

Abstract - The standard backpropagation theory for static feedforward neural networks can be generalized to include continuous dynamic effects like delays and phase shifts. The resulting non-quasistatic feedforward neural models can represent a wide class of nonlinear and dynamic systems, including arbitrary nonlinear static systems and arbitrary quasi-static systems as well as arbitrary lumped linear dynamic systems. When feedback connections are allowed, this extends to arbitrary nonlinear dynamic systems corresponding to equations of the general form f (x, x', t) = 0. Extensions of learning algorithms to include combinations of time domain and frequency domain optimization lead to a semi-automatic modelling path from behaviour to simulation models. Model generators have been implemented for a range of existing analog circuit simulators, including support for the VHDL-AMS and Verilog-AMS language standards.

Abstract - This paper presents a general scheme for the construction of device models for circuit simulators. Two general n-dimensional C¹ table models have been constructed under this scheme. Each table model can automatically reconstruct the exact behavior of the dc current expressions of two basic physical device models, namely the Ebers-Moll bipolar transistor model and the GLASMOST MOSFET model. Also, the evaluation times of the three-dimensional table model implementations are less than those of advanced physical CAD device models. The table models are generally very accurate and have negligible model development time. Both table models have been implemented in the SPICE-like circuit simulator PHILPAC.

Abstract - Device models determine to a large extent the attainable quality of electronic circuit simulation. This paper describes a general approach to the automatic construction of C¹ multidimensional (multivariate) device models from table values. By using suitable heuristics, a compact and accurate C¹ table model for highly nonlinear multidimensional behaviour can be obtained. An example demonstrates the basic ideas.

Invited conference presentations:

P.B.L. Meijer,  ``Compact behavioural modelling of electromagnetic effects in on-chip interconnect,'' invited presentation at MACSI-NET 2003, May 2-3, 2003, Zürich, Switzerland.

Abstract - Obtaining accurate circuit-level simulation models of very high speed on-chip interconnect can be very difficult. Distributed resistive, capacitive and inductive effects need to be accounted for, as well as the frequency dependent skin effect. In this talk, a new modelling flow is outlined for obtaining compact and accurate interconnect models that can be readily used in analog circuit simulators like Philips' Pstar, Berkeley SPICE or Cadence Spectre, while including automatic model generation support for simulators that use the modelling languages VHDL-AMS and/or Verilog-A. The modelling flow makes use of detailed electromagnetic simulations obtained through numerically solving the Maxwell Equations in the time domain, e.g., using FDTD-like methods [6]. Next, linear state space models are obtained from the time domain data using subspace methods like 4SID/MOESP [3,4,5]. The resulting linear state space models form a subset of the class of models that can be represented and optimized by our generalized dynamic neural network modelling formalism [1,2]. A software implementation of this formalism is applied to remove undesirable modelling artifacts from the subspace method outcomes, as well as to subsequently and automatically generate simulation models for a range of supported circuit simulation languages. Preliminary results of the use of the modelling flow will be illustrated by means of simple interconnect examples.

1. P. Meijer, ``Neural Networks for Device and Circuit Modelling,'' in Scientific Computing in Electrical Engineering, Proc. SCEE-2000, August 20-23, 2000, Warnemünde, Germany, U. van Rienen, M. Günther and D. Hecht, Eds., Springer-Verlag, 2001, pp. 251 - 258. ISBN 3-540-42173-4.

2. P. Meijer, ``Neural Network Applications in Device and Circuit Modelling for Circuit Simulation,'' Ph.D. thesis, Eindhoven University of Technology, May 2, 1996.

3. I Munteanu, T. Wittig, D. Ioan and Th. Weiland, ``Efficiency of model-order reduction through TSL and 4SID techniques,'' MACSI-NET Model Order Reduction Seminar, Eindhoven, Oct. 2001.

4. M. Verhaegen and P. Dewilde, ``Subspace model identification, part 1: The output-error state-space model identification class of algorithms'' International Journal of Control, Vol. 56, pp. 1187-1210, 1992.

5. M. Verhaegen and P. Dewilde, ``Subspace model identification, part 3: Analysis of the ordinary output-error state-space model identification algorithm,'' International Journal of Control, Vol. 58, pp. 555 - 586, 1993.

6. K. Yee, "Numerical Solution of Initial Boundary Value Problems involving Maxwell's Equations", IEEE Trans. Antennas and Propagation, Vol. 14, 302-307, 1966.

Recent publications about brain-computer interfaces (BCIs)