Modeling of ecological processes is demonstrated using a newly developed nonlinear dynamics model of microbial populations, consisting of a 4-variable system of coupled ordinary differential equations. The system also includes a modified version of the Monod kinetics equation. The model is designed to simulate the temporal behavior of a microbiological system containing a nutrient, two feeding microbes and a microbe predator. Three types of modeling scenarios were numerically simulated to assess the instability caused by (a) variations of the nutrient flux into the system, with fixed initial microbial concentrations and parameters, (b) variations in initial conditions, with fixed other parameters, and (c) variations in selected parameters. A modeling framework, using the high-level statistical computing languages MATLAB and R, was developed to conduct the time series analysis in the time domain and phase space. In the time domain, the Hurst exponent, the information measure–Shannon’s entropy, and the time delay of temporal oscillations of nutrient and microbe concentrations were calculated. In the phase domain, we calculated a set of diagnostic criteria of deterministic chaos: global and local embedding dimensions, correlation dimension, information dimension, and a spectrum of Lyapunov exponents. The time series data are used to plot the phase space attractors to express the dependence between the system’s state parameters, i.e., microbe concentrations, and pseudo-phase space attractors, in which the attractor axes are used to compare the observations from a single time series, which are separated by the time delay. Like classical Lorenz or Rossler systems of equations, which generate a deterministic chaotic behavior for a certain range of input parameters, the developed mathematical model generates a deterministic chaotic behavior for a particular set of input parameters. Even a slight variation of the system’s input data might result in vastly different predictions of the temporal oscillations of the system. As the nutrient influx increases, the system exhibits a sharp transition from a steady state to deterministic chaotic to quasi-periodic and again to steady state behavior. For small changes in initial conditions, resulting attractors are bounded (contrary to that of a random system), i.e., may represent a ‘sustainable state’ (i.e., resilience) of the ecological system.