Structural damage assessment under external loading, such as earthquake excitation, is an important issue in structural safety evaluation. In this regard, an appropriate data analysis and system identification technique is required to interpret the measured data and to identify the state of the structure. Generally, the recursive system identification algorithm is used. In this study, the recursive subspace identification (RSI) algorithm based on the matrix inversion lemma algorithm with oblique projection technique (RSI-Inversion-Oblique) is applied to investigate the time-varying dynamic characteristics. The user-defined parameters used in the RSI-Inversion-Oblique technique are carefully discussed, which include the size of the data Hankel matrix (i), model order to extract the physical modes, and forgetting factor (FF) to detect the time-varying system modal frequencies. Response data from the Northridge earthquake from the Sherman Oaks building (CSMIP) is used as an example to examine a systematic method to determine the suitable user-defined parameters in RSI. It is concluded that the number of rows in the data Hankel matrix significantly influences the identification of the time-varying fundamental modal frequency of the structure. An algorithmic model order selection method using the eigenvalue distribution of RSI-Inversion can detect the system modal frequencies at each appending data window without causing any abnormality.