Most Brain-inspired Visual Object Recognition Models(BVORMs) do not consider local and global reciprocal con-nections in visual pathway. We addressed this weakness and implemented an attention modulation mechanism based on feed-back connections in BVORMs, where feature-selectivity is shaped and modulated by categorization of objects based on theirvisual features. This modification is inspired by the top-down neuromodulatory signals that make changes in post-synapticactivities of the feature-selective neurons. We also incorporated an implicit memory unit in BVORMs to accumulate recentHebbian synaptic plasticity’s of the neurons in each task. This mechanism guides the top-down feature-based attention modula-tion to retrieve the interrelated feature-selectivity pattern for each task.HMax and CNN models were used as two BVORMs andtested on a visual categorization problem: natural versus artificial objects in CALTECH-256. Based on experimental results,our proposed modifications not only increased their biological-plausibility but also significantly improved their categorizationaccuracies compared to the original models.