The neural computation field had finally delivered on its promises in 2013 when the University of Toronto group reported a deep neural network that outperformed other machine learning approaches in image classification accuracy. That breakthrough was not due to algorithmic advances but rather the availability of high-performance graphical processors that enabled large-scale neural network modeling. Since then, the biologically-inspired neural network algorithms have become state-of-the-art approaches in many artificial intelligence tasks, and the future progress in this field hinges on even more powerful hardware. Such hardware, however, is unlikely to be implemented with the conventional digital circuit technology, whose performance seems to be saturating due to the faltering Moore’s law. On the other hand, further opportunities are presented by neuromorphic hardware that mimics critical features of biological neural networks, most importantly analog in-memory computing, in an attempt to match their energy-efficiency. Most importantly, neuromorphic hardware takes advantage of the physical-level analog implementation of vector-by-matrix multiplication (VMM), which is the most frequent operation in any neural network. The key component of such a circuit is a nanodevice with adjustable conductance —essentially an analog nonvolatile memory—used at each crosspoint of a crossbar array and mimicking the biological synapse. Prior work showed that analog VMM circuits based on redesigned eFlash memories and metal-oxide memristors, the most promising analog memory device technologies for neuromorphic computing, are much more energy-efficient as compared to the digital counterpart implemented in similar process node and performing a similar function. The main goal of this dissertation is to advance neuromorphic circuits based on memristors and eFlash memories on several fronts. The first part of the thesis is devoted to improving functional and physical performance of analog-domain vector-by-matrix multiplication with a specific focus on neuromorphic inference applications, including the development of novel programming algorithms, mitigation approaches for various device and circuit non-idealities, and design of efficient peripheral circuits. For example, we use novel programming algorithms to experimentally demonstrate <4% relative tuning error in a 64x64 passively integrated crossbar circuit despite significant variations, with 25% normalized standard deviation in device I-V characteristics. The developed post-fabrication methods for mitigating IR drops, I-V static nonlinearity, and device variations enable software-equivalent accuracy for the large-scale neural networks for the studied memristor technology. The efficacy of novel peripheral circuits is verified via SPICE modeling, which shows, e.g., POp/J-scale energy-efficiency for current-mode 55-nm NOR-flash memory circuits. The section is concluded with the discussion of our ongoing work on the design and fabrication of several large-scale neuromorphic chips. The second part of this thesis extends the work on analog VMM circuits to enable the implementation of more advanced probabilistic neuromorphic hardware, which is especially effective in solving combinatorial optimization problems. By operating the previously developed analog VMM circuit in a lower signal-to-noise-ratio regime, we achieve stochastic VMM functionality and utilize such circuits to prototype small-scale restricted Boltzmann machine and Hopfield neural network with runtime-controlled effective temperature. Furthermore, we suggest several novel hardware-friendly annealing approaches and successfully verify them by solving experimentally typical combinatorial optimization problems. The last part of this dissertation is devoted to hardware security primitives, such as physically unclonable functions and true random number generators. At the core of our idea are analog circuits based on metal-oxide memristors and eFlash memories, which are very similar to analog VMMs developed for neuromorphic computing. The main difference is that memory device non-idealities, e.g., randomness in tuning and memory I-V variations, are utilized as a rich source of static entropy, which is essential for implementing hardware security primitives. We developed three architectures - RX-PUF and VR-PUF that avoid the need for conductance tuning procedure in previously proposed memristor-based PUFs, and ChipSecure, which exploits variations in leakage current, subthreshold slope, nonlinearity, and stochastic tuning error in eFlash memory arrays to create a unique digital fingerprint. The key novelties of the proposed designs include enormous challenge-response pairs to enable strong PUF properties and a low-overhead key-booking scheme to dramatically improve the PUF reliability across a wide temperature range of operation. The analysis of the measured data in all our PUF demonstrations shows strong resilience against machine learning attacks.