This study explores the implications of different modeling choices when predicting mortalityduring intensive care visits using recurrent neural networks. Using the MIMIC-III database, models were trained and tested with varying memory cells, architectures, and other hyper- parameters. Performance gains from incorporating information from unstructured clinical notes was tested as well. The study finds that a range of relatively shallow networks with varying memory cells and architectures can perform well and produce similar results, all of which outperform traditional mortality risk scores such as SAPS II. Adding information from clinical notes boosts model performance even with a simple natural language processing algorithm. Although methodological differences make direct comparisons complicated, the most accurate model presented here achieves an AUROC score of 0.943 which represents a slight improvement over similar prior work.