The half-filled Landau level is expected to be approximately particle-hole symmetric, which requires an extension of the Halperin-Lee-Read (HLR) theory of the compressible state observed at this filling. Recent work indicates that, when particle-hole symmetry is preserved, the composite fermions experience a quantized π-Berry phase upon winding around the composite Fermi surface, analogous to Dirac fermions at the surface of a 3D topological insulator. In contrast, the effective low-energy theory of the composite fermion liquid originally proposed by HLR lacks particle-hole symmetry and has vanishing Berry phase. In this paper, we explain how thermoelectric transport measurements can be used to test the Dirac nature of the composite fermions by quantitatively extracting this Berry phase. First, we point out that longitudinal thermopower (Seebeck effect) is nonvanishing because of the unusual nature of particle-hole symmetry in this context and is not sensitive to the Berry phase. In contrast, we find that off-diagonal thermopower (Nernst effect) is directly related to the topological structure of the composite Fermi surface, vanishing for zero Berry phase and taking its maximal value for π Berry phase. In contrast, in purely electrical transport signatures, the Berry phase contributions appear as small corrections to a large background signal, making the Nernst effect a promising diagnostic of the Dirac nature of composite fermions.