Homomorphic encryption provides a way to keep sensitive data encrypted while operations are performed on it. It offers the possibility of hosting and processing sensitive data in an untrusted environment, such as a public cloud. However, the additional encryption affects performance, and it may cause degradation to latency or throughput. Special considerations are required when evaluating performance of an Enterprise Resource Planning (ERP) system with a homomorphically encrypted database. These result from different encryption types with different performance characteristics, combinations of encryption with non-linear performance behavior, and disparities between startup and steady-state performance. In this paper, we describe the challenges of homomorphic encryption performance evaluation and some methods to obtain accurate and reliable results, and we present the application of these methods to the evaluation of a CryptDB-based system.