Lenses based on diffractive optical elements and metalenses control phase or amplitude of light across an aperture to focus the light via engineered constructive interference of diffracted waves at the focal point. Both types of lenses work by effectively delaying the wavefront in a spatially varying fashion. Thin lenses of sufficient diameter and focusing power, however, cannot fully compensate for variation in free-space time of flight to maintain and focus an intact incident wavefront. Instead, they combine successive wavefronts. Broadband light, due to its limited temporal coherence, reduces the effectiveness of such interference. We represent lenses using equivalent medium theory and scalar diffraction and show that temporal coherence of the illumination imposes hard limits on the performance of lenses as measured by the Strehl ratio. The same bound will apply to DOEs and metalenses, including those using dispersion engineering. Our approach allows separate consideration of the contributions of the properties of the illumination, the constraints of lens construction, and the optical requirements.