Canada–France–Hawaii Telescope Legacy Survey optical photometry has been used to study the galaxy luminosity functions (LFs) of 14 X-ray selected clusters from the XMM Large Scale Survey (XMM–LSS) survey. These are mostly groups and poor clusters, with masses (M₅₀₀) in the range 0.6 to 19 × 10¹³ M_⊙ and redshifts 0.05 ≤z≤ 0.61. Hence, these are some of the highest redshift X-ray selected groups to have been studied. Lower and upper colour cuts were used to determine cluster members. We derive individual LFs for all clusters as well as redshift-stacked and temperature-stacked LFs in three filters, g′, r′ and z′, down to M=−14.5. All LFs were fitted by Schechter functions which constrained the faint-end slope, α, but did not always fit well to the bright end. Derived values of α ranged from −1.03 to as steep as −2.1. We find no evidence for upturns at faint magnitudes. Evolution in α was apparent in all bands: it becomes shallower with increasing redshift; for example, in the z′ band it flattened from −1.75 at low redshift to −1.22 in the redshift range z= 0.43–0.61. Eight of our systems lie at z∼ 0.3, and we combine these to generate a galaxy LF in three colours for X-ray selected groups and poor clusters at redshift 0.3. We find that at z∼ 0.3 α is steeper (−1.67) in the green (g′) band than it is (−1.30) in the red (z′) band. This colour trend disappears at low redshift, which we attribute to reddening of faint blue galaxies from z∼ 0.3 to 0. We also calculated the total optical luminosity and found it to correlate strongly with X-ray luminosity (L_X∝L^2.1_OPT), and also with ICM temperature (L_OPT∝T^1.62), consistent with expectations for self-similar clusters with constant mass-to-light ratio. We did not find any convincing correlation of Schechter parameters with mean cluster temperature.