Interpreting galactic luminosity requires assumptions about the galaxy-wide initial mass function (gwIMF), often assumed invariant in most stellar population synthesis (SPS) models. If stars form in clusters with metallicity- and density-dependent stellar IMFs, the integrated galaxy-wide IMF (IGIMF) can be calculated, with its shape depending on the star formation rate (SFR) and metallicity. The shape of the IGIMF thus depends on the SFR and metallicity. We develop the SPS-VarIMF code which enables us for the first time to compute the spectra, luminosities, and remnant populations of galaxies in the context of the varying gwIMF with time, SFR, and an assumed metallicity. Using the SPS-VarIMF code one can calculate how the interpretation from the integrated galactic light may change if the underlying galaxy-wide IMF is assumed to be environmentally dependent instead of being invariant. In particular, we compare the time evolution of the galaxy colour and the stellar mass-to-light ratio in different bands for the IGIMF and invariant canonical gwIMF assuming constant and delayed-τ star formation histories. We show that the underlying gwIMF can be determined by examining the colours and luminosities of late-type galaxies in ultraviolet and optical bands. On the other hand, for early-type galaxies, it is difficult to distinguish which gwIMF is valid since adopting the different gwIMFs yields almost identical colours. However, their gwIMF-dependent mass-to-light (M/L) ratios differ by up to an order of magnitude. Massive present-day elliptical galaxies would have been 10⁴ times as bright as at present when they were forming.