The output stage of the LM324 has low output impedance when it sources output current (when the 'push' half of the push-pull stage is active),
and when it sinks output current (when the 'pull' half of the push-pull
stage is active). It turns off (goes high impedance) at zero
output current, and that causes a 'dead spot' in the transfer
characteristic.
To reduce distortion, you must never allow zero output-stage current.
Ioutput=−Iload+Voutput−VRR≠0
That ensures that the load current plus the addon resistor's output
current is always nonzero, at all output signal voltage values.
That means a pullup resistor in conjunction with a load
that has a limited ability to sink current, or a pulldown resistor
in conjunction with a load that has a limited ability to source
current. A resistor to 'ground' might meet neither requirement.