Caffeine is an adenosine receptor antagonist. Its average half-life in healthy adults is roughly 5 hours, meaning 5 hours after consumption, 50% of the active compound remains in your system blocking sleep pressure.
Clinical Note
Even if you can "fall asleep fine" after coffee, residual caffeine significantly reduces slow-wave (deep) sleep and alters EEG micro-architecture.