A lot of this has come about because the only way to get a penalty, if you're tripped, is to go down. If you don't go down, no penalty, usually.
Yes, you are right that contact is not tripping, but then you can award a foul with no contact also, if the approach is considered reckless or dangerous. It's quite rare. This is from ancient memory, and maybe it's changed. For example, a defender swings a leg at an attacker violently, misses, but is still penalized. Actually, it's not that rare, saw it recently.