You're right, I felt the same during my progression. It is really cool to be able to challenge random players anytime in friendly fights.
Two improvements could yet be made:
- As stated in the answers, the level of others players should be clearly visible before engaging in a fight.
- There should be a small reward system, but only when fighting players with similar levels.
I only speak about friendly fights here, the competitions have already a polished system.