Some people would say "I should be able to dress however I want because of my freedom", but our freedom in Christ doesn't mean we get to do whatever we want. If we have the spirit of Christ, we will humble ourselves & think about others. A humble spirit won't just think about themselves & having a "free spirit".
Humility causes us to think about how we may affect other people, and how we may be causing a stumbling block to someone. I am addressing Believers when I say this because we know that people of the world will always do what their flesh desires without regard for God or anyone but themselves.
So, yes you have your preference of style & don't have to conform to any man-made rules, however, if the way you choose to dress will cause a brother or sister-in-Christ to stumble, (& you know it), you are not acting out of love for your neighbor. In fact, if we continue to do it anyway, we are acting from the flesh and from a spirit of pride.
Paul actually touches on this topic in 1 Corinthians 8 about doing things out of love for others even when you have your free will. "But you must be careful so that your freedom does not cause others with a weaker conscience to stumble." - 1 Corinthians 8:9
Although this passage in 1 Corinthians is addressing food and idols, I believe the same principle of "being careful in how we exercise our freedom" can be applied to any situation.
How It May Effect Others
I've seen women wear too short dresses, too tight, open cleavage, in the house of the Lord. This could cause a brother to stumble because he could have a hard time focusing on the Lord because of what he just saw. A prideful woman would say, "Well he shouldn't look or learn to control himself". Okay, but shouldn't you also control what you are wearing in the Lord's house? We are all imperfect people & anyone with a lustful heart can turn anything into a problem... but only God knows that. We are still responsible for how we show up just as much as men are responsible for what they look at & how they think.
It could cause a sister to stumble too because she may think that she can compromise and where what she wants. Maybe she's a new believer & she sees another woman wearing something from club attire in the Lord's house so she makes a decision to do the same. What we see in the world should look very different from what we see in God's house of worship.
It obviously has some kind of effect on others if your employer has a dress-code policy. I've noticed that when it comes to Christian values, people get offended, but we are more willing to do what it takes to get paid. This is idolatry and reveals that our love for money is greater than our love for God.
To push it further, we should show up in the world as God's House of worship; wherever we go, believers are God's temple & we are called to be set apart. If you dress modest and appropriate at church but blend in with the lusts of the world Monday - Saturday, your inconsistency & compromise can creep into other areas of your faith.
This isn't to shame anyone or to push man-made traditions, but to answer a common statement and question that I've seen often. We can't please everyone and some people have voluptuous body types that may seem impossible to style, but God is still looking at our hearts & how our actions follow.
Our clothing choices do matter and if you are blatantly choosing to be immodest, then there's a deeper issue in your heart that should be addressed. Your desire to appear sexy & revealing might reveal lust, pride, and sensuality in your heart. And we know the Bible says that anyone indulging in these things will not inherit the Kingdom of God.
So I hope this has answered the question of "Does the way I dress matter to other people?" And the statement, "men just need to control themselves".