Keith Brown said men's attitude to low level sexism had to be challenged to make women safer.
His comments follow the sentencing of Wayne Couzens for the kidnap, rape and murder of Sarah Everard while Couzens was a serving police officer in London.
A working group led by Helena Kennedy QC on whether misogyny should be a distinct crime will report in February.
The full details of the Sarah Everard case, which emerged when he was sentenced last week, have reignited debate on what more can be done to tackle violence against women.
In March the Scottish government faced calls to include misogyny - prejudice against women - in its hate crime legislation when it was debated at Holyrood.
Labour MSP Johann Lamont tabled an amendment that would have included women as a protected group, giving the courts enhanced sentencing powers.
The amendment was defeated, however, with the government instead setting up a working group led by Baroness Kennedy to look at whether misogynistic abuse should be a separate crime.
Mr Brown told BBC Scotland's The Sunday Show the ministers would be guided by the group's findings - but he believed such a move would be a "very important signal that these behaviours are not acceptable in society from men".
"Her work is progressing very well and it may well be that we end up, depending on her recommendations, with a stand alone offence of misogyny," Mr Brown said.
Police vetting procedures
Mr Brown welcomed Police Scotland's introduction of new verification checks for lone police officers as a "very positive step" after it emerged that Couzens used his police warrant card and handcuffs to abduct Ms Everard.
New procedures mean members of the public who encounter an officer working alone in Scotland can verify their identity with the police control room.
The justice secretary said the measures put "the onus on the police not women to take action if somebody is in a vulnerable situation with one police officer".
Extra vetting procedures had also been put in place for trainee police officers in Scotland, he said.
The justice secretary said men of all ages had to change their attitudes to low level misogyny.
"I think it is very important that I have to say this, as a man to men, we have have to change our behaviour," he said.
"There are too many women that feel the justice system doesn't serve them well."
Sandy Brindley, from Rape Crisis Scotland, said one of the most "chilling" aspects of the Sarah Everard case was that Couzens was reportedly nicknamed "the rapist" by colleagues.
"What that says to me is that people knew that he behaved in a predatory way towards women and nobody held him accountable, and nobody challenged him - that's what we need to change," she said.
Asked about a report last year from former Lord Advocate Dame Elish Angiolini that found evidence of a "canteen culture" and discrimination within Police Scotland, Ms Brindley said there was no doubt sexist and racist attitudes existed within the police as well as in many other institutions.
"I do think we need a clear commitment from the police to address these issues, but I don't think it's only for the police," she said.
"Time and time again we see a man convicted of a crime like rape where it turns out that people around them, people who worked with them were aware of their predatory behaviour, an institution was aware of it and took no steps to challenge it.
"We need zero tolerance of the type of behaviour that leads for example being called a rapist and still continue in their job."
She said a "complete cultural change" was needed, pointing out that rape had the lowest conviction rate of any crime type in Scotland.
Only 43% of rape cases brought before a court end in conviction, compared with 88% of other crimes.
"If you are serious about improving women's safety, the starting point has to be having a justice system we can have confidence in and also a justice system that does not systematically let men who are guilty of rape walk free," she said.