Amol Pandit, co-chair of the British Medical Association's Welsh BAME forum, said some from BAME backgrounds felt "unwelcome" in the NHS in Wales.
He felt there was "no formal place at the table" for BAME doctors to discuss policy with the Welsh government.
The government said it wanted a "productive working partnership" with the community.
About 40% of all doctors in the NHS in Wales are from Black or ethnic minority backgrounds, according to the profession's body, the British Medical Association (BMA).
In its last all-member survey in 2020, 55% of BAME doctors said there was respect for diversity and a culture of inclusion in the workplace - compared with 75% of white doctors.
The findings came against the backdrop of the Covid pandemic, which highlighted the disproportionate number of people from BAME communities affected by the virus.
Of the more than 40 doctors who have died with Covid-19 across the UK, the majority were from ethnic minority backgrounds.
"Given that we form such an important part of the health service, we have had very little recognition and an even smaller voice over the years," said Mr Pandit, a consultant urologist in Swansea.
One issue he highlighted was over the retention of doctors in Wales, who had been encouraged to move from overseas to take up roles here.
"Retention becomes an issue because people feel they are not welcome, their voices are not heard - and that needs to change," he claimed.
Mr Pandit said those doctors should be able to voice concerns, both about patient safety or personal experiences.
"They need to feel that somebody will listen to them, that they won't be victimised," he added.
The BMA study also found nearly a third of BAME medical students felt that bullying or harassment was a problem in their medical school and reported incidents of bullying and harassment at four times the rate of their white peers.
Mr Pandit said he was also concerned that those from BAME backgrounds appeared anecdotally to find it harder to secure training placements in the profession.
"At medical school we find about 45% of the admissions are of ethnic minority backgrounds. But when it comes to applying for national training, the names are not anonymised and they feel they would be less likely to be called for interview," he said.
The BMA Cymru's BAME forum, of which Mr Pandit is co-chairman, was established last year, as part of the reaction to the Black Lives Matters movement following the death of George Floyd in America.
Mr Pandit said it was a place where colleagues in the profession can safely discuss concerns, as well as becoming a united voice for those it represents.
But he said he also wanted more dialogue with leaders and officials in the Welsh government.
The urologist was also critical of the way ministers announced a consultation on its anti-racism race equality plan in March.
Mr Pandit argued that forums such as his should have been involved in developing the plan before it was put out to consultation.
"I've had no overtures from the Welsh government," he said. "It is just lip service - nobody really cares."
Responding to Mr Pandit's claims, the Welsh government said the race equality plan had been developed "with people from Black, Asian and Minority Ethnic communities across Wales, including people working in a range of professions in NHS Wales".
A government spokesman said officials also met with the BMA BAME forum recently to discuss joint working to implement the plan, and had met Mr Pandit "to discuss his ideas and how we could work together".
"We look forward to developing a productive working partnership," added the spokesman.