99% of the time it is related to one of 2 things. NAT or codec. To troubleshoot NAT, I would watch RTP packets paying attention to ports to make sure they are flowing to where they should be.
tcpdump -n -s 0 udp portrange 10000-20000
Do you have the extensions set to nat=yes?
If you Google asterisk one way audio you will see lots of solutions that might give you some ideas.
You mentioned that you are using VPN. Have you confirmed it works without the VPN? I have seen configuration issues with VPN or multiple routes where SIP packets took one route and RTP tried to take another route. I think the solution there is a properly configured sip_nat.conf.