The video codec that XBMC supports are MPEG-1, MPEG-2, H.263, MPEG-4 SP and ASP, MPEG-4 AVC (H.264), HuffYUV, Indeo, MJPEG, RealVideo, RMVB, Sorenson, WMV, Cinepak.
If you would like to view your video files in XBMC Media Player, the best way is converting your video to one of its supportable video format listed above, like MPEG-1, or MPEG-2. As for XBMC Video Converter, I recommend Aunsoft Video Converter for Mac which has a perfect performance in encoding various formats of video, like AVI, MP4, MKV, FLV, TiVo to MPEG for XBMC Media Player.
Hope my sharing is useful.
Enjoy your video.