I'd like to begin and say I am no senior expert at graphics programming and abstraction however I have spent numerous hours porting Horde3D to OpenGL ES 2.0 in my spare time and I have learned a lot about abstraction constructs and things so maybe I could explain a few things.
Someone mentioned applying good OOP-design via C++ using subclassing or virtual functions. These are indeed nice, clean and readable design approaches in an ideal world. If you're a hardcore 'to-the-metal' coder you'll find out Ogre3D (and I assume Irrlict) these don't really fit the target hardware. On x86 where you use an out-of-order CPU with massive caches it isn't a huge issue speed-wise, but on in-order CPU platforms like PS3 or Xbox 360 having virtual functions going into a vtable have horrendous performance due to the vtable lookup into memory causing cache misses and huge clock cycles missed in this timeframe, and in a place where it's making a hell of a lot of calls per frame at 16/33ms then this is a massive bottleneck. For more info look here (on Unity's abstraction layer from Aras P):
The approach to do #ifdefs is not nice to read (for an example, look at Panda3D's OpenGL codepath/abstraction for GL's) but can be advantageous speed-wise if you're doing it correctly, but you may need to make an executable per backend as you're compiling it for a specific target at build-time (which would make sense on platforms with only 1 API to target). I think iD Tech 2 or 3 use the approach outlined in the above link where they define abstraction by making a high-level struct with function pointers inside, and then make library .dll's/.so's for each backend they want to target whether it's a special OpenGL version or even a software renderer, and the game engine will choose the best one using the target system's library loader calls and grabs the function pointers that way, and then the engine will just calls into those while the backend handles the rest without the engine knowing the specifics. On this subject Fabien has made excellent breakdowns of how iD Tech engines are written: http://fabiensanglard.net/ Doom 3 is particularly interesting where they support quite a few code paths for OpenGL alone, to take advantage of extensions that some hardware vendors support like Nvidia's ultra shadow or depth bounds test (unfortunately there was no real support for OpenGL 2.x so it's mostly OpenGL 1.x fixed function with a few ARB shader programs on some hardware).
I can't really comment on how valve does their DX9-to-OpenGL translation, but I guess they have a frontend which matches DX9 more and then under the scenes they have a 'smart' way to deal with OpenGL's terms and state machine. Horde3D has a similar approach where the 'RendererDeviceInterface' has more DX10-like names but uses OpenGL's functions eg. a 'RenderBuffer' is really a 'FramebufferObject' bound to a texture. Some things which are unavoidable would be .dds textures needing to be flipped (D3D/GL differ here), you either remake the assets or re-use the windows one and just flip them in the shader or when loading them in which might cause some performance penalties.
I hope this explains a bit...