Hi Dean - I have just implemented a very similar solution, so hopefully I can help.
(1) How can I play music from a Mac (iTunes) into a DAC and then into a Hifi system. You can get a DAC with a USB input and connect the MAC to it via that.
(2) I would like to seriously improve the mp3 sound quality to emulate the sound quality expected from good CD recordings, what is the best way to achieve this? Don't use MP3, use either Apple Lossless (files take up about half the space of the CD originals and theoretically should be no loss in quality) or rip you CDs with no compression at all (WAV or AIFF files) to be absolutely certain of no loss in quality. Unless you have a very high-end CD player at present I'm confident that high-quality rips being replayed from your Mac via a decent DAC will provide better results.
(3) Lastly, what program would I need to install on my Mac to be able to send the music via a wireless connection to the DAC? Apple produce the Airport Express for this very purpose - it can connect either wirelessly or wired to you router and allows you to stream music from your iTunes. The AE connects via optical digital connection to your DAC. It comes with the Airtunes software that kind of bolts on to iTunes. Very slick, especially if you have an ipod touch or iphone to use as a remote controller via the Remote app.
The real problem is that, the only musical output on the Mac is a 3.5mm jack. I cannot connect this directly to a DAC because the sound is already analogue and not digital. Are you sure about this - all the current crop, and I believe several generations to date use a combined 3.5mm socket which outputs both analogue and optical digital signals - you can get adaptors for standard toslink optical digital cables to fit a 3.5mm socket. You should check your spec, but as I said above you could still use USB instead.
Most DACs can only receive a digital input, which tells me I cannot connect a DAC directly to a MAC. But rather, I would have to send the signal wirelessly to a Music Router (e,g SONOS, SOUNDSQUEEZE) and connect that to the DAC and then from there feed the signal into the Hifi Amp. Hopefully based on the above you will find that you can connect the Mac directly to your DAC, but in any case, I think that you'd find it inconvenient to have that as your only option and therefore if you're using iTunes as an interface, the Airport Express option is both seamless and very effective, and most importantly it allows you to use your Mac for other things whilst playing music (e.g. it doesn't have to be physically sat by your hifi). You can also get several and stream music from you Mac into any room of you house, selectively controlling which ones are active and which ones aren't.
Anyway - I can recommend the Cambridge Audio DAC-magic, which is absolutely excellent. Also I have to admit I haven't had the chance to experiment extensively yet, but I can't tell any difference in quality between having my computer connected directly to the DAC via USB, or streaming via the Airport Express.
HF: MacBook Pro+Airport Express+CA DAC-Magic|Creek OBH12+Arcam P35+P38|PMC TB2+|Chord Optichord|custom silver I/Cs| Atlas Hyper 2.0 spkr cables
HC: Pio PDP427XD|Sony STRDA3300ES|Sony BDPS550|Humax PVR9300T|MA Radius 180*5+90*2+REL Quake|QED cables