I'd lay the blame with your amp. The PM4400 isn't Marantz' finest moment and can sound very lethargic and flat at low volumes. I think you'll get much more out of the speakers by changing the amp.
I'd highly recommend looking at the following for a significant improvement:
Pioneer A400 (2nd hand)
Pretty much any Rotel amp, whether new or 2nd hand
NAD C325BEE (2nd hand) or C326BEE (new)
All of these work well at low volumes and have plenty of punch to their sound to get the B&Ws going.
If you have the budget to buy both second hand I'm finding my Pioneer/Mission amp/speakers combination very good at all volumes, including very low ones. I live in an appartment so low level performance is very important for me too. Amp + speakers cost me £250 total from a dealer, fully serviced and with a 3 month warranty.
Another combination I've found excellent at low levels was NAD C325BEE with Mordaunt-Short MS902i. Buy a C320BEE 2nd hand (virtually the same) and you could bag both items for less than £200.
I work for a Sony Centre, part of Sony's Direct retail channel. All opinions stated here are my own.