Building a car to order is different than buying off the lot because while showroom models are already sitting there (the dealer wants them gone to make room for more cars) if you build one then the dealer must order it. This means occupying space on the lot with it until you pick it up and one fewer "showroom" car that he can order. Theres also the chance that you put in the deposit and order but never pick the car up. While you would lose your deposit the dealership would now be taxed with selling the car which you may have ordered in less desirable colors/options. For these reasons you may be able to get a little bit off but if you order a car you will likely pay close to MSRP.
The same is true of shipping a car from another dealership to yours. When the dealership occupies a lot space specifically for you then your price negotiation becomes limited.