I have a 318d Sport and whilst the 320d will be faster, I'm not sure you will really notice day to day, and the 318d really does have quite decent performance. Plus you can drive it like you stole it in sport mode and it will still do 50mpg.
Of course, so will the 320d, but in the end you'll be fine wither way, so I would go for the spec you most like the look of.
Current: F30 318d Sport manual, AW, 18" alloys
Sold: R56 MINI Dooper 2010, Chilli, Bluetooth
Sold: E93 325i M-Sport auto 2007
Sold: E91 320d SE 2005
Sold: E46 318Ci Coupe 2002