Why is not Fasting Punished so harshly?

This has been on my mind for a while. It's said that as Muslims we fast during the blessed month of Ramadan to achieve a greater closeness to Allah and better ourselves as muslims and people. So then why is it that if somebody chooses to leave their fast they are punished so harshly by Allah? Surely it's their loss and while I don't encourage leaving fasts I'm a little confused as to why not fasting is punished so severely since as long as they are not leaving their fasts to sin and they still pray 5 times a day they are not doing anything "wrong" there is just a lack of them doing "rightful" things so they miss out on sawab at their own loss.