Someone Please Tell Me How This Works

Elevator Floors

These are the floors on the elevator and you have no idea how many times I go to the wrong floor because I think I need to go up as logically 5 is higher than 3 when I really need to be going down. Why did they do this to me?