This paper presents mathematical optimization models of residential energy hubs which can be readily incorporated into automated decision making technologies in smart grids, and can be solved efficiently in a real-time frame to optimally control all major residential energy loads, storage and production components while properly considering the customer preferences and comfort level. Novel mathematical models for major household demand, i.e., fridge, freezer, dishwasher, washer and dryer, stove, water heater, hot tub, and pool pumps are formulated. Also, mathematical models of other components of a residential energy system including lighting, heating, and air-conditioning are developed, and generic models for solar PV panels and energy storage/generation devices are proposed. The developed mathematical models result in Mixed Integer Linear Programming (MILP) optimization problems with the objective functions of minimizing energy consumption, total cost of electricity and gas, emissions, peak load, and/or any combination of these objectives, while considering end-user preferences. Several realistic case studies are carried out to examine the performance of the mathematical model, and experimental tests are carried out to find practical procedures to determine the parameters of the model. The application of the proposed model to a real household in Ontario, Canada is presented for various objective functions. The simulation results show that savings of up to 20% on energy costs and 50% on peak demand can be achieved, while maintaining the household owner's desired comfort levels.