He started building when he was 43 years old. Reportedly during his beat as postman he tripped on a large stone and was fascinated by its shape. He returned to the place next day and found several stones with unusual shapes. He carried a few to his place and started to build the palace from his imagination. He used to collect stones during his round as postman and built the palace in his free time working even in the night in the glow of a kerosene lamp. Initially he used to carry the stones in his hand, then in a basket and finally in a wheel barrow. The stones were joined by lime mortar and cement. It took him 33 years to complete the palace. The building was initially known as Temple of Nature but was later renamed Le Palais Idéal (The Ideal Palace).
Each facade of the palace is thematic - Source of Life, Source of Wisdom, Monument of Egypt, Temple of Nature, statues of horse and some wildlife, exotic flora, giants, mummies and assortment of large alcoves depicting a Muslim mosque, a Hindu temple, a Swiss chalet, a castle of Middle Ages etc. There is also access to a gallery of twenty meters long, deep in the Palace and decorated with sculptures.
View more photos and Google Maps at Panoramio
Some great photos and info at http://leavesgrass.blogspot.com
View in Google Earth / Google Maps (Unfortunately in low resolution.)
Source of info and more info: fr.wikipedia (English Translation)