Way back when, the Romans had a god named Janus. He was the god of doors and gates and had two faces — one looking forward and one looking back. Julius Caesar thought it would be appropriate for January, Janus' namesake month, to be the doorway to a new year, and when he created the Julian calendar, he made January 1 the first day of the year (this also put the calendar year in line with the consular year, as new consuls also took office that day).
For Caesar, the Julian calendar was a political tool and weapon. As the Roman armies conquered new lands, the Empire often gave its new subjects some freedom in retaining certain religious and social customs. After the calendar was created, though, it was used in every corner of the Empire, not just for consistency, but to remind all citizens of Roman authority and Caesar's power.
After Rome fell and Christianity spread through Europe, the celebration of the new year was seen as pagan (the Romans, after all, had observed the new year's first day by having in drunken orgies), so the first day of the year was moved to a more agreeable date to Christianize it. Some countries started their year on March 25, the day Christians commemorate the announcement to Mary that she miraculously was pregnant. Other countries used Christmas Day, December 25, and others used Easter Sunday, no matter what date it fell on. Often, this change only applied to the government calendar's. In common usage, January 1 was still the first day of the year, as regular non-clergy, non-royal folks didn't t see a need to change it.
This calendrical chaos worked for a while, but a frustrated pope would put an end to it during the Middle Ages. An error in Caesar's calendar had caused the Julian year to become misaligned with the solar year. By 1582, the difference had grown to 10 days. Over the years, the Spring Equinox (and, with it, Easter) kept getting moved up, and Pope Gregory XIII was tired of having to re-set the holiday. Gregory devised a new calendar that used a single leap day every four years to keep it aligned. He also restored January 1 as the first day of the year.
Most Catholic countries adopted the Gregorian calendar quickly, but the Protestant and Eastern Rite countries were a little more hesitant. The Protestants complained that the "Roman Antichrist" was trying to trick them into worshiping on the wrong days. The Eastern Rite churches wanted to maintain tradition, so some Eastern European countries kept the Julian calendar for centuries more. Russia didn't switch to the Gregorian calendar until after the 1917 revolution, and even today the Eastern Orthodox Church still follows either the traditional or revised Julian calendar to set its liturgical year.
Eventually the Protestant nations came around and switched to the Gregorian calendar. Most, though, changed the start of the year well before they adopted the whole thing. England, Ireland and the British colonies made January 1 the start of the year in early 1752 - Scotland had already switched about 150 years earlier - but waited until September to fully embrace the new calendar. The staggered move was perhaps symbolic, bringing the government calendar in line with the peoples' before bringing the nation's calendar in line the with Pope's.