Are there any differences that you cannot fit into the same story without changing the basic facts of either Matthew's or Luke's account?

Well there seem to be a number of basic facts that are different. Luke mentions the census under Quirinius while Matthew omits it. Only Matthew mentions a star in the east and the wise men (magi). Luke mentions the angels singing but Matthew says nothing about angels. Matthew writes about Herod ordering the slaughter of all baby boys 2 years old or younger while Luke says nothing about this. Matthew implies that Joseph and Mary lived in Bethlehem and then moved to Nazareth while Luke says that they lived in Nazareth but moved to Bethlehem to be counted in the census. Both Matthew and Luke report the genealogy of Jesus but while Matthew includes 28 generations Luke mentions 41. Many of the names mentioned by Matthew and Luke do not agree. Matthew has Mary and Joseph living in a house while Luke describes Jesus' birth as happening in a manger.

There are basic facts that seem non-negotiable though. Joseph and Mary were in Bethlehem when Jesus was born to them. Mary was a virgin at the time of Jesus' birth having been impregnated not by Joseph but by the Holy Spirit. Sometime after Jesus' birth the family travels back to Nazareth.