Are there any differences which you can't fit into the same story without changing the fundamental facts of either Matthew's or Luke's account?

Well there seem to be a number of fundamental facts which are different. Luke illustrates the census under Quirinius while Matthew omits it. Only Matthew illustrates a star in the east and the wise men (magi). Luke illustrates the angels singing however Matthew states nothing regarding angels. Matthew prepares regarding Herod ordering the slaughter of all baby boys 2 years old or younger while Luke states nothing regarding this. Matthew implies that Joseph and Mary lived in Bethlehem and then moved to Nazareth whereas Luke states that they lived in Nazareth however moved to Bethlehem to be counted in the census. Both Matthew and Luke report the genealogy of Jesus however while Matthew comprises 28 generations Luke illustrates 41. Most of the names illustrated by Matthew and Luke do not agree. Matthew consists of Mary and Joseph living in a house while Luke describes Jesus' birth as happening in a manger.

There are fundamental facts which seem non-negotiable though. Joseph and Mary were in Bethlehem when Jesus was born to them. Mary was a virgin at the time of Jesus' birth having been impregnated not by Joseph however by the Holy Spirit. Sometime after Jesus' birth the family travels back to Nazareth.