The author, W. Scheidel, sees several apparent causes to this, although he does not venture into a conclusion as to which of these was or were most determinant–even if he seems to privilege the nature of the successor regimes as having played an important role.
A few comments could be made concerning the paper. First, concerning the reasons for the prospect of reunification of the Roman Empire, we must point out that the empire, besides being surrounded by barbarian peoples, also incorporated other peoples with a long history behind them (Egypt and Syria most particularly). To what extent did ‘nationalistic’ movements arise in these countries against the Roman and that would have helped fragment the empire? Also, Rome constantly faced Persia, of equal military power, while the Chinese did not, so far as I know, have to divert resources toward a quasi-permanent conflict with a power of equal military strength. To what extent did the Persian wars weaken the empire is another question that could be asked.
Second, and perhaps more significantly, the author considers Western Eurasia as a single entity. Yet, starting precisely from roughly the 6th century, this area, centered around the Mediterranean, became increasingly diversified culturally. Once Islam arose to prominence, all prospect of unification under a single power was lost forever. Division between Eastern and Western Christendom also made, on the long run, the reincorporation of the West within a larger Roman-Byzantine empire, or of the latter within a large domain headed by the Pope, impossible. Around the year 1000, while China had managed to reunify its former empire, the Mediterranean basin was a battleground on three levels: first between Islam and Christendom; second between the direct successor of the Roman Empire, of Orthodox faith, and the Western kingdoms and principalties under the sway of the Papacy; third, between these Western kingdoms, which each competed for supremacy and eventually functioned as separate nation-states. While the dreams of imperial reunification in the scale of the former Empire never died among all these contestants, diverging cultural differences made the prospect of such reunification increasingly impossible.