Neanderthal contraction and modern human colonization of Europe

JEAN-PIERRE BOCQUET-APPEL & PIERRE YVES DEMARS *

A complex statistical analysis of I4C dates from European Neanderthal and Early Modern

humans has enabled important new understanding of h o w and when this major population change took place. This data allows the debate on the end of the Neanderthals to continue on much firmer ground. The large dataset m a y be found at http://intarch.ac.uk/antiquity/additional/bocqtablel .html
Key-words:Neanderthal contraction, modern human dispersal, hunter-gatherer demography, Palaeolithic demography, palaeodemography

Many questions regarding the initial invasion of Europe by modern humans and the replacement of the Neanderthals lack the basic parameters allowing proper correlation of the process. If the east-west geographical direction is not in question, other questions still include: Which route(s) did the invasion take? How fast did it occur? How did the geographical contraction of the Neanderthals take place, and at what rate? When and for how long did the two metapopulations coexist, and in which zones? Was the duration of coexistence uniform, or did it vary depending on the zones and the chronology? To these questions some tentative answers might be given, if it were possible to locate chronologically and geographicallythe cultural occurrences or markers which represent the end and beginning of the two meta-populations. Here we will analyse the I4Cdates which are supposed to date the ending of the Neanderthals and the beginning of modern humans, by using space-time distribution statistics. Then, we will map the geographical contraction of the Neanderthals in Europe and the expansion of the modern humans.

proxy variables for Neanderthals and modern humans, or directly from human remains. The Neanderthals are represented by Mousterian, Chstelperronian, Szeletian, Bohunician and Uluzzian; modern humans by Aurignacian, Bachokirian and Olchevian. Compared to the traditional radiocarbon technique of dating, the AMS technique has greater precision, i.e. a reduced standard deviation, but it does not produce a positive or negative systematic deviation of date. All the radiocarbon dates were thus regrouped in the same sample. The process of invasion-contraction is supposed to have its origin in the Middle East. But in this region, the socalled Levantine Aurignacian is more recent by about 10,000 years than the dates of early Aurignacian sites located in Europe. The issue of the geographical origin of these Aurignacians is unsolved (seeBar-Yosef & Kra 1994).The geographical distribution of the data is thus limited to Europe, at roughly, 35"E longitude. The entirety of these are called the large dataset.

The selected dataset

It is necessary to take into account that there is a risk of bias by contamination in majoring or minoring the age at proxy variables. Although it is not easy to quantify this risk, the latter risk is greater than the former because it is easier to add than to remove carbon. The large dataset was thus filtered on criteria of chronological

Material and methods

The large dataset

The data are represented by 468 indexed 14C dates, positioned at geographical coordinates, which according to the literature have provenience from archaeological levels, used as

* Bocquet-Appel, CNRS, EP 1781, 44 rue d e 1'Amiral Mouchez, 75014 Paris, France. bocquet-appel@ivry.cnrs.fr Demars, Institut de Prehistoire et de GBologie du Quaternaire, Universitk Bordeaux, 1 avenue des Facultes, 33405 Talence Cedex, France.
Received 8 July 1999, accepted 3 November 1999, revised 1 8 March 2000.
ANTIQTIITY 74 (2000): 544-52

NEANDERTHAL CONTRACTION AND MODERN HUMAN COLONIZATION OF EUROPE

545

60
50
cn
Q) w

I
:
r .

0 v 7

40

30
20 -1 0

0 10 Longitude

20

-1 0

10

20

Longitude
FIGLJRE 2 . Distributions by longitude of the residual *C dates representing Neanderthals, after filtering on criteria of chronological coherence within-site. The dotted line represents the first decile, taken a s rejection threshold for corresponding l4C dates.

FINJRE1.Distribution b y longitude of the I4C dates representing the Neanderthals, after filtering on criteria of chronological coherence within-site
( ~ 0 . 2 0 3E , 0 . 0 2 4 , N=124).

coherence within-site and statistics, taking into account the assumption of the dissymmetrical risk stated above. Chronological coherence within-site: in a multilevel site, if a date from a lower level were more recent than that of a higher level, it was eliminated and regarded as contaminated. Typically, this was the case with Mousterian levels or Neanderthal remains which were more recent than those of Aurignacian levels (located stratigraphically above). A rejection threshold of recently published dates for Neanderthals, suspected to be contaminated, on a statistical criterion: I4Cliterature provides dates for proxy Neanderthal up to sub-actual periods, which were eliminated from the archaeological literature without explanation. Therefore, there is a kind of threshold of surreptitious archaeological rejection, whose criteria are not expressed. A statistical approach was used to delimit an explicit rejection threshold. A significant positive correlation is observed between 14Cdates and longitude, going in the direction of older in the east than in the west (FIGURE 1).To delimit the rejection threshold, geography was thus taken into account. Instead of taking a rejection threshold, in terms of a I4C date in absolute value, which could penalize the dates of the sites located in the west, the residuals of I4C dates to the linear

regression on longitude were retained. The 14C dates that corresponded to the residuals located in the first decile of the residual distribution were eliminated, i.e. at the level -8100 years (FIGURE 21. This rejection threshold corresponds to the dates BP 224,050 in the west and 129,480 in the east. Finally, the selected dataset, on which the following analysis is based, has 429 indexed 14Cdates (Neanderthals: N=124 dates, 60 sites; modern human: N=305 dates, 85 sites) in Eu3). rope (FIGURE

Space-time distributions of I4C dates and populational processes A I4C date can be considered as the random drawing of a marker of a space-time population process whose generating 3D distribution is unknown. This process is, in the case of Neanderthals a contraction, and in modern humans a colonization. This population process can also be seen in the form of a continuum of dates representing its realization step by step on the map. To estimate this continuum and to represent this population process there are two main problems to overcome: i to obtain samples of the unknown generating distribution on different points of the map;

546

JEAN-PIERRE BOCQUET-APPEL 8r PIERRE YVES DEMARS

FIGURE 3 . Locations of the data points on the niap (large dataset).

1.o

4-0

2 0.6 0
0 .-

a m
K

0.8

I ; :

5 0.4 m
0.2
0.0 15 20 25 30 35 40 45 50 55 60 14C dates BP HOMO

* Modern
Neanderthal

FIGURE 4. Distribution of "C dates in a square (number 45, latitude 44-464 longitude O-ZOE). T h e dotted lines represent the first and last decile of the distributions.

ii to reconstitute (to estimate) the hypothetical continuum of dates on the map from the estimated statistical parameters of these samples. The patterns of invasionlcontraction were thus obtained from the selected dataset by using the following two-step approach.

Estimates of the minimum and the maximum dates for Neanderthals and modern humans, in 2"x2' grid squares. The map of Europe was subdivided into 2Ox2" regular grid squares. In each square, all the dates of a human group form a sample of local distribution. For example, FIGURE 4 represents two observed distri-

NEANDERTHAL CONTRACTION AND MODERN HUMAN COLONIZATION OF EUROPE

547

60 00

50 00

&40 00

FIGURE 5. Neanderthal variogram (average of the two m i n i m u m dates by square, selected dataset). Vertical axis: semivariance of the **C dates; horizontal axis: geographical 0 distance ( ~ 1 0 kmj.

$30 00

20 00

I
10 00

8000 7000

:
4

:
4

6000 :
5u00
c4

FIGURE 6 . Modern h u m a n variogram (average of the two m a x i m u m dates by square, selected dataset). Vertical axis: semi-variance of the I4C dates; horizontal axis: geographical distance (XI00 k m ) .

54000

E
m3000

2
4
1 -

.
* -/--x *
d
4
/

L 0

2 0 0 0 : ,A

butions of I4C dates in a square. An invasion and a contraction are represented at the limits of these distributions. In a square, the dates indicating the ending of Neanderthals and the beginning of modern humans were estimated by taking the minimum and the maximum of the corresponding observed I4C distribution within-square. The minimum and the maximum of each were estimated by taking the average of the two most recent or oldest I4C dates. When there was only one date in a square, it was omitted. Then, these two estimated parameters were positioned in the cen-

tre of the grid squares from where the sample I4C distributions come. Spatial interpolation (kriging). From the estimated minimum (Neanderthals) and maximum (modern humans) l4C dates in squares, the two hypothetical geographical continua of the dates were reconstituted by using the kriging technique of spatial interpolation. Briefly, let us recall that the kriging technique is the transposition of the multiple linear regression in a spatial context. A measure of spatial autocorrelation is expressed by a model variogram, fitted to the observed variogram from the data.

548

TEAN-PIERRE BOCQUET-APPEL & PIERRE Y VES DEMARS

From the data points, the model variogram is injected into the regression on any point of studied geographical space. The kriging technique lies outside the scope of this paper and is described, for instance, in Wackernagel (1998). FIGURES 5 & 6 show the variogram for Neanderthals and modern humans, respectively. The variance at distance zero is greater for Neanderthals (16 thousand years BP) than for modern humans (6.5 thousand years BP). This indicates local heterogeneity, a background noise, reflecting most probably anomalies of dates, greater for Neanderthals than for modern humans. The Neanderthal variogram shows an increase in variance, until the geographical distance of 6 ( ~ 1 0km), 0 where it takes the value of 38, from which the information redundancy increases slightly again at a short distance and is stabilized around a value of approximately 30. This type of variogram, where the variance increases then decreases with the geograpliical distance before being stabilized, is known as indicating a 'hole effect' in the continuum of the state variable (Wackernagell998).The variogram of modern humans shows information redundancy which decreases with the geographical distance, first exponentially at a very short distance, approximately of 3 ( ~ 1 0 0 km), then much less rapidly, expressed by a power function. The interpolation parameters for the kriging technique (ordinary kriging) were obtained from those observed variograms. Finally, relative to the exact date value at the localities, the interpolater slightly over-estimates and underestimates the low and high values respectively. By estimating the surface of the hypothetical continuum of dates, the interpolator is conservative. In short, it respects the overall geographical pattern but slightly flattens its relief. In order to form isochrons, i.e. areas of homogeneous dates, instead of a continuous surface, the continuum of datings can lie subdivided into successive slices, like level lines on a relief. The thickness of these slices depends on the precision of information, of its density on the map. In principle, the denser and more precise the information is, the finer the chronological slice forming the isochrons will be. To compare the two population phenomena at the same intervals, which occurred during the same Palaeolithic period, the surface dates were thus subdivided into successive periods of 2500 years, including, a slice for modern humans,

at mid-period 35,000-32,500 BP, to get some insight into 33,750 BP, where population change accelerates considerably (see below). FIGURES 7 & 8 represent these isochrons, superimposed on a map representing Europe around 12,000 BP. The sea level was then at -80 m, a rough average value for the whole period of the Upper Palaeolithic that interests us here. The successive chronological areas of homogeneity the isochrons - represent, respectively, the Neanderthal contraction and the modern human invasion on the map such as they can be modelled from the data. Results Neanderthal Before 40,000 BP, the Neanderthals are present everywhere, within the longitudinal limits (about 17-20"E) of the reconstituted presence 7). At 40,000-37,500 BP, three abarea (FIGURE sence zones appear, in Britain, latitudinally in a band of 500-600 km width across eastern Europe, in northern Italy and along the coast of northwestern Spain. At 37,500-35,000 BP, there is a vast absence zone surrounding the western part of the European peninsula. In the Iberian peninsula the zone of initial absence has increased considerably and reaches the Atlantic coast. From 35,000 BP onward, it is easier to describe the presence areas than to describe the large absence areas. At 35,00032,500 BP, there are only two presence areas. One is located at the lower half of what is now France, but reaches the Pyrenees only in the west, along the Atlantic, and does not reach the Mediterranean in the east. The other is located in the western third of the Iberian peninsula, on the Atlantic front. At 32,500-30,000 BP, these two areas are strongly contracted. They disappear at 30,000-27,500 BP.

Mod ern h urn ans Before 40,000 BP, modern humans are absent, within the longitudinal limits (about 30"E) of the reconstituted presence area (FIGIJRE 8). At 40,000-37,500 BP two very small presence areas appear to the east (both roughly at 49"N). One is below the Carpathian mountains, the other is represented by one point (13"E) above the Danube, which is not visible at the definition scale of the map in FIGURE 8 . At 37,50035,000 BP, a large presence area covers eastern Europe, obliquely on longitude 13-24"E. Three

NEANDERTHAL CONTRACTION AND MODERN HUMAN COLONIZATION OF EUROPE

549

FIGURE 7. Isochrons of the Neanderthal contraction superimposed on a m a p representing Europe around 12,000 BP (sea level about -80 mi.

FIG~JR 8. EIsochrons of the modern h u m a n invasion, superimposed on a m a p representing Europe around 12,000 BP (sea level about -80 m).

more very small presence areas appear, scattered at the latitudes 51-41N, and longitudes 5OE-3"W. At 35,000-33,7500 BP, the large presence area continues its slow western expansion, the other areas extend separately with no connection between them. On both sides of the Alps two new presence areas appear. At 33,75032,500 BP, all of the colonization areas are connected. At 32,500-30,000 BP, a small zone of

absence remains. It is located in the southernmost western part of the Iberian peninsula. By 27,500 BP, all of Europe is colonized.
Discussion The scenario of invasion-contraction suggested by the two maps can be described in the manner that follows. At 40,000-37,500 BP, a colony of modern humans penetrates into Europe,

550

JEAN-PIERRE BOCQUET-APPEL & PIERRE YVES DEMARS

---

I
\ \ \

S
Y

20L

\
\

I
Homo

~
Time BP
perhaps via the corridor of the Great North European Plain, located above the Danube. At 39,000-36,000 BP (41,400-38,500 radiocarbon years BP), this zone corresponds to coniferous open woodland (van Andel & Tzedakis 1996). At 37,500-35,000 BP, a vast area of colonization extends into eastern Europe, from north to south. Elsewhere in Europe, small colonies are established in the west that are very dispersed from each other (>500 kin). At 35,00033,750 BP, the main eastern area of colonization continues its western and southern expansion. Small isolated colonies extend or appear, as independent centers of regional expansion. At 33,750-32,500 BP, all of the colonies are connected and form a single population area which reaches the Ebro river in the Iberian Peninsula. Large holes of populations remain at different points (Rhone Valley, central Italy). At 32,50027,500 BP, all of the Iberian Peninsula is colonized. This occurred much more slowly than the rest of the continent. This slowness could be due to the ecological conditions prevailing in this peninsula, that were much less favourable to the herds of ungulates than in the north, and/or to the presence of Neaiiderthals in this area. This marks the end of the expansion of the modern humans in Europe. At 40,000-37,500 BP, the Neanderthal contraction begins, in several poles (north, southeast, southwest), preceding the 'hole effect' observable from 35,000 BP on. Perhaps it somewhat precedes the arrival of modern humans. This hole effect is a formation process of two

o f t h e presence ~ area, ~ at each time period, for Neanderthals and modern humans.

events. First, that of a single area of contraction in the west of the European peninsula, occupying primarily a large band along the Atlantic front. Second, this area splits into two refuge zones. One is centred on the southern two-thirds of France, delimited by the Loire river, the other on the western third of the Iberian peninsula. This could represent the beginning of a fragmentation, typical of an extinction process. These two refuge zones narrow continuously until disappearing before 27,500 BP. The expansion or contraction rate of a pop"lation can be estimated by the square root of the presence area at each time period (Skellam 1951). This is shown for both groups in FIGURE 9, the northern inhospitable areas having been subtracted. For modern humans the expansion rate decreases asymptotically, with an acceleration at 35,000-32,500 BP. For Neanderthals the regression rate is linear, with an acceleration at 37,500-35,000 BP. The space-time coexistence of the two metapopulations, which is obtained by the difference between the two continua of dates (maximum dating for proxy of modern humans - minimum for proxy of Neanderthal <o), follows a pattern determined by the progression of the colonization of modern humans and the regional stability of the contraction areas of Neanderthals. It must be understood that coexistence refers to the simultaneous presence of cultural attributes of the two metapopulations, on the same geographical area, at

NEANDERTHAL CONTRACTION AND MODERN HUMAN COLONIZATION OF EUROPE

551

1 32.530BP

30-27.5 BP

FIGURE 10. Space-time estimate of Neanderthal-modern humans coexistence. In grey scale: space-time coexistence.

the definition scale of the isochrons. This coexistence does not give information on the nature, intensity and frequency of possible interactions. Two small groups wandering within the same zone will never meet as they are separated by one century (or one millennium) in time. Before 37,500-35,000 BP, two very small coexistence areas are visible. These correspond to the two initial northernmost and southernmost colonization areas of the modern humans. At 35,000-32,500 BP, an area of regional coexistence appears, located in the southern half of France. Another area appears in the southwestern point of the Iberian peninsula at 32,500-30,000 BP. The first coexistence area lasts from 35,000 to 27,500 BP (7500 years), the second from 30,000 to 27,500 BP (2500 years). Coexistence, therefore, does not seem to be general, but rather regional in the two initial refuge zones of Neanderthals that were gradually colonized by the modern humans. Concluding remarks The two-stage approach of descriptive and spatial statistics made it possible to highlight a coherent pattern of invasion-contraction, moving chronologically from east to southwest. This approach relates to the hypothesis-generating phase. It provokes other questions related to climate, ecology and demography. For example, around 36,000 BP the slow climate trend toward the Glacial Maximum is characterized by an increase in aridity. Is the Neanderthal contraction related to the simultaneous effects

of two factors, the impact of the cold and the presence of modern humans? In a first phase, the impact of the cold would have caused an increase in aridity along with the Neanderthal regression in the Atlantic refuge zones at optimum precipitation; in a second phase these refuge zones would have been very slowly colonized by modern humans. In this work, demography was approached indirectly, via the asymptotic increase of the modern human expansion and the linear decrease of the Neanderthal contraction. To the process of expansion-contraction of the presence areas must correspond a similar demographic process. Lastly, to which migration models and to which filling does the modern human expansion correspond? By being given the spatial distribution of the archaeological sites, mainly in the form of concentration areas (Bocquet-Appel & Demars in press), one may ponder whether modern human expansion did not take place following a model of hierarchic diffusion (Cliff et al. 1981). This would be by slow progression, through discrete units that were determined by ecology, i.e. in jerks or by steps, from locality to locality ecologically favourable, and not in a quasi-continuous manner. Archaeologically,it can be considered that two great theories on the origin of the Aurignacian in Europe clash. They can be summarized as follows (for a summary see Mellars 1996): Either the Aurignacian arose spontaneously from the Mousterian in various areas of Europe without any connection between them,

552

JEAN-PIERRE BOCQUET-APPEL & PIERRE YVES DEMARS

Or the Aurignacian is intrusive, coming from the east, probably hom the Middle East. It would be necessary to moderate this view, however, according to the authors: in particular some consider that Mousterian industries would have been manufactured by modern humans, and others think that the Aurignacian could be the work of Neanderthals. The scenario which the analysis of 14Cdates suggests consolidates the second theory. However, the duration of coexistence in the south of France and the north of Spain between the last Neanderthals and modern humans largely exceeds what the archaeological data would suggest. The Neanderthal-modern human coexistence, in the refuge zone in France, (as outlined by the 14Cdates analysis),finds weak archaeological support with two cases of inter-stratificationof ChStelperronianAurignacien (Piage, Roc de Combe). But the residual presence of Neanderthals in the south of the Iberian peninsula is now generally accepted (Duarte et al. in press; Hublin in press).
References BAR-YOSEY,0. & L.R. KRA (ed.). 1994. Lute Quaternary chronology of the eastern Mediterranean. 14th Internation Radiocarbon Conference. Tucson (AZ): IJniversity of Arizona. BOCQUET-APPEL, J.P. & P.Y. DEMARS. I n press. Populational kinetics in IJpper Palaeolithic in Western Europe, [oiirnal of Archaeological Science 2 7. CLIl'F, A.D., P.HXGGETT, J.D. ORD&G.R. VERSEY. 1981. Spatial diffusion. Cambridge: Cambridge University Press. DIJAKTE, C., J. MAIJRICIO, P.B. PETTI'W, P. SOUTO, E. TRINKAUS, H. VAN DER PLICHT & J. ZILHAO.I n press. The early Upper Paleolithic human skeleton from the Ahrigo do Lagar W h o (Portugal) arid modern human Emergence in Iberia, Proceedings of the National Academy oJ Sciences USA. HUBLIN, J.-J. In press. Moderninon-Modern hominid interac-

The qualitative degree of robustness of this scenario generated by the I4C dates currently available, given the introduction of new dates, is unknown. But it would seem that in its broad outline it is relatively stable, at the degree of definition of the isochrons, due to the spatial correlation of the data taken as a whole. The dates determine surface reliefs, on two tilted planes in the eastern (top) and southwestern (bottom) directions, roughly parallel, one representing the invasion, the other the contraction. A greater number of data points, better spatially distributed, should improve the space-time degree of definition of this global scenario.
Acknowledgements. We thank W Davies, who kindly communicated to us some Italian I4C dates, a n d L. Aiello, C. Gamble, M. Lahr, P. Mellars, a n d C. Stringer, for valuable comments o n a n early draft during the workshop 'Human dispersal panel' of the 'Stage isotopic 3 (10s 3)' project (Tjeerd van Antlel, Department of Earth Sciences, University of Cambridge, project coordinator), Renee Garcia a n d two anonymous referees.

tions: a Mediterranean perspective, in 0. Bar-Yosef & D. Pilbeam [ed.), The geography of Neandertals a nd Modern Hiinions in Europe a nd the Greater Mediterranean. Cambridge (MA):Peabody Museum, Harvarrl University. P. 1996. The Neanderthal legacy. An archaeological MELLARS, perspectivefrom Western Europe. Princeton (NJ):Princeton University Press. J.G. 1951. Random dispersal in theoretical populations, SKELLAM, Biornetrika 38: 196-218. VAN ANDEL, 'KH. & P.C. 'TZEDAKIS. 1 Y Y 6 . Palaeolithic landscapes of Europe and environs, 15O.flflO-25,OOO years ago: an overview, Quaternary Science Revicws 1 5 : 481-500. WACKERNAGEL, H. 1998. Multjvarinte grostatistics. Heidelberg (NY): Springer.