{"id":1317,"date":"2019-10-04T10:27:40","date_gmt":"2019-10-04T10:27:40","guid":{"rendered":"http:\/\/www.marekrei.com\/blog\/?p=1317"},"modified":"2021-03-11T18:23:00","modified_gmt":"2021-03-11T18:23:00","slug":"geographic-diversity-of-nlp-conferences","status":"publish","type":"post","link":"https:\/\/www.marekrei.com\/blog\/geographic-diversity-of-nlp-conferences\/","title":{"rendered":"The Geographic Diversity of NLP Conferences"},"content":{"rendered":"

The growth of interest in NLP technology, fuelled largely by investment in AI applications, has been accompanied by unprecedented expansion of the preeminent NLP conferences: ACL, NAACL and EMNLP in particular. Grzegorz Chrupa\u0142a\u2019s graph<\/a> shows the rapid growth in submissions to each conference in recent years:<\/p>\n

\"\"<\/a>
Figure 1. Graph of conference submission counts, by Grzegorz Chrupa\u0142a<\/figcaption><\/figure>\n

These trends can also be seen in the blog post by the ACL 2019 Chairs<\/a>. Meanwhile, acceptance rates have tended to remain constant, meaning that the conferences have grown in line with submissions, e.g. from ACL 2019<\/a>:<\/p>\n

\"\"<\/a>
Table 1. Acceptance rates at ACL 2019.<\/figcaption><\/figure>\n

NLP has developed as a field narrowly focused on English, a point highlighted by the recent emergence (and need for) the #BenderRule<\/a>. The ability of NLP models to deal with English is important, due to its status as an international language of politics, commerce and culture, and indeed with these tools we can handle, for instance, more than half the content on the internet<\/a>. But with such a strong focus on English we miss cross-linguistic insights and lack coverage for the majority of languages in the world.
\n<\/p>\n

Many NLP researchers work cross-linguistically but still there is a core assumption that the L in NLP means written English, to such an extent that this fact is not always explicitly stated in papers and presentations. One way to continue to diversify the scope of the field would be increased diversity in geographic representation. This would give exposure to wider concerns than those of current participants in NLP research, and (optimistic assumption) lead to evaluation of new methods on a larger set of languages.<\/p>\n

To take a measure of current geographic diversity in NLP, we extracted as many author affiliations as possible from fulltext papers in the ACL Anthology<\/a> for 5 major conferences held in 2018: ACL, NAACL, EMNLP, COLING and CoNLL. We associated each author\u2019s institution with a city and country, making some simple assumptions in the case of multinational organisations.<\/span>1<\/sup><\/a><\/span><\/p>\n

In total from the N fulltext papers in the Anthology we extracted 2695 author affiliations. Within this set of affiliations we found 653 distinct institutions, in 325 different cities and 56 countries. These counts are impressive at first glance, and they point to a field with a healthy level of global interest. But recall that the United Nations recognises 193 member states, so the 2018 conferences achieved 29% coverage<\/span>2<\/sup><\/a><\/span> and inevitably there\u2019s a bias towards North America, Asia, and Europe in terms of the regions represented by our list of countries. In particular there is a very stark lack of African representation in the dataset.<\/p>\n

[table]
\nRegion,N.author affiliations
\nNorth America, 1114
\nAsia, 826
\nEurope, 641
\nMiddle East, 54
\nOceania, 44
\nSouth America, 11
\nAfrica, 5
\n[\/table]

Table 2. The number of author affiliations by region.<\/figcaption>We illustrate the count of authors per country for ACL, NAACL, EMNLP, COLING and CoNLL 2018 with the chloropleth maps below. The first shows author counts as they are, the second shows the counts normalised by 2018 population counts.<\/span>3<\/sup><\/a><\/span><\/p>\n
<\/div>
Figure 2. Paper count by country at the 2018 NLP conferences.<\/figcaption>
<\/div>
Figure 3. Normalized paper count by country at the 2018 NLP conferences.<\/figcaption>These findings probably won\u2019t have surprised you. Nevertheless we hope that they are thought-provoking. We wonder how we as a field can encourage wider participation from under-represented regions? We assume that would be a good thing. Do you agree? Comment below.<\/p>\n

It is also a matter of concern that academic research in general lacks demographic diversity, with NLP being no exception. There are root causes here in terms of education systems, gender bias and global inequality, which are wider problems than NLP can solve alone. But doing everything possible to diversify the field would be a very positive move, especially given the link between AI, power and discrimination (see the \u2018discriminating systems\u2019 report by the AI Now Institute<\/a>).<\/p>\n

It may inform discussion if we report one other statistic we\u2019ve been able to retrieve thanks to data sharing by Emily Bender & Leon Derczynski, the program chairs for COLING 2018. They provided us with anonymised submissions data from that conference, in which each submission was associated with a country thanks to the contact details provided by the corresponding author, as well as a final decision (accept, reject, ineligible, withdrawn, desk-reject). It became clear that acceptance rates were broadly equal, and gladly a bias against papers from certain regions was not apparent.<\/p>\n

[table]
\nCountry, Submissions, Acceptance Rate
\nBelgium, 5, 0.8
\nNetherlands, 17, 0.71
\nHong Kong, 8, 0.63
\nNorway, 5, 0.6
\nSingapore, 14, 0.5
\nIsrael, 8, 0.5
\nItaly, 13, 0.46
\nSwitzerland, 7, 0.43
\nCanada, 24, 0.42
\nDenmark, 5, 0.4
\nFrance, 28, 0.39
\nGermany, 67, 0.39
\nUK, 36, 0.36
\nUSA, 194, 0.35
\nTurkey, 6, 0.33
\nJapan, 59, 0.32
\nChina, 297, 0.3
\nSouth Korea, 20, 0.25
\nCzech Republic, 9, 0.22
\nAustralia, 9, 0.22
\nIndia, 76, 0.21
\nTaiwan, 11, 0.18
\nRussia, 7, 0.14
\nSweden, 7, 0.14
\nIreland, 7, 0.14
\nBrazil, 15, 0.13
\nSpain, 16, 0.13
\nIran, 5, 0
\n[\/table]

Table 3. Acceptance rates at COLING 2018 by country, for countries with at least 3 submissions.<\/figcaption>This finding suggests that the lack of publications and representation from certain countries may be traced back long before conference submissions are made. The problem must be more deeply-rooted, at the institutional or societal level, hence not one that NLP alone can solve. However we assume that there are still steps we can take to make things better.<\/p>\n

Of course we are not alone in hoping for increased diversity in NLP (conferences). We have noticed some positive developments in this regard especially in the past two years. For instance, COLING 2018 ran a writing mentoring programme<\/a> expressly to help authors new to the field. The importance of guidance for those to whom (scientific) English is not a first language should not be underestimated. Nor should we assume that it is equally straightforward for all to learn English, with teaching quality and access to resources varying enormously by region (see John Clegg\u2019s essay about sub-Saharan Africa<\/a>, for example). There are also occasional calls for papers to be accepted in all languages<\/a>. Perhaps there could be WMT shared tasks expressly working towards automatic paper translation into many languages.<\/p>\n

NAACL has taken several positive steps including a diversity & inclusion committee<\/a> for its 2019 conference, live captions by Microsoft<\/a>, and the option to give remote presentations<\/a> (good for those who cannot or do not like to travel, or cannot afford it). There\u2019s also a NAACL emerging region fund to foster a larger and more cohesive comp-ling community in Latin America<\/a>; meanwhile it\u2019s been announced that the 2021 conference will be located in Mexico City<\/a>, the first time it’s gone outside the US and Canada since the 1st conference in 2000.<\/p>\n

2018 saw the launch of the Asia-Pacific Chapter of the Association for Computational Linguistics<\/a> (AACL), which is organising its first conference next year<\/a> (co-located with IJCNLP) in Suzhou, China. One good reason to take our major conferences to new countries is to equalise the barrier of obtaining visas to attend. Marina Dubova makes the point<\/a> that potential participants from developing countries might only get visas for a minority of CogSci conference venues in recent times, as they have been almost exclusively held in the US, Canada and western Europe.<\/p>\n

While there are positive signs in the Americas and Asia-Pacific regions, there is still a notable lack of NLP events and activity in Africa. ACL 2022 offers an opportunity to correct that with a call recently sent out to the members\u2019 list for hosts in the Europe\/Middle East\/Africa region in July or August 2022. The interest and value in NLP (and AI more generally) in Africa is indicated by the creation of a Google AI office in Accra, Ghana<\/a>, ICLR 2020 being held in Addis Adaba, Ethiopia<\/a>, and a heavily over-subscribed master\u2019s course run by the African Institute for Mathematical Sciences<\/a>.<\/p>\n

We set out to write this post with a focus on geographic diversity in NLP. In the meantime Strubell, Ganesh & McCallum\u2019s paper on CO2<\/sub> emission<\/a> estimates from NLP model training made a telling impact at ACL 2019 and we realised that this issue is of wide interest in the field. We therefore sought to quantify CO2<\/sub> emissions from conference attendance, by summing the air travel emissions for all locations in our database to the conference venue. Our calculations, made with some major caveats, suggest that delegate travel to the major NLP conferences in 2018 amounted to 2.1 million kg (4.6m lb) of CO2<\/sub> emissions.<\/span>4<\/sup><\/a><\/span> This on top of the tonnes of CO2<\/sub> put out training the models described at those conferences! There is of course much that is positive about in-person experience of international conferences, but each of us in the field has a responsibility to consider whether we should cut down on our travel. Some leading academics have already gone cold turkey<\/a>, while others call for virtual attendance<\/a> as a normal option<\/a>. Put this together with a recent study finding no relationship between air travel and metrics of academic productivity<\/a> and it\u2019s clear that reducing long-distance conference travel should be something to consider.<\/p>\n

\"\"<\/a>
Figure 4. Flight paths for the major NLP conferences in 2018.<\/figcaption><\/figure>\n

In summary, we wrote this blog post to highlight a lack of global representation at NLP conferences. This won\u2019t be a surprising finding but we hope it is thought-provoking. How can the situation be improved? Of course, the root causes of the problem are much bigger issues than NLP can address alone. However, we\u2019ve pointed to positive developments relating to NLP conferences, which we think could help improve the situation and should be widely adopted: writing mentoring programmes, diversity officers on conference committees, and remote presentation. In addition, holding more conferences in more regions (rather than allowing the existing few conferences to grow unchecked) is probably the better route to choose, both for diversity and environmental reasons. The first AACL conference in 2020 is very encouraging. We hope that an African bid might come through for ACL 2022, that EACL will take place more frequently, and that we\u2019ll soon see inauguration of a South American (SAACL) and African (AfrACL) chapters of the ACL. How do we lobby for any of these things to happen? Please comment below!<\/p>\n

\n

This is a guest post by Andrew Caines<\/a>, written in collaboration with Marek Rei<\/a>. Andrew is a postdoctoral researcher at the University of Cambridge, working on automated language teaching and assessment.
\nThe statistics in this post are based on the data crawled for the previous post on analysing ML\/NLP publications in 2018<\/a>.
\nWe chose to use 2018 papers, as all the publications for 2019 are not available yet. Once these proceedings are published, there will be a new analysis coming for 2019 as well.<\/p>\n