Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
154,584 characters · 19 sections · 127 citation commands
Deep Learning in Science
Most economic and policy analyses on the new wave of technological changes triggered by Artificial Intelligence (AI) and robotization have looked at the effects these technologies can have on economic growth aghion_2017, labour market and productivity dynamics furman_2019, acemoglu_2020, vanroy_2020, changes in skills graetz_2018, and inequality and discrimination oneil_2016. The paper at hand deals with the diffusion of Deep Learning (DL) in science and its consequences on scientific development. Our overarching goal is to add some empirical insights into the broader question of how AI shapes the process of knowledge creation.
The theory of re-combinatorial knowledge creation holds that new knowledge predominantly results from the recombination of existing pieces of knowledge weitzman_1998, uzzi_2013, wang_2017. The recombination principle opens the possibility of exponential knowledge growth. Indeed, measurable research outputs such as papers, patents, or innovations have been subject to high enduring growth rates over the last century. Yet, recent empirical evidence suggests that research productivity is ever falling and new ideas are increasingly getting harder to find bloom_2020.
Several reasons may account for this decline in research productivity. There may be a `fishing-out' effect whereby the number of useful recombinations is inherently limited and low-hanging fruits are harvested first. Technological opportunities may thus naturally decline, until a new principle or natural phenomenon is discovered, which in turn opens up a plethora of possible combinations. Another possibility is that the potential for useful recombinations is ever increasing, but we have more and more difficulties in realizing the existence of those recombinations due to cognitive and social limitations fleming_2001. Using existing knowledge effectively involves the challenge of searching for potentially relevant bits of knowledge, properly assessing their quality, ensuring their relevance for a given context and legitimizing their use in the absence of a universal canon. All of this becomes increasingly difficult within an expanding knowledge landscape that is not only becoming richer, but also more rugged. The `knowledge burden' translates into increased specialization and fragmentation in science where sub-disciplines are flourishing and researchers are working on fewer and fewer topics jones_2009. Interdisciplinary research and teamwork can only partially recover the potential of cross-fertilization that is lost. Finally, even within the most narrow specialization, researchers are increasingly confronted with needle-in-the-haystack problems agrawal_2018. For example, discoveries in the pharmaceutical sector have become progressively more difficult to achieve due to the proliferation of plausible targets for therapeutic innovation pammolli_2011. Similarly, in molecular biology microarrays assess the individual activity of thousands of genes, among which a few of interest must be identified leung_2015.
Expectations are high that AI may resolve at least some of these issues. In particular because of the numerous breakthroughs and rapid improvements in predictions achieved with DL. The principle idea behind DL is that any mapping from input to output, even the most complex, can be arbitrarily approximated through a (deep) chain of very simple mappings that can be fitted with data on exemplary projections. This idea gave rise to a broader family of (data and computing intensive) machine learning methods that are used to discover representations, invariance and laws, unusual and interesting patterns that are somehow hidden in high-dimensional data hey_2009, brynjolfsson_2014, lecun_2015, agrawal_book_2018.
The diffusion of AI in general and DL in particular across scientific disciplines has been documented in two previous empirical studies: i) cockburn_2018 and ii) klinger_2020. This paper complements these studies by providing a more fine-grained identification of deep learning research (compared to .i) and evidence on a larger multidisciplinary Web of Science sample (compared to .ii). Our results corroborate the idea that DL serves as a general method of invention across the sciences and around the globe.
Given its diffusion, how does DL affect scientific development? agrawal_2018, in a theoretical growth model, propose that AI may alter the knowledge production function in combinatorial-type research problems by affecting either `search' (i.e., knowledge access) or `discovery' (i.e., combining existing knowledge to produce new knowledge). AI in search makes potentially relevant existing knowledge available to the researcher. AI in discovery helps identifying valuable combinations among the available knowledge elements. In a needle-in-a-haystack problem, search would arrange the haystack and discovery would then find the needle. DL search can enhance haystack quality by yielding more, and probably more relevant, components; whereas DL discovery can increase the chances and speed of finding needles. The distinction between search and discovery is certainly relevant and fruitful. Yet, it tells us little about the direction of knowledge development, because it only deals with one body (or one haystack to stay in the picture) of pre-existing knowledge elements. However, knowledge explosion has two sides: increasing knowledge within each domain (larger haystacks) and increasing number of domains (more haystacks). A fundamental question is therefore whether DL deals with knowledge explosion within a domain, or facilitates knowledge creation across domains.
There are many examples in science where DL search and discovery remain within the boundaries of established research areas -- e.g., protein-protein interactions (PPIs), nanoscale material properties, or power grid energy supply -- augmenting human scientific reasoning and finding unusual and interesting patterns in vast datasets. However, DL can also facilitate cross-fertilization across topics or sub-disciplines. Neural models for information retrieval (IR), for example, can establish connections that go beyond word similarities, but act on similarities of concepts, lexicons, semantic relations and ontologies mitra_2018. Cross domain recommender system can assist target domain recommendation with the knowledge learned from other domains zhang_2019. Also, DL is well suited to transferring methods that perform well in constrained, well-structured problem spaces (such as game playing or image analysis) to noisy, flawed and partially observed scientific problems. This is typically achieved by integrating previously unrelated data -- not only in the form of numerical measurements but also unstructured heterogeneous information such as text or images -- from different realms, and perform analysis on them zhang_2018.
This leads us to investigate empirically how DL contributes to science in terms of re-combinatorial novelty and impact. Our analysis here is confined to the health sciences. Although much of the discussion around DL is on whether DL qualifies as GPT and its potential to leverage re-combinatorial knowledge creation agrawal_2018, cockburn_2018, hain_2020, klinger_2020, our paper is, to the best of our knowledge, the first to empirically assess the effects of DL as a research tool for knowledge creation.\footnote{In a similar vein, furman_2020 investigate the impacts of the Microsoft Kinect gaming system (fuelled by AI pattern recognition software) on the rate and type of knowledge production in the domains of computer science and electrical and electronics engineering. Kinect automatizes several tasks required to track, collect, and analyze complex 3D motion data in real-time; as such, it can influence knowledge workers' behaviour. The study shows that AI research technology in knowledge production leads to an increase in research output, an increase in research diversity, and a shift in research trajectories.}
In this study, the concept of re-combinatorial novelty refers to novel re-combinations across domains, as proxied by scientific journals, whereas the concept of impact refers to the the relative importance of a work in the scientific community, as proxied by citation indices. Overall, we find that DL adoption is negatively associated with re-combinatorial novelty, yet shows the `high risk/high gain' profile of breakthrough research, reflected by a higher variance in citation performance. Our results suggest that researchers are using DL as a research tool primarily to cope with the explosion of knowledge within domains rather than across domains. Thus, DL seems to be currently deepening existing knowledge structures rather than overthrowing them.
The remainder of this paper is structured as follows. Section (ref) retraces the evolution of deep learning and its alleged effects on the process of knowledge creation. Section (ref) provides the method for identifying search terms through Natural Language Processing (NLP), our DL search terms, and sample construction. Section (ref) documents some aspects of the diffusion of DL in science. In Section (ref) we present the analysis on the contribution of DL to the health sciences. Section (ref) provides a discussion and concludes by indicating some areas for policy considerations.
Artificial Intelligence is at the heart of the current technological paradigm. This paradigm shares several similarities, in scale and scope, with previous technological revolutions that have shaped and fueled long-term cycles of economic growth and structural change. The term `Artificial Intelligence' was coined by the computer scientist John McCarthy in 1955 in the proposal for the Dartmouth Summer Research Project on Artificial Intelligence, which took place in 1956. This workshop was a seminal event for artificial intelligence as a field, whose objective would have been to “[m]ake machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves” mccarthy_1955. Since its inception, AI has suffered from shifting definitions, although most definitions have revolved around the simulation of intelligent behavior by machines, whereby intelligence generally implies the ability to perform complex tasks in the real-world environment and learn from experience.\footnote{Some definitions were more oriented towards describing the operational characteristics of an intelligent machine, while others focused on the objectives of AI research. We witness a collective effort to establish definitions that are understandable, technically accurate, technology-neutral and applicable to short- and long-term horizons. For instance, the European Commission recently refers to AI as “[m]achines or agents that are capable of observing their environment, learning, and based on the knowledge and experience gained, taking intelligent action or proposing decisions” annoni_2018. According to the OECD, “An AI system is a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments” oecd_2019. WIPO defines AI systems “[a]s learning systems, that is, machines that can become better at a task typically performed by humans with limited or no human intervention” wipo_2019.} Often the terms machine learning, deep learning, and artificial intelligence are used interchangeably. This Section aims to briefly retrace the history of AI research, emphasizing the different approaches to machine intelligence and especially the approach based on deep neural networks.
In the early days, AI has tackled and (often) solved problems that could be described by a list of formal mathematical rules. Problems of this kind are intellectually difficult for humans but relatively straightforward for computers because real-world knowledge can be hard-coded into formal languages and logical inference rules can be used to achieve solutions. This approach to machine intelligence is commonly refer to as `knowledge-based' approach. The typical architecture of a knowledge-based system includes a knowledge base and an inference engine (i.e., inference rules): the knowledge base contains a collection of real-world information and the inference engine enables the machine to deduce insights from the information stored in the knowledge base. This approach has been the dominant practice for the first decades. Applications, such as expert systems, were introduced in the 1970s and were aimed at simulating the judgement and behaviour of a human being who has knowledge and experience in a particular field. These applications proved to be very effective for solving certain types of problems (e.g., recommend antibiotics and dosages to clinicians based on the presence of bacteria and patient symptoms) but terribly scarce for problems that require a great deal of subjective and intuitive knowledge of the world, as well as the perceptual capabilities of the environment (e.g., performing sensory tasks such as recognizing a face in the midst of a large crowd). Much of these problems are indeed easy for humans to perform, but hard for humans to articulate formally and through mathematics nilsson_2009.
During the same period, an alternative approach to machine intelligence began to take hold in the scientific community. This approach soon became known as `machine learning', and consisted in designing intelligent systems with the ability to acquire their own knowledge by extracting patterns from raw data. In other words, machine learning methods construct hypotheses directly from the data through inductive inference. Here is a classic example: if a large data set contains several instances of white swans and no instances of swans of other colors, a machine learning algorithm may infer that `all swans are white'. Inductive inferences consists of hypotheses which are always subject to falsification by additional data; for instance, there may still be an undiscovered island of black swans. Machine learning soon proved to be a valid alternative to knowledge-based systems. Machines could tackle problems involving real-world knowledge and reach certain human abilities, such as recognizing simple objects. From the 1980s, machine learning became one the most prominent branches of AI. Yet, some problems remained. Suppose the goal of a machine learning algorithm is to recognize a face in a picture, then the machine may use the presence of a nose as a feature -- i.e., piece of relevant information of the real-world to extract for that task. However, describing exactly what a nose is in terms of pixel composition can be difficult since there are countless different shapes, shadows can modify and obscure part of the nose, and the viewing angle can further change the shape. All these attributes are known as factors of variations, essentially constructs in the human mind that can be thought of as high-level abstractions that help us make sense of the rich variability of the observed data. Traditional machine learning methods encountered enormous difficulties in extracting these high-level abstract features from raw data nilsson_2009, goodfellow_2016.
The `deep learning' approach to machine intelligence turned out to be a good solution to this problem. A DL system learns from experience and understands the world in terms of a hierarchy of concepts, with each concept defined through its relation to simpler concepts schmidhuber_2015, lecun_2015, goodfellow_2016. The DL approach has two important advantages. First, as with simple machine learning algorithms, the machine collects knowledge from past experience, hence the human does not need to embed in the machine all the formal knowledge necessary to attain a given goal. Second, the level of complexity and abstraction of concepts is no longer a barrier, since the machine can reconstruct and aggregate them on top of each other. Returning to the previous example, a DL system can represent the concept of a nose by combining simpler concepts such as angles and contours that are then aggregated in terms of edges. This hierarchy of concepts makes the learning process a process that can be thought of as being structured into multiple layers, hence the term `deep'.
The function mapping from a set of features to an output can be often very complicated. Deep learning breaks down the complex desired mapping into a series of simple nested mappings, each described by a different layer of the model. The variables that we observe are presented at the input or visible layer. Then a series of hidden layers extracts increasingly abstract features from the data. The term `hidden' represents the idea that there is no predetermined structure but it is the model itself that determines which concepts are useful to explain the relationships observed in the data. The general architecture of a DL system can therefore be thought of as a neural network because nodes in the input, hidden and output layers are vaguely similar to biological neurons, and the connections between these nodes can be thought of as somehow reflecting the connections between neurons hassabis_2017. Today there is no consensus on how much depth a learning system requires to be considered as `deep'. However, there is consensus on the fact that DL involves a greater amount of learned functions or concepts compared to traditional machine learning methods. This allows DL to achieve great performance in an incredible variety of tasks.
“Deep learning, as it is primarily used, is essentially a statistical technique for classifying patterns, based on sample data, using neural networks with multiple layers [...] Deep learning is a perfectly fine way of optimizing a complex system for representing a mapping between inputs and outputs, given a sufficiently large data set” marcus_2018. Although the term `deep learning' is recent, DL has a long and rich history dating back to the 1940s. The field of research has been re-branded many times, reflecting the influence of researchers who have contributed to its development over time and who came from different backgrounds. For the narrative of our study, we find it useful to understand why DL has only begun to spread in recent years.
In their in-depth review on the history of deep learning research, goodfellow_2016 identify three major waves of developments: (i) cybernetics in the 1940s--1960s has marked important developments in theories of biological learning and the training of simple models with a single neuron; (ii) connectionism in the 1980s--1990s has brought methodological advances that have allowed faster training of neural networks with a few hidden layers; and (iii) the recent wave that started around 2006 (and is still ongoing) during which the appellative `deep learning' was coined.
The first predecessors of DL were simple linear models aimed at emulating computational models of the biological brain. The neuroscience perspective was motivated by the idea that the creation of intelligent machines could be achieved by reverse engineering the computational principles, albeit greatly simplified, behind the biological brain and replicate some of its basic functionalities nilsson_2009, hassabis_2017. These models were intended to provide in turn some insights to better understand the brain and the principle of human intelligence. The McCalloch-Pitts neuron is perhaps considered the first linear model of brain function that could classify two categories of input on the basis of a set of weights, although defined by human operators. A few years later, rosenblatt_1958 proposed the first model, known as perceptron, that could learn weights directly from examples of inputs without human intervention.
Neuroscience has inspired many principles that today form the backbone of DL architectures, such as artificial neural network or theories of mammalian visual system for computer vision. The intuition that many computational units become intelligent only via their interaction with each other is indeed regarded as the dawn of DL systems. However, the limited knowledge of the brain to use it as a guide, soon posed a barrier to further theoretical and practical developments of the field. Several critiques were levelled mainly against the excessive simplification of biological learning, and this approach to artificial intelligence lost its popularity. Although neuroscience is still regarded as an important source of inspiration for DL, a recent bibliometric analysis of the evolution of artificial intelligence research and its related fields from the 1950s to the present suggests that neuroscience is no longer the predominant guide for the field. Modern DL research predominantly refers to many other areas including mathematics, information theory and computer science frank_2019.\footnote{There is still a tendency, often reinforced by the media, to perceive deep learning as an attempt to simulate the human brain. Our knowledge of the function of the human brain is yet very limited, and there is broad agreement in the scientific community that deep learning cannot be seen as an accurate model of how the brain actually works.}
In the 1980s a new wave of neural network research emerged in the context of cognitive science with a movement known as connectionism nilsson_2009. During the early 1980s, models of symbolic reasoning (knowledge-based approach discussed in Section (ref)) were slowly overtaken by models of cognition that could be anchored in neural implementation. Connectionists shared the idea that a large number of neuron-like processing units can achieve intelligent behaviour when they are intensely networked together, thereby emphasizing the role of hidden layers as a way to increase the complexity of interconnections between units. Great achievements were made during those years, including techniques and models that still play a fundamental role in modern deep learning. Examples include the concept of distributed representation aimed at capturing meaningful `semantic similarity' between data through concepts, the successful use of back-propagation algorithm to train deep neural networks which had previously been insoluble, and long-short term memory networks (LSTMs) for modelling sequences with long-term dependencies goodfellow_2016. However, deep networks were too computationally expensive to empower real-world applications with the hardware available at the time, so research on neural networks began again to lose some popularity. The decline was further accentuated by the introduction of other machine learning techniques, in particular kernel machines, which could achieve similar performances in various applications with much lower computational requirements.
In spite of the difficult period, neural network research was kept alive. The Canadian Institute for Advanced Research (CIFAR) via its Neural Computation and Adaptive Perception (NCAP) research programme brought together some leading machine learning groups led by Geoffrey Hinton (University of Toronto), Yoshua Bengio (University of Montreal) and Yann LeCun (New York University). This programme paved the way for the last wave of DL research, which officially began in 2006, with a major breakthrough in the efficiency of neural network training. An important constraint to the training of deep neural networks was traditionally due to a problem known as vanishing gradient, which meant that weights in layers close to the input level were not updated in response to errors calculated on the training data set. Geoffrey Hinton and colleagues introduced a topology of network known as deep belief network that could be efficiently trained using a technique called greedy layer-wise pretraining hinton_2006.\footnote{This technique is `layer-wise' as the model is trained one layer at a time, and `greedy' as the training process is divided into a succession of layer-wise training processes. The procedure acts as a shortcut leading to an aggregate of locally optimal solutions, which in turn results in a reasonably good global solution.} This strategy has finally enabled very deep neural networks to be successfully trained and to achieve cutting-edge performance. Since then, researchers have begun to popularize the term `deep learning' and to focus on the theoretical implications of depth in neural networks. Further innovations and important milestones in the field of DL have been achieved over the last decade, including convolutional neural networks (CNNs) and generative adversarial networks (GANs). Today we are still in this third wave of research and deep learning outperforms any other machine learning technique in almost any real-world application.
Scientific discovery can been seen as the process or product of successful scientific inquiry.\footnote{In the narrowest sense, the term discovery would refer to the alleged `eureka moment' of having new insights, although here we adopt its broadest sense -- i.e., we use the term discovery as a synonym for `successful scientific endeavour' as a whole.} Historically, the process of scientific inquiry has evolved through paradigms, seen as symbolic generalizations, metaphysical commitments, values and exemplars that are shared by a community of scientists and that guide the research of that community kuhn_1962.
For most of human history, scientists have been observing phenomena, postulating laws or principles to generalize the complexity of observations into simpler concepts. The laws of science can be viewed in fact as compressed, elegant mathematical representations that offer insights into the functioning of the universe. Originally there were only experimental and theoretical sciences. hey_2009 refer to empirical observation and logical (theory) formulation as the first and second scientific paradigm, respectively. Towards the middle of the last century, however, many problems turned out to be too complicated to be solved analytically and researchers had to start simulating. Science entered into a third paradigm, a paradigm characterized by the development of computational models and simulations to understand complex phenomena. Data and information have begun to grow and accumulate on an unprecedented scale, also benefiting from the advent of other technologies such as remote sensing and the Internet of Things. The search over an increasingly vast combinatorial search space have soon become prohibitive for humans. Common to all scientific paradigms is in fact the idea that scientists use existing bits of knowledge to produce new knowledge and this new knowledge becomes then part of the knowledge base from which subsequent discoveries are made. As the volume and the complexity of the knowledge landscape increase, human cognition becomes a major limitation to experiment and re-combine distinct elements of the knowledge stock, understand the landscape and push the knowledge frontier further fleming_2001, jones_2009. We are moving towards a fourth scientific paradigm, a paradigm in which scientific exploration is grounded in data-intensive computing with a massive deployment of intelligent machines capable of finding representations, rules and patterns from an ever-increasing volume of structured and unstructured data hey_2009, king_2009. Much of this paradigm shift can be attributed to AI systems enhanced by deep learning brynjolfsson_2014, agrawal_book_2018.
DL redefines and enriches the knowledge base by observing the real world through examples, thus affecting both the process of `search' and `discovery' agrawal_2018. As for search, DL can support access to knowledge at a time where we are witnessing an explosion of data and information, predicting which pieces of knowledge are most relevant to the researcher. DL-based cross domain recommender systems, for instance, offer high-quality cross domain recommendation by exploiting numeric measurements, images, text and interactions in a unified joint framework. Transfer learning can further improve learning tasks in one domain by using knowledge transferred from other domains, in turn catching the generalizations and differences across different domains. DL is well suited for transfer learning as it learns high-level abstractions that disentangle the variation of different domains zhang_2019.
As for discovery, DL allows a better prediction of which pieces of knowledge can be combined to produce new knowledge, and the value of that knowledge. In other terms, DL allows the researcher to identify valuable combinations in a rugged landscape where knowledge interacts in highly complex ways. Humans can indeed consider a few hypotheses at a time whilst machines can generate and test an almost unlimited number of (more complex) hypotheses, explore unknown experimental landscapes and select which hypotheses are worthy of testing using a sort of economic rationality nilsson_2009, daugherty_2018. There are very efficient forms of learning that allow intelligent machines to reduce the uncertainty associated with regions of experiment space that are sparsely populated with results. Deep active learning systems, for instance, dynamically pose queries during the training process with the aim at maximising the information gains in the search space. The machine proposes which regions to navigate on the basis of the amount of new knowledge that is likely to be obtained in a given region, whilst the researcher can further screen the regions according to priorities and insights.\footnote{Active learning is a successful example of the `human-in-the-loop' approach to balance exploration and exploitation in the presence of uncertainty. Human scientists and DL systems have indeed different strengths and weaknesses, and combining forces can create synergies and forms of human-machine collaboration that will eventually produce a better science than can be performed alone. Humans are characterized by creativity, improvisation, dexterity, judging, social and leadership abilities. On the other hand, machines are characterized by speed, accuracy, repetition, prediction capabilities, and scalability. Intelligent machines may augment human scientific reasoning, free up time, creativity and human capital, “[l]etting people work more like humans and less like robots” daugherty_2018.} DL may therefore overcome the `knowledge burden' within a scientific domain, but also act as a cross-fertilizer for knowledge recombination across domains. All these properties have allowed DL to qualify as the nucleus of a General-Purpose Invention in the Method of Invention cockburn_2018, klinger_2020.\footnote{Other advantages offered by DL as a research tool are worth mentioning. While human scientists get tired, a machine can work without interruption and preserve the same degree of performance on a given collection of tasks. Intelligent machines can also improve data sharing and scientific reproducibility since they can easily record experimental actions, metadata and procedures, and results at no (or very limited) additional cost.}
One important consequence to be gained from thinking of DL as a General-Purpose Invention in the Method of Invention is that its impact is not limited to its ability to reduce the costs of specific scientific activities, but also to enable a new approach to science itself, by altering the scientific paradigm in the domains where the new research tool is deployed hey_2009, king_2009, cockburn_2018. Exploring the rise of DL as a research tool and the impact it can entail on scientific development represents the backbone of our study.
We are interested in the development of DL in science across disciplines. Our empirical analysis of scientific publications rests on two databases: arXiv.org and Web of Science (WoS). In a first step, we use arXiv.org to develop an appropriate list of search terms referring to DL through Natural Language Processing of scientific abstracts from publications in `Computer Science', `Mathematics' and `Statistics' subject areas. In a second step, these search terms are used to query the WoS database and extract our sample of DL papers across all scientific fields.\footnote{WoS is widely used for scientometric analyses, and seems also appropriate in our case. WoS lists all scientific papers published in a defined set of journals and conference proceedings. This shortlist approach necessarily introduces some heterogeneity in the coverage of the different scientific disciplines depending on the type of outlet preferred. Nevertheless, WoS coverage of journals and proceedings seems vast enough to capture science dynamics more generally. We were able to gather detailed information about each publication, including title, keywords, abstract, publication year, journal information, topical information, author and institutional affiliations, as well as cited references.} This second step is straightforward. How to create the list of search terms requires more in-depth discussion.
Reliance on a list of search terms for document retrieval is a common practice in research on emerging technologies or sciences, where no relevant structural information is available (e.g., an a priori appropriate classification). Unfortunately, extant studies do not provide us with an authoritative `ready-to-use' list of search terms. cockburn_2018 use a list of AI-related search terms to extract publications from WoS, but their definition of AI (also if restricted to what they characterized as `learning' approach) is too broad for our focus on DL. The same applies for vanroy_2020 who adopt a keyword-based approach to select AI patents. klinger_2020 explicitly deal with DL publications, but they do not define search terms. Instead, they identify DL papers through topic modeling of abstracts from arXiv.org. This approach essentially results in the relevance of the topic `deep learning' for each article. Hence, it allows them to tag papers as DL papers, but does not result (immediately) in search terms.
Our first step is therefore to create a list of DL search terms.\footnote{Other alternatives would be available. A first alternative could be to start with a set of `core' deep learning papers and include additional papers citing them. Two arguments, however, speak against such an approach. DL core papers should be defined in the first place. Secondly, papers that cite citing papers (the logical second step of this snowball sampling approach) may or may not apply deep learning as research tool, and that would distort the sample for our subsequent analysis by increasing the presence of false positives. The second alternative could be to query WoS simply using the term `deep learning'. Although simple, this approach would lead to the opposite problem, that is to increase the risk of false negatives. Further details about this issue can be found in Section (ref) when we discuss robustness checks.} There are by now various methods to create search terms and queries (see, e.g., huang_2011, huang_2015). In general, an additional search term in the list may be useful by adding correctly identified documents to the sample (here documents related to DL), but may also be harmful when adding incorrectly identified documents (here documents not related to DL). In order not to `forget' relevant terms, evolutionary lexical queries start with a set of core terms, retrieve the corresponding documents and then use them to identify further terms, possibly in a loop. Additional bibliometric information such as citations, journals or authors can also be used. However, over-expanding the search query quickly raises the risk of false positives.
To find the right balance, studies often rely on experts with domain-specific knowledge to establish appropriate search terms. Due to the nature of emerging technologies, however, experts often lack a shared perspective. Thus, the delineation of the sub-domain and the identification of domain-specific terminology intertwine, eventually exacerbating the problem of validity and reliability.
We propose a data-driven approach to delineate the perimeter of the deep learning domain, and to identify recurrent terms in that domain. In a nutshell, our approach consists in training the word embedding model `Word2Vec' mikolov_2013a with scientific abstracts from arXiv.org's documents in order to learn DL-related terms.
Roughly speaking, text embedding algorithms such as Word2Vec process text by vectorizing words -- i.e., they project words from a text corpus into a common vector space. The structure of the projected space reflects the semantics of the text, so that the semantically related words tend to cluster together in vector space. We build on this idea. Using a text embedding algorithm, we project terms from scientific abstracts into a vector space. In that space, we identify the word cluster that includes the term `deep learning' in order to obtain other terms that relate semantically and syntactically to the DL domain.\footnote{Word embedding algorithms have recently gained prominence in Natural Language Processing community li_2018, but also in scientometrics and bibliometrics. On the one hand, text embedding may enrich the analysis of a given corpus. For example, citation analysis may be enriched through word embedding methods to identify why a citation has been given, the sentiment associated with the citation, and what exactly has been cited jha_2017. On the other hand, text embedding is used for the identification of the corpus itself, such as for document retrieval. For example, dynomant_2019 propose and discuss various word embedding techniques to identify `similar' documents in PubMed database.}
Text analysis is more a practice than a science. Many decisions have to be made, often iteratively depending on what works and what does not: from cleaning text input, hyper-parameter settings, to various post-processing steps. The following Section conveys the main ideas and discusses the choices we have made to produce the list of DL search terms. Appendix (ref) provides details.
Our training data consists of scientific abstracts from arXiv.org. Recall from Section (ref) that DL blends statistics and informatics, but develops predominantly within computer sciences. Informatics is a fast-developing field in which conference proceedings are traditionally very important. More recently, however, the rapid dissemination of research is (better) achieved via open access journals and platforms. Of these, arXiv.org is the most prominent. It hosts not only many, but also very recent and highly cited research papers on AI in general and DL in particular. Therefore arXiv.org provides us with a rich corpus for the identification of DL-related terms. We downloaded a total of 197,439 abstracts of papers that fall in the subject areas `Computer Science', `Mathematics' and `Statistics', over the period 1990--2018. The three areas represent roughly 50% of all arXiv.org documents in 2018, and only 10% in the early 2000s.
We use the abstracts as input for the word embedding algorithm `Word2Vec' mikolov_2013a, mikolov_2013b. The words of a vocabulary of size $V$ are positioned in a $D$-dimensional space, giving rise to $V \times D$ parameters to fit. Hence, each word is represented by a $D$-dimensional continuous vector -- i.e., the word representation. The Word2Vec algorithm fits word representations in such a way that the probability that two words are close together in the corpus increases with the vector product of their word representations. A very intuitive outcome is that words that tend to appear close to each other in the text will have similar vector representations, since word representations only produce a high vector product if their larger values are in the same components. A less intuitive, but more striking and useful outcome is that two words that do not co-occur together but co-occur with the same other words are also close in vector space. Both effects together produce clusters of syntactically and semantically related terms in the projected space.
This is particularly useful in our case. The term `deep learning' is certainly to be included in the list of search terms. Hence, we can identify in the vector space the cluster of terms including the term `deep learning'. Synonyms and other semantically related elements (e.g., `neural network') are likely to be identified. `Neural networks', a syntactically related term, may also be found in the same cluster. As terms show up in similar text contexts, they appear in the same cluster in the projection. On the other hand, terms that are not closely related to DL but, say to informatics or other machine learning methods, remain excluded (e.g., `support vector machine'). These terms sometimes also appear in text contexts similar to DL, but even more often in other contexts. Therefore, their word representation will be different and so will their cluster. By looking at the terms in the cluster `deep learning' in the projected space, we make sure we do not miss relevant search terms. In addition, the boundaries of the word cluster provide an indication on how to delineate the DL domain.
Clustering of syntax-related words is convenient as it reduces the needs for preprocessing. Stemming or lemmatization is no longer necessary because variants of the same word stem are clustered by design. In contrast, preprocessing $n$-grams that refer to idiomatic phrases is essential. Many technical terms are idiomatic phrases that consist out of multiple words. For example, the term `neural network' refers to one specific concept although it consists of two words. The Word2Vec algorithm produces for each word exactly one vector (embedding) and that vector does not vary with the context. Thus, by default, the word `neural' will be one vector and the word `network' another vector. This algorithmic feature is inconsistent with the fact that many words take on different meanings depending on the context. The word `neural', for instance, has a different meaning in brain research (probably referring to neural activity) than AI (probably referring to computational neural architectures). As proposed in mikolov_2013b, we take this issue into account during preprocessing by pasting together subsequent words (bi-grams) into single tokens whenever these subsequent words co-occur frequently in our full corpus, as explained in the Appendix (ref). For instance, `neural' and `networks' are pasted into one token `neural_networks'.\footnote{camacho_2018 discuss other (often more elaborated) approaches that are explicitly designed to handle this issue. However, pasting $n$-grams during preprocessing, as proposed in mikolov_2013b is very simple, intuitive, and turned out to be satisfactory.} Finally, the use of acronyms is a standard practice in the AI community. Several terms in our clusters were acronyms (e.g., ANN). We replaced the most frequent acronyms with their appropriate full names.\footnote{Acronyms are also quite convenient in the learning process as they allow to capture concepts without inflating the size of the vocabulary. For example, without any preprocessing of the corpus, a concept such as `deep convolutional neural network' can be represented by a single token (DCNN). The list that we have developed manually to convert the acronyms to full names is reported in the Appendix (ref).}
The data is then used to train the Word2Vec model in its Skip-Gram with Negative Sampling (SGNS) version, as discussed in mikolov_2013b. Fitting the model involves various parameter settings that are described in Appendix (ref). The main outcome of the model is one vector representation for each term in the vocabulary. We identify those terms that appear in the same cluster as `deep_learning'.\footnote{The results were obtained with $k$-mean clustering. The optimal number of clusters via the `gap statistic'. We also tried different clustering methods and the results were very robust in this respect; apparently because the estimated vector space is clearly structured.}
The resulting list of potential search terms includes individual words (uni-grams) but also technical terms consisting of multiple words. We decided to retain only those terms consisting of multiple words -- i.e., to remove all uni-grams from the list of search terms -- in order to remain conservative and include only terms that relate unambiguously to DL. Moreover, we retained only the 30 most frequent $n$-grams after having dropped terms that are too generic (e.g., `short_term' or `supervised_learning').\footnote{Since there is no established canon, our approach was a fairly exploratory trail-and-error process, aiming to balance false positives and false negatives. Considering 30 terms and removing too general terms, we were able to retrieve 260,459 DL documents from Web of Science (our sample for the analysis). To get an idea of the sensitivity, if limit the list to 30 terms but include generic terms, the query results into 639,317. If we limit the list to 20 terms, we identify 616,349 documents; if we increase the list up to 50 terms, we get 685,631 documents. And 711,905 documents when considering 80 terms. Therefore, as these numbers suggest, we preferred to adopt a conservative strategy and limit the presence of false positive in the sample.} A manual check of various random extractions from WoS confirmed that this choice greatly reduces the presence of false positives in the final sample. The exception being the term `neural_network' which may in fact refer to a biological neuronal network. We decided to keep that term, however, because the confusion between biological neural networks and artificial neural networks seems to be confined to the field of neuroscience. This issue is therefore marginal and does not affect descriptive statistics across scientific fields, nor any subsequent results on the role of the DL in health sciences (see also robustness checks in Section (ref)). The final list of search terms is shown in Table (ref). A more complete list of terms for all clusters identified through word embedding can be found in the Appendix (ref).\footnote{It is fair to remark that a neural network architecture may not necessarily be deep, although what is meant by depth is still a matter of contention (Section (ref)). This implies that our sample may also include `shallow' networks with only one hidden layer; a potential problem that also characterizes previous research on deep learning cockburn_2018, klinger_2020. Aware of this, we made sure that the diffusion patterns presented in Section (ref) are robust with respect to a much more restrictive definition of deep learning -- that is, restricting the sample to documents that contain only the terms with the prefix `deep' (e.g. `deep_learning', `deep_neural'). At the same time, a strict separation between deep learning and neural network makes no sense. The two are closely linked because one relies on the other to function. Put simply, without neural networks, there would be no deep learning. We omit any further reference to this issue in what follows.}
This Section documents the diffusion of DL in science across geographies and scientific areas. Our sample includes all publications in the WoS Core Collection that were published between 1990 and 2018, and have at least one of the search terms (Table (ref)) in their title, keywords or abstract. In total, we identify 260,459 DL documents (144,095 articles; 39,925 conference proceedings; 76,439 others).\footnote{To get an idea of sensitivity with respect to the most influential terms, if we consider only the term `deep learning' the query provides us with 15,085 documents. Adding `neural network' to the query pushes the number up to 219,782 documents.}
In this subsection, we provide some insights into the spatial diffusion of DL on a global scale. Spatial diffusion is not the focus of the paper at hand. However, the fact that DL spreads globally and that countries show patterns of specialization in DL research supports the idea that DL is a general and relevant method in science.\footnote{Our spatial observations also open up research questions that go beyond the scope of this paper. Particularly interesting seems to be the issue of competition and cooperation between national scientific systems, as well as alignment and differentiation in DL research which, in turn, can shape the trajectory of the technology. We leave these issues for future research.}
Figure (ref) shows deep learning science dynamics at the country level. A complete table with numbers is provided in the Appendix (ref). Each DL document is attributed to a given country when at least one author's affiliation is in that country. The upper left panel shows the pattern for the first period, 1990--1999. During that period, most of the documents (about 5,000) were published by scientists in the United States. Publishing activity is relatively low in absolute numbers in the European countries, Australia and China, and negligible or non-existent in most other countries. In the following decade, 2000--2009, China becomes the most prolific country with about 20,000 DL documents. The US ranks second with around 14,000 articles, whereas European countries and Australia grow sufficiently to preserve their relative strength. Interestingly, in an increasing number of countries DL research activity starts. These trends are reinforced in the third, last period, 2010--2018. Compared to the previous decade, China has doubled its DL research output, thus widening the gap with the US and, to a lesser extent, with the EU.\footnote{Two remarks are noteworthy here. First, a document with multiple affiliations in different countries is counted multiple times. We verified that weighted paper counts yield essentially the same patterns. Second, considering the EU as one single player would rank the EU first in the first and second period, and second in the third period.}
Figure (ref) documents regional specialization in DL research. Throughout, the scientific area (defined according to the WoS research areas) `Technology' takes high shares, accounting for around 70% to more than 80% of all publications. Yet, there is substantial variation across geographies. In Asia and Eastern Europe, DL activity centers in particular on `Technology' and `Physical Sciences'. In Western Europe and North America a larger proportion of DL research takes place also in `Life Sciences & Biomedicine'.
Taken together, we can conclude that DL diffuses rapidly on a global scale. A high volatility in the rankings has characterized the early stages of development of DL research, with some countries rapidly climbing up the ranking while others lagging behind. The main players are China, the United States and Europe. Also note that DL research activity is now present virtually everywhere. DL as a research tool seems to find applications in a variety of domains, and world regions show heterogeneous patterns of `specialization' in different scientific areas. These trends are in line with previous evidence cockburn_2018, oecd_2019, wipo_2019, klinger_2020, vanroy_2020 and consistent with the diffusion process of pervasive technologies.
Conceptually, a General Method of Invention (GMI) blends the concepts of Method of Invention (MI) griliches_1957 and General Purpose Technology (GPT) bresnahan_1995. The seminal study of Griliches discussed double-cross hybridization as a MI -- i.e., a way to breed new corn varieties for specific local environments. Double-cross hybridization however is not very `general' because it only applies to corn breeding. On the other hand, there are GPTs. These are technologies that originate in one sector and are usefully applied in many other sectors of the economy. Taking a dynamic perspective, innovation complementarities create positive feedback loops whereby innovations in the originating and application sectors reinforce each other. Combining these ideas, we refer to GMI as a MI that is applicable across many domains. Similar to GPTs, development of the GMI and complementary developments in application domains are mutually reinforcing.\footnote{Instead of GMI, cockburn_2018 use the term `General-Purpose Invention in the Method of Invention'. We note that a technology that qualifies as a GMI could be used for other purposes than invention (or more generally knowledge creation) and thus also qualifies as a GPT. For DL this is probably the case but is not the focus of this study.} \\
As discussed in Section (ref), the originating domain of DL is predominately computer science. Looking at the scientists involved and the influential publications that have pushed forward DL methods over time leaves little room for doubt. On that ground, it seems appropriate to follow cockburn_2018, and assume that DL publications in all areas apart from computer science represent applications of DL methods to address field-specific research questions -- i.e., adoption of DL as a research tool. However, when it comes to science, one may doubt whether such a strict separation between originating and application domain is actually useful and tenable. The history of scientific instruments, for instance, suggests that the development of instruments often coincide with their scientific use rosenberg_1992. In the case of DL, the connection between the `instrument' and the scientific application is given by definition because DL derives predictions from specific examples. This could flip the role of the application domains: Do these domains just supply data for DL instead of requiring DL for their science? We turn to that question below, after examining some dynamics of DL activity across domains.
Figure (ref) shows time trends of DL publications in our WoS sample by scientific area (Panels A, B, C, E, F). Panel D refers to `Health Sciences', defined as a subset of `Life Sciences & Biomedicine'; health science is at the core of our analysis in the next Section. Cross-classified papers are included in each relevant panel. Panel G, `All Documents', simply combines all papers from the WoS sample. Panel H, `arXiv', provides complementary insights on our arXiv.org sample (discussed in Section (ref) and Appendix (ref)). Looking at Figure (ref) as a whole, we note a rapid growth in DL research activity in all scientific areas. Yet, the volume of DL papers (blue line) is highly different across areas. `Technology' (Panel A) dominates all others, which is at least partly explained by the fact that it includes `Computer Science', the main originating field. With about five times fewer papers, `Physical Sciences' (Panel B) comes second, closely followed by `Life Sciences and Biomedicine' (Panel C). `Health Sciences' (Panel D) parallels `Life Sciences' (Panel C) because both documents sets are highly overlapping. Publication counts in `Social Sciences' (Panel E) are relatively low, and negligible for `Arts and Humanities' (Panel F). Panel G combines all WoS documents into one picture. In that panel, the (three-year average) growth rates (orange line) show a high growth of 10% in DL publication activity around 2005, a decline around 2010, and a subsequent recovery with steady growth rates reaching 20% at the end of the observation period. This growth pattern is close in form and magnitude to the one observed for `Technology' (Panel A), trivially because that is the dominating area; however, also the other areas exhibit very similar growth patterns.
The publication activity on arXiv.org (Panel H) follows essentially the same dynamics. Growth rates mimic the same shape over time but are about five times higher than growth rates in WoS panels. The comparatively higher growth rates may result from the fact that open platforms are increasingly popular as an efficient and fast way of communication between researchers, particularly in machine learning and computer science communities sutton_2017. The arXiv.org dataset corroborates the finding on the WoS dataset: strong growth of DL research in two waves across the sciences.
The overall number of DL-related documents varies over sub-disciplines within scientific areas (not displayed). The general trend in `Technology' is mainly driven by `Computer Science' (103,729 documents), `Engineering' (95,638) and `Automation & Control Systems' (24,721). In the case of `Physical Sciences', we find `Physics' (7,239), `Mathematics' (5,123) and `Chemistry' (3,702). And for `Life Science & Biomedicine' we see the preponderant role of `Environmental Sciences & Ecology' (2,632), `Neurosciences & Neurology' (2,032), and `Biochemistry & Molecular Biology' (1,728).
DL publication activity increases not only in absolute numbers but also relative to the overall number of papers in scientific areas; albeit from a low level. In 2018, for example, DL documents account for 2.6% of all papers in the category `Technology', 1.02% in Physical sciences, and 0.3% in `Life Sciences and Biomedicine'. Thus, DL publications still account for only a tiny fraction of the whole research volume, in particular in application domains. Yet, recent growth rates of shares are remarkable. DL has the highest growth rates in the `Life Sciences & Biomedicine' with 47.3% from 2017 to 2018. `Physical Sciences' comes second with a DL growth rate of 42%, and `Technology' shows roughly 18%.
The growth pattern of DL research with a first boom, subsequent decline, and a second (bursting) boom, reminds the double-boom-cycle that has been observed before for emergent technologies schmoch_2007. The narrative goes like this: a new, emerging technology seems at first to offer a high potential. High expectations trigger high development efforts -- the first boom. However, during these early development activities, actors learn about the difficulties to put the principle into practice. Most fail and stop their innovation activities, which puts an end to the first boom. Some continue and, as time goes by, may overcome important practical hurdles and demonstrate real benefits in praxis -- starting a second boom.
An in-depth analysis of citation dynamics suggests that the double-boom-cycle story holds for DL. We consider the top five cited references in each year of the observation period (i.e., documents with the highest annual shares of all cited references in our DL publications). This gives us a list of 18 unique articles and their corresponding citations counts, as shown in Table (ref). Using dynamic time warping (DTW) to measure dissimilarity between time series, we cluster these temporal sequences by mean of $k$-medoid berndt_1994. As shown in Figure (ref), we obtain two clusters. In the first period, most cited articles in our sample are theoretical contributions, including the possibility of using multilayer feedforward networks as universal function approximators, training algorithms (backprop), and parallel computing theories (cellular NN). In the second period, the most influential articles are no longer theoretical contributions, but rather articles that have shown how to put theoretical principles into practice -- with tremendous success in various AI competitions. These contributions include inventions that have brought enormous performance gains on real-world tasks, particularly for image and text analysis (deep convolutional neural networks and LSTM, as discussed in Section (ref)).
Observed dynamics are consistent with the idea of positive feedback between DL development -- within `Technology' -- and DL applications -- mostly in `Physical Sciences' and `Life Sciences and Biomedicine'. However, one might question the extent to which different scientific domains are inclined to incorporate deep learning methods as a practice in their disciplinary research. In other words, is there indeed a diffusion of DL into applications or rather a cross-disciplinary effect of DL?
We investigate this question by considering the cross-classification of publications in our sample. Each document is labelled by WoS as belonging to at least one subject category according to the journal in which it was published. In most of the cases a document falls into more than one scientific category. The extent to which publications in a given scientific area are cross-classified as computer science contributions may therefore proxy cross-disciplinarity with respect to computer science. For each broad scientific area and year, we calculate the fraction of deep learning documents that are (also) labeled as `Computer Science'.\footnote{We define `Computer Science' the set of the following Web of Science subcategories: `Computer Science, Artificial Intelligence'; `Computer Science, Cybernetics'; `Computer Science, Hardware & Architecture'; `Computer Science, Information Systems'; `Computer Science, Interdisciplinary Applications'; `Computer Science, Software Engineering'; `Computer Science, Theory & Methods'.}
Figure (ref) provides results. Turn now to the first, upper-left panel `Technology' (Panel A). Each point of the plot represents the average number of `Technology' DL documents cross-classified as `Computer Science' in a given year. For example, in 1990 about 60% of `Technology' DL publications fell (also) into the `Computer Science' category (the first red dot). The trend (blue line) follows a flat U-shape approaching around 70% in 2005, before decreasing to less than 50% by the end of the observation period. In 2018, over 50% of DL documents in the area `Technology' are no longer labeled as computer science contributions. The upper-right panel `Physical Sciences' (Panel B) shows an inverse U-shape, with an increase in cross-classified computer science documents of up to 20% in 2000, before falling down again to 10% by the end of the period. The pattern in `Life Sciences & Biomedicine' (Panel C) is different in that there is no increase of computer science cross-classification. Instead, there is a very high share of 70% at the beginning of the period which continuously decreases to about 20%, with significant drops around 2000 and again in 2010. The `Health Sciences' (Panel D) experience the same pattern. `Social Sciences' (Panel E) increase their share of computer science documents to 40% around 2010, followed by a sharp downturn. Finally, for `Arts & Humanities' the fraction of computer science documents is very noisy, and so we do not observe any particular tendency.
To interpret these patterns, recall that the areas `Technology', `Physical Sciences', and `Life Sciences & Biomedicine' exhibit the strongest DL publication activity in terms of absolute and relative numbers, as well as growth rates. For these three areas, DL activity takes off around 2010 (Figure (ref)). At the same time the fraction of publications that are cross-classified as `Computer Science' decreases, as shown in Figure (ref). These dynamics indicate that DL diffuses indeed from computer science, the originating discipline, into other application-oriented scientific disciplines. The high share of cross-disciplinary research involving computer science in the early periods could be explained by the fact that the transfer of DL and its adoption to the field of application requires close interaction between researchers from both domains. Computer scientists need to learn what can be done in practice and scientists in the application sector need to understand the potential of the (new) technology for their research.
In sum, the evidence presented so far points to a simple statement: DL meets the conditions of a General Method of Invention. The technology originates predominantly from computer science and is increasingly integrated as a research tool into many other scientific fields. Hence, it is a method of invention that is generally applicable in various domains. Moreover, the (joint) dynamics across sciences are consistent with mutual reinforcement between originating and application domains, a core idea of GPT. This opens the question of how DL affects scientific development in its application domains. We turn to that question in the next Section.
This Section deals with the impact of DL specifically in health sciences.\footnote{We delineate the `Health Sciences' by 83 Web of Science subject categories within `Life Sciences & Biomedicine' research area. The complete list of included categories can be found in Appendix (ref).} We focus on health sciences because it is among the scientific domains where the adoption of DL is more widespread and dynamic, as shown in Section (ref). Furthermore, AI in general and DL in particular have already led to a variety of innovations in the health realm -- improving healthcare systems, supporting clinicians in surgery, monitoring patient diseases. DL research demonstrated high societal impact in the short run miotto_2018. Investigating the impact of DL in scientific domains other than health sciences would certainly be interesting, but is beyond our limits. At this early stage of research on DL in science, contextualizing the empirical analysis is essential. Doing so for several scientific domains would not only exceed the page limit of an article but also our expertise in those domains. Furthermore, the empirical analysis itself is highly demanding in terms of data requirements and computational burden. We keep that manageable by focusing on one scientific area.
The next subsection (ref) illustrates how DL research has advanced several areas within the health sciences. Subsection (ref) develops the conceptual framework for the empirical analysis. Subsection (ref) discusses the data, methods, and results of the analyses.
In recent years, several areas within health sciences have seen a shift from systems with hand-crafted features (i.e., systems completely designed by humans) to intelligent machines that learn the features from data. As discussed in Section (ref), this approach is foundational to the DL principle. Provided that the architecture is optimally weighted, DL leads to an effective high-level abstraction of raw data, thus increasing perceptive and predictive capabilities lecun_2015, schmidhuber_2015. DL has enabled the development of multiple data-driven solutions in health informatics and biomedical research, making it possible to automatically generate features and reduce the amount of human intervention in the process ravi_2017. Below we provide a cursory review of the main areas in which DL finds applications, highlighting the benefits that the technology can bring in the process of knowledge search, experimentation, knowledge production and patient health care.
One domain in which DL has gained considerable success in recent years is translational bioinformatics, understood as the study of biological processes at the molecular level by means of biomedical and genomic data and informatics methodologies leung_2015, ravi_2017. The application of DL in translational bioinformatics has turned out to be particularly relevant for genomic research. This area of research aims to determine how variations in the DNA of individuals can affect the risk of different diseases and find causal explanations in order to design targeted therapies. The details of the mechanisms at work in the cell are hidden. What we can observe is the outcome of many layers of biophysical processes and interactions, most of which are not fully understood.
Progress in biotechnology has contributed to reducing the costs of genome sequencing and shifted the research focus on prognostic, diagnostic and treatment of diseases through gene and protein analysis. Modern biology allows also high-throughput measurement of many cell variables, including gene expression, splicing, and proteins binding to nucleic acids, all of which can be treated as training targets for predictive models marx_2013. Modern deep learning systems can accurately interpret the text of the genome just as the machinery inside the cell does, making it possible to explore the effects of genetic variations and potential therapies quickly, cheaply and more accurately than can be achieved using `standard' laboratory experiments leung_2015.
Most of the DL applications have been deployed for the accurate prediction of splicing patterns and gene variations, which is a key to providing early diagnosis of various diseases and disorders such as cystic fibrosis, Parkinsonism, spinal muscular atrophy, myotonic dystrophy, amyotrophic lateral sclerosis, premature aging, and dozens of cancers. An exhausting review of the literature on the advent of intelligent machines in genetic research suggests that computational methods will not be able to completely replace laboratory and clinical diagnosis, but should significantly reduce the time needed for these methods of analysis by reducing the search space for the hypotheses that need to be validated leung_2015, angermueller_2016.
Genomic medicine is not the only area where DL can bring benefits. The ability to abstract large, complex and unstructured data makes deep learning also an effective solution for the prediction of protein-protein and protein-compound interactions (CPI). CPI are crucial for drug discovery, the identification of new compounds and toxic substances, and for advances in pharmacogenomics. DL allows a richer representation of possible interactions beyond the genetic and molecular structural information encoded in large datasets, thus paving the way for data-driven discoveries. For example, a team of machine learning researchers took advantage of a number of different Quantitative Structure -- Activity Relationships (QSAR) datasets from Merck's drug discovery effort and significantly improved the state-of-the-art drug discovery pipeline, despite having no prior knowledge about the biochemical properties of training features ma_2015. The purely data-driven approach is not the only one possible when intelligent machines are available as research tools. Instead of examining the parameters of the model and coming up with an interpretation, researchers can also ask the system for presumable relationships between inputs and outputs that cannot be `checked by eye'.
Medical imaging is another domain in which the advent of deep learning, and especially of CNNs, has played a central role. As soon as it was possible to collect and catalogue medical images, computer systems were used to assist researchers and doctors in the analysis of these images. Indeed, whereas diagnosis based on the interpretation of images runs the risk of being highly subjective, computer-aided diagnosis (CAD) can result in a more objective assessment of the underlying disease processes. However, CADs have long suffered from some limitations mainly due to differences in shape and intensity of abnormal tissues (tumours or lesions) and variations in imaging protocol. Non-isotropic resolution in Magnetic Resonance Imaging (MRI), for instance, has been a major challenge to manage with traditional machine learning methods. Recent advances in deep CNNs have found fertile ground in the medical imaging research community due to their outstanding performance in various computer vision tasks, challenging the accuracy of experts in some tasks litjens_2017, shen_2017.
Image or exam classification (e.g., disease present or not) was one of the first areas in which deep learning made a decisive contribution to medical image analysis. Yet, the current range of applications is much wider and includes organ, region and landmark localization, object or lesion detection, organ and substructure segmentation, and lesion segmentation.\footnote{The detection task typically consists in identifying and localizing small lesions in the full image space. It represents an important pre-processing step in the clinical workflow for therapy planning and intervention. The segmentation task consists instead in identifying the set of voxels that make up the contour or the interior of the objects of interest. The segmentation of organs and other substructures enables quantitative analysis of clinical parameters related to volume and shape (e.g., brain analysis).} Both detection and segmentation tasks play a key role in the diagnosis of tumours and other diseases, and represent one of the most labour-intensive activities for doctors since accurate classification requires both local analysis on lesion appearance and global contextual analysis on lesion location litjens_2017. DL methods have also found applications in other medical imaging tasks, such as content-based image retrieval (CBIR) and combining image data with text reports (Rav\`i et al., 2017). Comprehensive reviews of the literature on the impact of DL in the field of medical imaging highlight that deep learning algorithms empower machines for automatic discovery of object features and automatic exploration of features hierarchy and interaction, once again supporting and facilitating the work of scientists and clinicians litjens_2017, shen_2017.\footnote{Despite incredible progress, the potential of deep learning in image analysis has not been fully unfolded. One of the main limitations seems to be the shortage of large labeled datasets. Labeling data is a time-consuming and expensive activity that requires some expertise. There is an ongoing debate about the possibility of crowdsourcing (even by non-experts) as a viable alternative for creating low-cost, truth-based medical imaging datasets litjens_2017.}
The development of intelligent devices and cloud computing has allowed the generation and collection of an incredibly high volume of health data from various sources in real-time. Wearable, implantable and ambient sensors, as well as the data they provide, enable continuous monitoring of health and well-being marx_2013, raghupathi_2014. The adoption of DL has increased the benefits of pervasive sensing in a wide range of health applications such as measure of food calorie intake, energy expenditure, activity recognition, sign language interpretation and the detection of anomalous events in vital signs (e.g., blood pressure and respiration rate). Most applications use DL algorithms to achieve greater efficiency and performance for real-time processing on low power devices. These devices are beneficial for science because they increase the understanding of diseases by enabling a more thorough and systematic analysis of the patient's condition.
Given its versatility, deep learning has also proven to be efficient in handling multimodal unstructured information by combining several neural network architectural components on big data infrastructures stored in hospitals, cloud providers and research organizations. An example of this type of data are the Electronic Health Records (EHR). EHR provides an extremely rich source of patient information that includes history details such as diagnoses, diagnostic exams, medications and treatment plans, immunization records, allergies, radiological images, laboratory and test results. DL permits an efficient navigation, extraction and analysis of these data, hence providing valuable information on disease management and the discovery of new patterns (e.g., long-term time dependencies between clinical events and disease diagnosis and treatment) that result in completely new hypotheses and research questions rajkomar_2018.
Public health has also witnessed an upsurge in deep learning applications. The latter involve epidemic surveillance, modelling lifestyle diseases (e.g., smoking and obesity) with relation to geographical areas, monitoring and predicting air quality, contamination of food and water supplies, and many more miotto_2018, ravi_2017. Traditional machine learning methods can accurately model several phenomena but have the limited ability to incorporate real-time information. On the contrary, current systems in public health studies are based on online deep learning and can build hierarchical models and encode information sequentially as new training datasets become available zhang_2018.
Finally, the neuroscience and cognitive sciences communities have begun to argue that the brain is, at least in part, an optimization machine marblestone_2016, hassabis_2017. As such, optimization algorithms in DL such as backpropagation may have analogies in biological brains, although, from a practical point of view, the design of an AI system does not require any adherence to biological plausibility. It is not yet well known which processes in the brain emerge from the optimization of cost functions, which emerge from other forms of self-organization, which are pre-structured through genetics and which are based on the interplay of all these mechanisms. Yet, progress in understanding the architecture and behaviour of artificial neural networks could play a vital role for understanding biological brains. For example, recent work on improving the performance of deep CNNs has yielded new insights into the nature of neural representations in high-level visual areas. Further, some properties of the LSTM architecture offered key ideas that motivated the development of working memory models hassabis_2017. Hence, by focusing on the computational and algorithmic levels when building intelligent machines in silico, researchers can obtain transferable insights into the general mechanisms of brain function, in turn opening new prospects for neuroscience. As hassabis_2017 put it: “[D]istilling intelligence into an algorithmic construct and comparing it to the human brain might yield insights into some of the deepest and the most enduring mysteries of the mind, such as the nature of creativity, dreams, and perhaps one day, even consciousness”[p.255].\footnote{As discussed in Section (ref), the architectural and algorithmic constraints underlying deep learning were originally inspired by neuroscience. The role played by this discipline has gradually diminished and many of the major developments in artificial intelligence have been driven by insights into the mathematics of efficient optimization. Notwithstanding this, neuroscience may still inform machine learning in many respects. lake_2017, for instance, have recently suggested some directions for neuroscience-inspired AI research. These studies emphasize the ways through which neural networks could be modified and improved, especially by trying to incorporate more powerful, human-like cognitive abilities such as attention, imagination and planning, efficient and transfer learning, and intuitive understanding of the physical world. In short, exchange of ideas between AI and neuroscience may create a virtuous circle advancing the objectives of both fields.}
The idea of a scientific contribution commonly includes two aspects: scientific novelty and impact. Different terms for essentially the same idea have been employed in earlier research on science. Back then, debates were focused on originality, discovery or breakthrough and their contribution to scientific progress price_1986, merton_1957, bourdieu_1975. kuhn_1962 coined the term `novelty' to describe a more radical contribution that does not advance incrementally the `normal science' in place, but instead breaks the current paradigm. In the more recent literature, the term novelty partly lost this radical connotation, but still carries the idea of a higher degree of originality. The concept of `re-combinatorial novelty' adds to this the idea that new knowledge comes from the recombination of previously generated bits of knowledge weitzman_1998, fleming_2001, uzzi_2013, wang_2017.
The main argument is that we realize only a small part of the potential of useful recombinations in the knowledge space because humans face inherently cognitive limits. In other words, scientific development depends not only on the nature of the object of investigation, but also on the nature of the investigator herself. Deep learning may change the way science develops because it affects research behaviour and helps to connect ideas from seemingly different scientific domains agrawal_2018, cockburn_2018, furman_2020. However, even if DL has this alleged re-combinatorial potential, one may doubt whether it is actually used in that way. The way in which DL is employed in science as a research tool depends not only on the properties of the technology, but also on the scientific complex in which it is introduced callon_1994. This implies that the same technology may be used either to further deepen established research trajectories or to explore new avenues, leading to lower and higher re-combinatorial novelty respectively. The prevailing effect most likely depends on the stage of a scientific domain's life-cycle and the degree of integration of the tool into existing scientific practices.\footnote{A case study on scientific creativity at a large research infrastructure (European Synchrotron) confirms this conjecture \ ferrando_2019. The synchrotron offers favourable conditions for creative research, but this potential depends heavily on the discipline, its degree of maturity and the structure of user communities.} The extent to which DL is associated with `re-combinatorial novelty' in health sciences is therefore an empirical question that we address in the analysis.
The second aspect of our study concerns the scientific impact of deep learning research. Impact is related to but different from novelty. If research provides novelty, that novelty must be taken up by the scientific community in order to unfold impact. On the other hand, research may well have impact on subsequent research for reasons other than (re-combinatorial) novelty, especially by providing new insights within established knowledge structures.
Yet, impact is in general not independent of novelty. The reasons can be found in the process of knowledge production itself as well as in the social dimension. Throughout the knowledge creation process, a higher level of novelty is likely to increase the risk of delays and failures azoulay_2011. On the one hand, the complexity and ruggedness of knowledge landscapes makes navigation difficult. On the other hand, more novelty sometimes necessitates more complex and risky collaborative social structures fleming_2007, foster_2015. Both arguments suggest that research carrying high novelty may be subject to considerable variations in `quality' fleming_2001, wang_2017. This alone would translate into an increased variance of the impact. Social arguments further reinforce this variation. Although originality is praised as fundamental for scientific advancement, high novelty may encounter high resistance in the society because it can cause structural changes in roles and norms merton_1957, bourdieu_1975. These structural changes are particularly relevant in the context of AI where the intersection of ethical and legal considerations shapes the future of both individuals and society as a whole.\footnote{Taking a broader perspective, we note that new digital technologies have a profound effect on almost every aspect of people's lives. Think of algorithmic decisions which can be used for recruitment, credit authorization, policy decision-making, and many other purposes. These decisions are often seen as opaque, unregulated, and amplifying discrimination oneil_2016.}
Most of the empirical evidence on the nexus between re-combinatorial novelty and impact comes from bibliometric studies on scientific publications and patents. Re-combinatorial novelty is commonly measured by the combination of different elements figuring in the document. Citations, keywords, and scientific/technological classes are the most common elements to proxy ideas, concepts, and domains, respectively. Incremental science advances mostly within domains through the addition of concepts and their relations. Compared to novel keyword combinations, novel re-combinations of classes are therefore closer to novelty in the Kuhnian sense.
Most studies build on citations to measure novelty, citations being a way of systematically tracking individual ideas through scientific documents. small_1973 suggests that the co-occurrence of document pairs in the list of references is indeed a good measure of ideas' association. Citations are therefore useful to trace the flow and connection of ideas within and across domains (or scientific categories).
uzzi_2013 lift this idea at the journal level. They contend that different academic journals tend to reflect different scientific domains. In light of this, the co-occurrence of journal references signals domain combinations, and hence can be used to proxy re-combinatorial novelty. More precisely, uzzi_2013 count how often each journal pair is referenced across papers within a given year. Frequently observed journal pairs indicate conventional domain combinations, while rarely observed journal pairs indicate novel (or `atypical') domain combinations. They find that papers incorporating many conventional and some atypical combinations are particularly likely to be among the top 5% cited documents (high impact). In addition, the right balance between high conventionality and some degree of atypical combinations is more likely to be observed in team work than in individual work. Finally, impact propensity is higher for teams than individuals, given any combination pattern. One interpretation of these findings is that a good research strategy is to balance conventionality and novelty, and that teams may be instrumental in implementing that strategy. lee_2015, based on a very similar methodology complemented by a survey, investigates more closely the role of scientific teams in the novelty-impact nexus. Their findings suggest that medium-sized teams, covering a variety of fields, are particularly successful in producing novel combinations.
wang_2017 shift from measuring `atypical' combinations to `novel' combinations. Atypical combinations are journal combinations that are rarely observed in the same year. Novel combinations are journal combinations that have never been observed in prior years. Therefore, the difference lies in the degree (rare vs. never) and in the reference set (same year vs. previous years). Also note that first-time-ever combinations of referenced journals may have different degrees of difficulty; the latter takes into account the knowledge distance between the newly combined journals based on their co-cited journal profiles (i.e., their `common friends', using authors' terminology). The findings confirm that novelty in research is characterized by a `high risk/high gain' profile, delayed recognition, and publication biases.
For the main analysis, we operationalize the concept of `re-combinatorial novelty' as the first appearance of a knowledge combination, closely following wang_2017. Novelty \`{a} la Wang et al. complies more with the idea of DL as a general method of invention (i.e., a method for creating something new and not only unusual), as discussed in the Section (ref). In Section (ref), we also implement the method \`{a} la Uzzi et al., to test the robustness of our results and further examine the degree of conventionality of DL research.
We measure scientific knowledge creation on scientific papers published in peer-reviewed journals and conference proceedings in health sciences.\footnote{Henceforth, the term `journal' refers interchangeably to both peer-reviewed scientific journals and conference proceedings. We restrict the focus on journals that are not cross-classified as `Computer Science' journals. The sample of DL publications used in this Section is therefore a subset of the sample used in Section (ref). This subsample is further complemented with additional non-DL publications, as discussed hereafter.} The principle idea is to compare publications that apply DL with those that do not apply DL, while controlling for a set of confounding factors. Comparison is made in terms of re-combinatorial novelty and subsequent use of the paper measured by the number of citations received.
\paragraph{Sample.} Sample selection is at the journal level. We aim to include all the articles for the whole observation period published in those health journals where research involving deep learning has been the most prominent. This provides us with a relatively coherent knowledge base against which we may judge on novelty and impact. Some sampling is necessary because we do not have the entire universe of health science papers at our disposal. Yet, the focus on journals with the bulk of DL research activity is advantageous in at least two ways. First, we keep a relatively large number of DL articles in the sample for a given number of journal downloads. Second, our final sample contains DL and non-DL publications that are more likely to be comparable: on the one hand, DL is currently undergoing a diffusion process, and therefore its current level of adoption is unevenly distributed across research (sub)domains; on the other hand, re-combinatorial novelty and subsequent use vary depending on these domains wang_2017. It seems plausible that `early DL adopters' come from domains that are generally characterized by high re-combinatorial novelty. We somehow limit this potential bias by selecting those journals in which we indeed observe deep learning research.\footnote{To see the argument more clearly, imagine we would sample from all domains, while DL research would only appear in specific domains that are particularly apt to novel re-combinations within their `normal science' mode. This would lead to the conclusion that DL is likely to provide new combinations. But that would be mainly due to a `domain'--effect rather than a `DL'--effect. Sampling on domains where DL is present limits the influence of the `domain'--effect on the results, and thereby helps to capture the `DL'--effect we care about.}
In total, we count 26,461 DL health papers in about 5,000 health journals and proceedings. Roughly 45% (11,520) of these documents are published in the top 100 most frequent health journals in the sample. The list of journals is reported in the Appendix (ref). Following our sampling strategy, we downloaded the entirety of these journals for the period 1990--2018. Combining DL and non-DL publications, our final sample contains 1,081,223 articles over the period 1990--2018. A general characteristic of the sample is that the number of observations increases over time. We start in 1990 with 14,317 sampled papers and gradually increase to 62,342 sampled papers in 2018. Although we sample all the journals over the whole observation period, our sample includes less journals at the beginning and more at the end of the period. This is caused by the positive trend in DL adoption, as documented in Section (ref). Since most of DL papers are published recently, we are likely to sample `younger' journals. This aspect will be taken into account in the analysis.
\paragraph{Variables.} All variables for the analysis are measured on the sampled papers. Our main explanatory variable is a binary indicator of DL content in the paper (DL): 1 if the paper is classified as DL, 0 otherwise. Our main dependent variables are (various measures of) re-combinatorial novelty and scientific impact based on citation counts. In the following we describe the construction of our main response variables and other covariates.
Re-combinatorial novelty is measured on referenced journals. The basic idea consists in examining for each paper whether it makes first-time-ever combinations of referenced journals -- i.e., its list of references contains journal pairs that have never appeared jointly in any prior list of references. In order to exclude journal pairs that simply formed once by happenstance, we further impose the condition that journal pairs be observed again within the next three years. A paper with at least one journal pair in the reference list that is both novel and re-used, is judged as providing some novelty. This way we construct a binary indicator of novelty, henceforth defined as Novelty Dummy.
One further consideration is that a novel journal pair may span more or less distant domains. This subtlety is captured through the co-citation profiles of the two journals forming a novel pair. The idea is that if both journals are often (rarely) cited with the same third journal(s), they are likely to span less (more) distant domains. This can be used to construct a weighted (continuous) measure of novelty, in the following Novelty. Calculations of the binary and weighted novelty measures follow wang_2017, therefore we provide only a concise formal description of the procedure. Other mathematical details and parameter settings are reported in the Appendix (ref).
Each paper includes a set of references to journals, say $\{J_1, \ldots, J_i, J_j, \ldots, J_J \}$. Novelty Dummy assumes a value equal to 1 if at least one journal pair $(J_i, J_j)$ has never been observed before and is re-used five times within the next three years, 0 otherwise.\footnote{Five times is motivated by the fact that by adopting the threshold used by wang_2017 (3 times), the amount of novel documents was too high, being about 50%. Using a more restrictive threshold, we lowered the share of novel papers to roughly 30%. The patterns discussed in the following are not sensitive to changes in the threshold.}
Although the pair $(J_i, J_j)$ is novel, the two journals may be related through co-occurrence with other journals in prior reference lists. This in turn leads to a weighted version of the novelty measure. We construct a journal co-occurrence matrix $C$ of dimension $J \times J$. Its $i$th column, $C_{i}$, provides counts of how often journal $J_i$ appeared together with any other journal in a journal reference set from the prior three years. The cosine similarity between $C_{i}$ and $C_{j}$, say $\textit{COS}_{i,j}$, measures the `closeness' of journals $J_i$ and $J_j$, and it ranges between zero and one. One if both journals share exactly the same profile in their co-cited journals; zero if they do not have any co-cited journal in common. Therefore, $(1-\textit{COS}_{i,j})$ measures the co-citation distance of the journal pair. There may be more than one novel journal pairs in a paper's reference list. The weighted novelty measure of the paper is obtained by summing $(1-\textit{COS}_{i,j})$ over all novel journal pairs $(J_i, J_j)$ in that paper. In a final step, we take the log transform (we apply $log(x+1)$). In sum, Novelty score increases with the number of novel journal pairs in its references and their co-citation distances.
We differentiate from wang_2017 in two important aspects. Firstly, we judge novelty and co-citation distance only on journal pairs that are observed in the reference lists of our sampled papers. Thus, we do not measure novelty per se but with respect to a knowledge base covered by the sampled (health) journals. The interpretation is that this adds an addressee -- i.e., novel for whom? The addressee here is the health sciences community.
Secondly, we calculate different measures of novelty by considering different sets of journals in the references. In this way, we try to capture the source of novelty: where does the novelty come from? ICT, health, or other domains? Let us try to clarify this argument. Although all the articles in the sample are published in health science outlets, they can reference journals in various domains. For instance, a health science paper that involves deep learning as research tool is likely to cite computer science journals where DL methods were published.\footnote{A visual inspection of a random sample of articles in our sample confirms that this is the case.} This translates into a re-combinatorial novelty `simply' due to the adoption of the method, but does not necessarily reflect the re-combinatorial potential of DL to connect and re-combine knowledge pieces in complex knowledge landscapes. Recall that the general objective of our analysis is to investigate whether DL is mainly used to cope with knowledge explosion within a given domain or to facilitate cross-fertilization of knowledge across domains. In other words, we aim to measure whether the adoption of DL fosters novel re-combinations within health sciences and/or between health sciences and disciplines other than computer sciences. Therefore we calculate novelty not only on any journal pairs, indicated by `All Sciences', but also limit to journal pairs where: (i) no referenced journal is classified as a computer science journal, indicated by `No CS'; (ii) both referenced journals are uniquely classified as health sciences, indicated by `Only HS'. By way of example, the combination of `Biology & Biochemistry' and `Computer Science' journals would be regarded as an `All Sciences' combination; `Engineering' and `Molecular Biology & Genetics' as a `No CS' combination; and `Neuroscience & Behavior' and `Psychiatry/Psychology' would be considered an intra-domain `Only HS' combination.
Combining the above three re-combinatorial possibilities with the option to calculate the novelty either as a binary indicator or continuous score, we obtain six different novelty measures, namely: Novelty Dummy (All Sciences), Novelty Dummy (No CS), Novelty Dummy (Only HS), Novelty (All Sciences), Novelty (No CS), and Novelty (Only HS).
Impact is measured by the number of citations (\# Citations) received by a paper from its year of publication up to 2019, the time of data extraction. Furthermore, we code dummy indicators for `big hits' contributions -- i.e., highly cited papers. Whether a paper is among the top 5% or 10% cited papers (Top 5% Cited and Top 10% Cited) is calculated with reference to other papers published in the same year and falling in the same WoS subject category.\footnote{A paper may fall into several WoS subject categories (two-character field WC). We consider all papers having at least one subject category in common with the focal paper to obtain the citation distribution and identify those papers that receive an exceptionally large number of citations.}
We consider a set of control variables to capture various characteristics of a focal paper. We control for the number of references made by a paper (\# References) which might mechanically increase the likelihood of having new combinations and affect the number of received citations. In prior research, the number of authors has proven to be positively associated with both novelty and impact, hence we control for that (\# Authors). The adoption of deep learning can indeed have an ambiguous effect on team size. Size may increase as new members are needed to manage the technology (at least in the early stages), but the technology may also automatize some tasks, thereby generating a replacement effect in the scientific workforce.\footnote{Although beyond the scope of our study, it interesting to note that the distribution of the number of authors of non-DL papers dominates the distribution of papers using DL. At the same time, we see that single-author contributions seem to have a lower probability of appearing within the DL category. We leave open the question as to whether DL acts as labour-saving or labour-augmenting technology in the scientific realm.} International collaborations may also be a source of novelty and impact, and may be instrumental to the adoption of DL. We proxy international collaboration by a dummy (International Collab.) taking a value of 1 if there are at least two different countries in the authors' affiliations, 0 otherwise. For the same reason, we construct a dummy for private collaborations (Private Collab.) taking a value of 1 if the paper has at least one non-university affiliation in the list. We consider the journal impact factor (JIF) since high impact journals may be biased against novelty, on the one hand, but increase visibility and hence citations, on the other hand. We additionally control for the journal age (Journal Age). Finally, we include a dummy indicating whether the paper provides a review or survey of existing literature (\textit{Survey}). A survey may in fact cover separate streams of research without really connecting them.\footnote{ \textit{Private Collab.} takes value of 1 if we detect in the authors' affiliation at least one of the acronyms present in the Wikipedia page: `List of legal entity types by country'. We use the SCImago Journal Rank to get the impact factor (\textit{JIF}) for each journal in each year. \textit{Journal Age} is calculated as the time elapsed from the date of the journal's creation and the publication year. \textit{Survey} takes value of 1 if we detect in the title of the paper the terms `Survey', `Overview' or `Review'.}
\paragraph{Descriptive statistics.} DL health papers represent about 1.3% of our sample. Table (ref) provides basic statistics to compare papers involving DL (first column) with papers not involving DL (second column).
First consider novelty. The first row in Table (ref) shows the Novelty Dummy (All Sciences) based on journal references to all sciences. We see that DL documents exhibit novelty more often than others (36% vs. 30%). The second row, Novelty Dummy (No CS), excludes journal references to computer sciences. That reduces the novelty gap. Novelty of DL papers decreases by four percentage points, while the novelty of non-DL papers remains unchanged. The third row, Novelty Dummy (Only HS), restricts further to journal references to health sciences only. This results in a significant drop of novelty for both DL and non-DL papers. However, the reduction is larger for DL papers than for non-DL papers. This flips the pattern: only 21% of DL papers provide novel recombinations within health sciences, which is less than 23% of non-DL papers.
Now consider the continuous novelty measures (rows four to six). In general, novelty distributions for DL and non-DL publications are highly skewed and heavily overlapping in such a way that no distribution dominates the other. The mean for Novelty (All Sciences) of DL papers is somewhat higher than the mean of non-DL papers (0.81 vs. 0.74). However, excluding computer science journals and restricting further to health science journals, again leads to a drop in novelty, which is even stronger on DL papers. The standard deviations of novelty for DL papers are relatively small, from 60 to 70% of the others. Taken together, these statistics point to two main patterns: DL papers offer as much or more novelty than non-DL papers through new recombinations including computer science journals; once journal references to computer sciences are excluded, DL papers bring on average less novelty than others.
The second aspect of interest is the impact of publications that we measure by the number of citations. There are clear differences in the citation profiles of DL and non-DL papers. We do not see differences in terms of the median number of citations, but we do see a slightly larger mean for DL contributions (38 vs. 35). Interestingly, DL papers are characterized by a higher dispersion in citation performance (114 vs. 82). The most cited papers (top 5% and 10%) are more likely to be contributions involving DL.
As far as controls are concerned, we see that DL papers have on average (i) more references, (ii) slightly smaller team size, (iii) more international collaborations, (iv) slightly less private collaborations, (v) and are published in younger and lower ranking journals.
\paragraph{Estimation methods.} We model three different types of outcomes: (i) binary indicators of novelty and impact, (ii) positive continuous measures of novelty, and (iii) positive discrete measures of impact (number of received citations). Each type of outcome requires a specific econometric setting.
All binary indicators are modeled with a Probit. Our continuous novelty measure is censored at zero, hence we use a Tobit model. The Tobit model introduces a latent (unobserved) outcome variable. Unlike the observed outcome, the latent outcome is unrestricted and may thus be modeled with a simple linear model with normally distributed error term. When the latent variable exceeds a certain threshold, in our case zero, the realization of the latent variable is assumed to equal the observed outcome. If the latent variable realizes below zero, the observation is assumed to be zero. The distributional assumption on the error term makes it possible to parameterize the model through Maximum Likelihood.
Citations are count data for which the Poisson and Negative Binomial models are natural candidates. In the Poisson model, expectation and variance necessarily coincide for each draw. The sum of Poisson distributed realizations is itself Poisson distributed, and this implies equality of mean and variance in the sample. In the Negative Binomial model the two moments may differ. Over-dispersion and conditional mean of the outcome variable much lower than its variance are the most common arguments to favor the Negative Binomial on Poisson model. Both empirical arguments hold in our case. Furthermore, prior empirical findings suggest that novelty may affect not only the expectation but also the variance of received citations fleming_2001, wang_2017. Therefore, we use Negative Binomial to model mean and dispersion separately, each with a linear predictor incorporating our main left-hand side variables and controls.
In all estimations, we include the control variables discussed above and a set of dummies to control for scientific fields and cohort effects. We proxy scientific fields through WoS categories (field WC). A paper may fall into several categories, hence we code dummy variables taking value of 1 for each category. Throughout, robust standard errors clustered at the journal-level are obtained via bootstrapping all journals.\footnote{Bootstrapping is essential in this context because there are group effects induced by the fact that some journals are more inclined to accept articles with a certain degree of novelty. Bootstrapping allows us to calculate the variance of estimates taking into account clustering at the journal level. Note that non-bootstrapped robust standard errors are much smaller, and all the DL effects discussed hereafter would be much stronger in terms of statistical significance. To get an idea of the difference, bootstrapped standard errors of DL coefficient estimates are always at least twice as large as the non-bootstrapped ones.}
We begin with the results of how DL correlates with novelty, as shown in Table (ref). Columns 1--3 refer to Tobit regressions of our continuous measures of novelty, Novelty. Columns 4--6 report Probit estimates of the binary novelty indicators, Novelty Dummy. Recall that both measures are calculated on three different sets of journal references: all journal references (all sciences), journal references excluding computer sciences (no CS), and journal references only to health science journals (only HS).\footnote{The tables presented in this Section provide the main results, i.e. the results on the six outcome measures that always include the entire set of control variables and fixed effects. We have verified that the DL coefficient estimates are robust to various configurations of controls.}
First, consider the Tobit regressions and focus on the first row that reports DL coefficient estimates for different novelty measures. When considering re-combinatorial novelty across all sciences (Column 1), the estimated DL coefficient is positive but insignificant. The exclusion of computer science references (Column 2) turns the coefficient into negative but keeps it insignificant. Restricting further to health sciences only (Column 3) magnifies the negative coefficient and turns it significantly below a 1% significance level. We observe exactly the same pattern when considering the results of the Probit regression of the novelty dummy. How much does DL adoption change our expectations on associated re-combinatorial novelty within health sciences? We see that adopting DL decreases by 18.6% the degree of novelty. In addition, the marginal effects for Probit (model of Column 6) tell us that, for the median observation, DL decreases by 0.031 the probability for an article to be novel (0.037 for the average observation).
In sum, controlling for confounding factors, deep learning is not significantly correlated with novel recombinations across the entire knowledge landscape, nor with novel recombinations involving anything but computer sciences. Deep learning is significantly and negatively correlated with novel recombinations within the health sciences though. These findings suggest that deep learning tends to be adopted within a `balancing strategy' in which the risk of DL adoption is counterbalanced by keeping the health science research landscape stable. Another way to interpret our result is that deep learning works mainly as a research tool for deepening existing knowledge structures rather than exploring/establishing new connections between more distant domains. This evidence is consistent with the idea of extending science while maintaining the advantages of conventional domain-level thinking uzzi_2013, wagner_2019. Both explanations are likely to carry some truth. The first emphasizes agency in complex environments. The second points to the properties of DL as `scientific instrument'.
Estimates of control variables echo previous research. Larger teams are associated with more novelty fleming_2007, lee_2015. International collaborations are negatively associated with novelty wagner_2019. Mechanically, the chances of providing a new combination of journal references increase with the number of references wang_2017. Literature reviews also tend to draw from a wider range of sources leading to novel combinations of references. We find a negative effect of private collaboration. On the other hand, the journal age and its impact factor seem to play no role for novelty.
We now turn to how DL correlates with impact, as shown in Table (ref). Column 1 presents the results of the Negative Binomial regression of citation counts. Here, the mean and dispersion parameters may vary with various right-hand side factors.\footnote{We excluded dummy variables other than $DL$ to model the dispersion of citations because these variables caused problems of convergence of maximum likelihood estimator. In modeling the dispersion, we also tried simpler specifications by progressively incorporating a few variables at a time. The results are in line with those presented in this document.} We find that DL adoption positively and significantly affects the number of citations received, both in terms of expectation and variance. The same applies to novelty. These two effects are akin to the `high-risk/high-gain' profile that characterizes the adoption of emerging technologies and novel research wang_2017.\footnote{In the main analysis at hand we enter the continuous novelty measure on all sciences, but we have verified that the results are robust to the various novelty measures discussed in this paper. We also find, not shown here, that novelty and DL act rather independently of each other. Estimates hardly change when one or the other variable is removed from the model, implying that DL and novelty add to each other when it comes to impact. This is perhaps not too surprising if we consider that the adoption of DL is often a specific type of novelty per se.} Compared with non-DL papers, ceteris paribus, DL papers receive on average 10.32% more citations. The citation count expectation increases in median by 6.01 citations if DL is used. The dispersion of the citation distribution is 19.57% higher for DL papers than non-DL papers. \\
As for the controls, the number of authors is positively related to impact lee_2015; a new insight here is that it also reduces impact variation. International collaborations also increase citation expectations glanzel_2001. Publishing in a high impact factor journal further increases the average number of citations, being this through increased visibility, signaling or selection effects. Moreover, surveys and other papers with many references tend to attract more citations. Finally, we find a negative effect, albeit not particularly significant, between private collaborations and scientific impact.
The Probit regressions used to model the probability that a paper falls in the right-tail (top 5% or 10%) of the year--field citation distribution confirm the above results. The marginal effects suggest that, in median, deep learning papers have a 0.019 higher probability to fall in the 10% of the most cited documents (0.027 in mean), and a 0.009 higher probability to fall in the top 5% (0.014 in mean).
In sum, the econometric analysis shows that publications involving deep learning methods exhibit citation patterns consistent with the `high-risk/high-gain' profile associated with breakthrough research. DL seems to play a positive effect on citation count, on average; however, DL papers have a higher variance in citation performance than do other health counterparts, confirming their risky profile. As such, research involving deep learning has a high potential for major impact, on the one hand, but also carries a higher uncertainty of having impact, on the other. As discussed in Section (ref), uncertainty (which could in turn affect success and scientific quality) may find several explanations, including the challenge of integrating the scientific instrument into existing scientific practices, the capacity to extract the true potential from the instrument and not adopt it simply because `everybody does' and, to a lesser extent, possible social resistance especially in sensitive domains as some areas of health sciences may be. We do not find any effect of deep learning on the degree re-combinatorial novelty on the entire knowledge landscape, and a systematic negative effect on the novelty intra-field. These results suggest that deep learning acts mainly as a research tool to support and deepen already formalized and well-defined research in the health sciences community.
Our results are robust across a wide range of additional tests. Here below, we discuss the main exercises that were carried out. Tables and further material can be found in the Appendix (ref).
First of all, we excluded all articles that fall into the WoS `Neurosciences' category. As pointed out throughout the document, this domain can be potentially problematic in that some terms (neural network in primis) may not necessarily refer to artificial intelligence but rather to human intelligence and biological brain. The sample drops by about 30% and the number of DL articles almost halves. However, the main results are consistent when replicating the estimates on the reduced sample.
Second, we excluded all articles that contain exclusively the terms `neural_network' and `neural_networks' in their title, keywords or abstract. Keep in mind that an article may still contain a term such as `artificial_neural_network' or `convolutional_neural_network' which should now refer unambiguously to artificial intelligence. In this case, neuroscience papers may be part of the sample but we make sure that they explicitly deals with deep learning. This restriction is severe insofar as the number of DL articles decreases by more than 70%. Yet our results are robust to this constraint. The only pattern we lose is the significant effect of DL on the dispersion of citations received. A plausible explanation is due to the smaller sample size and the consequent drop in the variance of the number of citations, as suggested by the descriptive statistics reported in the Appendix (ref).
The third exercise consists of a different econometric approach. Instead of regression analysis we compared each DL papers with `twin non-DL papers'. More precisely, the empirical strategy considers the adoption of DL as a `treatment', hence we employ exact matching and 1:1 nearest neighbor matching on propensity scores (PSM) to select an appropriate control group of untreated papers. Exact matching is performed considering Web of Science categories, publication year, and journal (i.e., we compare a DL article in terms of novelty and impact with a article belonging to the same domain(s), published in the same year and on the same journal). As for PSM, we obtain the propensity scores associated with the binary treatment via the estimation of the Probit model (or selection equation) containing the original set of variables. Although we refrain from giving any causal interpretation, the average treatment effects (ATT) for the selected variables bring further support to our results.
fontana_2020 show that different novelty indicators are often inconsistent with each other as they return different sets of novel contributions. Thus, in the forth exercise, we implemented the indicator proposed in uzzi_2013 to define the `atypical' (novelty/conventionality) quadrant. For the period 2001--2015, we computed pairwise combinations of cited references of each article. We then compared the observed frequency with the frequency from a randomized citation network that preserves the realistic aspects of the data and its network structure (i.e., same number of references to prior work, the same number of citations from subsequent papers, and the same distribution of these citations over time). We obtained a distribution of $z$-scores for each document and calculated two statistics: the median $z$-score and the 10th percentile $z$-score of that article. High conventionality is coded as 1 if the median $z$-score value is above or equal to the overall median, 0 otherwise. High novelty is coded as 1 if the 10th percentile $z$-score value is below the median value of the overall 10th percentiles. We combined these two binary variables into a nominal variable with four categories, namely: high-conventionality/high-novelty (HC--HN); high-conventionality/low-novelty (HC--LN); low-conventionality/high-novelty (LC--HN); and low-conventionality/low-novelty (LC--LN).\footnote{Although we replicated the uzzi_2013 method, we differ in two respects. Firstly, we simulated 50 random citation networks (instead of 10) to calculate the $z$-scores. This substantially increases the computational burden, but reduces the likelihood that an article makes combinations that do not exist in the simulated networks; to notice that a combination that does not exist in the simulation has no standard deviation and therefore its score $z$-score cannot be defined. Secondly, we consider the high novelty threshold as dependent on the citation network instead of using a default value (0 in the original paper). This allows us to set the share of high-novel papers to 50%. Considering a threshold equal to 0, we would get roughly 70%. The discrepancy between our shares and those in Uzzi et al. is simply due to the size of the sample on which the indicators are built (1 million vs. 18 million articles).}
The four categories are employed in a Multinomial Logistic regressions. Category LC--HN acts as the reference category for all models. This category represents the group of papers characterized by the most atypical combination profiles. Within the health sciences knowledge landscape, we find that deep learning articles are more likely to draw on highly conventional mixtures of knowledge. Ceteris paribus, the estimates suggest that even when DL injects some highly (field-specific) unusual combinations, it does so primarily in an exceptionally conventional knowledge space.
We developed this paper with the aim of studying the diffusion and impact of the DL in the scientific system. We proceeded in three steps. In the first part, we developed a list of DL search terms through Natural Language Processing of scientific abstracts from arXiv.org. We used a word embedding algorithm that projects words into a mathematical space that reflects the semantic structure of the abstracts. Words cluster in that mathematical space by `topic', which allows us to define the DL perimeter. The second part of this paper documents the diffusion of DL research in science. Our sample includes Web of Science documents across all sciences identified through our DL search terms. Starting from a low level in the early 2000s, DL research activity is growing exponentially in almost all sciences and worldwide. Asia and Eastern Europe are focusing their research mainly on DL methodologies, while Western Europe and North America seem to have a `relative advantage' in application areas such as life sciences and biomedicine. Scientific co-classification of articles suggests that the diffusion of DL into application domains began with an interdisciplinary effort involving the computer sciences, breaking its way into `pure' field-specific research within the various application domains. The third part deals with the consequences of DL adoption on scientific development, making the case for the health sciences. We find that the adoption of DL is negatively correlated with re-combinatorial novelty. On the other hand, it is positively correlated with the expectation and dispersion of citations received, increasing the likelihood for a contribution to become an influential `big hit'.
Conceptually we considered scientific development as a re-combinatorial process in which existing pieces of knowledge are recombined to create new knowledge. This continues perpetually in a dynamic knowledge landscape. A traditional image of science is one in which the knowledge landscape is made up of islands, (sub)-disciplines or scientific fields, where most recombination takes place. The islands reflect the structure of nature but also the necessity for a scientific mind to organize the complexity of the world. The scientist is the sailor. The aim of the sailor is to navigate through the islands, figure out their structure, and explore the surrounding landscape. So she can stay in the `comfort zone' by further deepening her knowledge of one (or neighboring) island(s). Or she can sail to more distant islands and connect new areas of the landscape. Both actions enrich the knowledge space, one digging into a well-formalized knowledge structures, the other reshaping and rearranging the landscape. Our findings suggest that, at least as it is used today, deep learning -- the boat or the compass to keep the analogy -- seems to be more in line with the first option. Yet the possibility of discovering new valuable things about the already known islands is not obvious. This may result from the fact that the sailor needs to know how to use the new equipment but also from the fact that the crew -- the society and/or the rest of the scientific community -- have confidence and trust in it.
Our findings lead us to take a moderate position in the recent debate on how DL affects the development of knowledge. A passing fad in science? We don't think so. DL does not (yet?) work as an autopilot to navigate a sea of knowledge, but stands as an extremely powerful and versatile research tool that impacts knowledge creation in measurable ways.
With the results of our study in mind, let us conclude with some considerations for policy and management. The evidence that deep learning owns the characteristics of a general purpose invention in the method of invention implies that its socio-economic impact in terms of innovation, growth, productivity gains and societal well-being will be pervasive. Impact pathways when it comes to AI are very diverse and often indirect. The new research paradigm creates huge opportunities but also comes with some challenges.
First, intelligent machines as input in the research production process question the organization and management of science. Deep learning may trigger short-term substitution towards capital and away from highly skilled labour in the knowledge production process. Whether or not such a substitution effect is occurring is questionable and clearly requires further empirical investigation. At the same time, it is undeniable that the arrival of deep learning as a research tool jeopardizes a wide range of research tasks, either by reducing the cost of performing these tasks or by surpassing the performance of human scientists in performing them. Some tasks within the occupation may be suitable for automation while others may not, and the overall effects on employment in science are very complex. Research-oriented organizations need to better understand the collection of tasks that make up the job of their scientists, how to coordinate these tasks, the strengths and weaknesses of humans (H) and machines (M), to finally unleash the benefits from H + M cooperation.
Although we cannot rule out the possibility that deep learning has the potential to ease researchers in predicting which combinations of distant knowledge in the landscape will produce useful new discoveries, our findings suggest that the technology is not used in that way. Deep learning seems to be adopted in a scientific field by maintaining the existing knowledge structure relatively stable. Therefore, the full potential of deep learning (and its future development) could be achieved by further spanning the boundaries between scientific areas. Bringing together expertise and knowledge from various domains could help to see blind spots and opportunities in the knowledge landscape. The concepts of `knowledge communities' and `communities of practice' seem particularly apt in this context. Although communities often self-organize and self-sustain, they may also benefit from policy endorsement. It seems crucial to us to develop institutions and a policy environment that is conductive to enhancing dialogue and cross-fertilization between communities. This can be achieved, for instance, through the reinforcement of horizontal (intra-field) but also vertical (inter-field) knowledge management. Digital platforms and knowledge hubs should be complemented by physical `collaborative spaces' where the tacit knowledge of different communities can be transferred face-to-face, documented and made accessible for later use. Another standard instrument is research funding, which should not target individual areas but mainly research `priorities' (i.e., fighting a given disease) involving different communities framing research questions together.
However, promoting international collaboration between communities may pose some challenges in terms of governance and data ownership. Data is a polymorphous category, so the standards, principles and rules governing the various types of data are not homogeneous across countries. No data, no AI. Bad data, bad AI. This opens the question of how data should be generated/used in compliance with different regulations, and also how the value of the data should be distributed (see the discussion in savona_2019).
The diffusion of deep learning, also as a research tool, can be self-sustaining if and only if there is social acceptance -- i.e., if the crew trusts the captain and the equipment. Several DL applications are innovations that can change almost every aspect of our daily lives. These social innovations can have negative and unintended consequences in terms of security, privacy and social equity oneil_2016, agrawal_book_2018, furman_2019. People will no longer tolerate being excluded from the debate. Here again the scientific and policy community have a key role to play. Both parties may improve the channelling of scientific evidence into the public debate and fight the dangers of fake news, particularly common when it comes to AI, robotics and automation. On the one hand, policy can promote communication by setting the right, often intrinsic, incentives to encourage as many scientists as possible to engage with various segments of the public. However, communicating science to non-scientific audiences might be difficult since it requires a different approach from communicating science to scientific audiences. Scientists need to understand how to detach the layers of scientific complexity of their research in order to deliver some clear messages to the public. These messages should include both potential impacts and ethical issues. `Listening mechanisms' can also be used to bring citizens' knowledge, expectations, and imaginaries about intelligent machines. There are a variety of forms ranging from in-depth interviews, material deliberations, to citizen science. In the context of AI, we believe that citizen science may probably bring the greatest benefits to both the public and the scientific system. The non-professional involvement of volunteers in the scientific process, either in the data collection phase or in other stages of research, offers great opportunities for the public to become familiar with the technology but also great opportunities for researchers to improve their results bonney_2014. To get a sense of impact, the research participation by more than 300,000 volunteers over 1 year could provide nearly 33 million classifications of subcellular localization patterns sullivan_2018. But accountable institutional mechanisms are a precondition for guaranteeing trust between scientists and the public, thus ensuring continuity in their relationship. For instance, the results and the process used for arriving at these results should be open to scrutiny. Policy should promote feedback activities to get back to citizens and explain how their inputs were used for research aims, reconcile conflicting values and objectives, and put in place collective intelligence mechanisms to help citizens develop a systemic understanding of the future implications of technological progress and make better consensus decision-making; very much in line with the notion of `decisions 2.0' bonabeau_2009. Finally, we fully embrace the conception of `boundary organisations' specifically designed to deal with socio-economic transformations in the digital age. These organisations would sit at the intersection of scientific and political spheres and allow scientists and policy-makers to maintain a constant dialogue with each other.
Although the AI revolution is the subject under scrutiny, ironically that revolution offers the tools that can bring the greatest potential for a radical transformation in the interaction between the public, the scientific community and the policy environment. These interactions, if conducted fruitfully enough, will give a boost to the human attempt to better understand the largest mystery we know: the origin and function of the world and our place in it. That's science! \\
{ Acknowledgement. Earlier versions of this paper were presented at the INNOVA MEASURE III Expert Workshop `Brainstorming in Ispra'; 7th European Conference on Corporate R&D and Innovation, Seville (Spain); EMAEE 2019 `Economics, Governance and Management of AI, Robots and Digital Transformations', Brighton (UK); OLKC 2019 `The Human Side of Innovation -- Understanding the Role of Interpersonal Relations in an Increasingly Digitised Workplace'; CAGE-NESTA Workshop on `Data Science for the Economics of Science, Technology and Innovation', London (UK); and the Workshop Series on `The Economics and Management of AI Technologies', Copenhagen (Denmark) and Strasbourg (France). The authors thank seminar participants and in particular Giacomo Damioli, Mirko Draca, Daniel S. Hain, Bj\"{o}rn Jindra, Bertrand Koebel, Roman Jurowetzki, Patrick Llerena, Juan Mateos-Garcia, Simone Vannuccini, Daniel Vertesy. Stefano Bianchini gratefully acknowledges the financial support of the CNRS through the MITI interdisciplinary programs [reference: Artificial intelligence in the science system (ARISE)] and the French National Research Agency [reference: DInnAMICS -ANR-18-CE26-0017-01].}