AI is an emerging topic and will soon be able to perform administrative decisions faster, better, and at a lower cost than humans. Machines can consistently process large amount of unstructured data to identify pattern and make predictions on future events (e.g. Agrawal and Dhar 2014; Baesens et al. 2016). In more complex and creative contexts such as innovation and entrepreneurship, however, the question remains whether machines are superior to humans. Machines fail in two kinds of situations: processing and interpreting “soft” types of information (information that cannot be quantified) (Petersen 2004) and making predictions in “unknowable risk” situations of extreme uncertainty that require intuitive decision-making. In such situations, the machine does not have representative information for a certain outcome and overfits on training data at cost of the live performance of a learner (Attenberg et al. 2015).
One example where both “soft” information signals as well as “unknowable risk” are crucial, is predicting the success of early stage start-ups. In this case, entrepreneurs face the challenge to decide if the start-up at hand is worth working on or not. Entrepreneurial decision makers often make decisions before neither the feasibility of a new product nor the existence of a market is proven (Maxwell et al. 2011). In such contexts, entrepreneurs do simply not have enough information to assess the quality of a start-up and thus predict the probability of its future success (Dutta and Folta 2016). Moreover, such information might simply not exist under conditions of extreme uncertainty and make the outcome thus unknowable. Consequently, predictions are made for ideas that serve markets, which do not yet exist or novel technologies, where feasibility is still unknown but may provide great returns of investment (Alvarez and Barney 2007). Nevertheless, identifying such unicorns, start-ups that are highly innovative, disrupt traditional industries, and offer tremendous return is highly relevant.
In these situations, humans are still the “gold standard” for processing “soft” signals that cannot easily be quantified into models such as creativity, innovativeness etc. (Baer and McKool 2014) and make use of an affective judgment tool to recognize pattern in previous decisions: intuition (Huang and Pearce 2015). Using one´s gut feeling proved to be a valuable strategy to deal with extreme uncertainty. However, individual human judges are tainted by bounded rationality in making predictions, which emphasizes that instead of optimizing every decision, humans tend to rely on heuristics (i.e. mental shortcuts) and thus rather focus on highly accessible information (Simon 1955; Kahneman 2011). This often leads to biased interpretation (cognitive processes that involve erroneous assumptions) and may finally result in disastrous predictions (Busenitz and Barney 1997). To solve this problem, research in the field of human computation provides a valuable solution: utilizing the “wisdom of crowds” through collective intelligence (e.g. Brynjolfsson et al. 2016; Larrick et al. 2011; van Bruggen et al. 2010). This is a suitable approach to leverage the benefits of humans in prediction tasks, such as providing subjective evaluation of variables that are difficult to measure objectively through machines (e.g. innovativeness) (Colton and Wiggins 2012) or using their prior domain-specific knowledge to make intuitive decision (Blattberg and Hoch 1990). The aggregation of knowledge and resulting predictions than eliminates the statistical errors of individual human decision makers (Larrick et al. 2011). While each of the methods might work well in separation, we argue that combining the complementary capabilities of humans and machines in a Hybrid Intelligence approach allows to make predictions in contexts of extreme uncertainty such as the case of early start-up success through applying formal analysis of “hard” information as well as intuitive decision-making processing also “soft” information
The aim of this research is to develop a method to predict the probability of success of early stage start-ups. Therefore, I follow a DSR approach (Hevner 2007; Gregor and Hevner 2013) to develop a Hybrid Intelligence method that combines the strength of both machine intelligence such as ML techniques to access, process, and structure large amount of information as well as collective intelligence, which uses the intuition and creative potential of individuals while reducing systematic errors through statistical averaging in an ensemble approach (Shmueli and Koppius 2011). I, thus, intend to show that a hybrid approach improves predictions for the success of start-ups under extreme uncertainty compared to machine or human only methods.
Within the scope of this paper, I first developed a taxonomy of signals that are potential predictors for the success of early stage start-ups based on previous work and domain knowledge (Shmueli and Koppius 2011).
I then designed a method that uses these predictors as input for both ML algorithms as well as collective intelligence to individually assess the probability of success and then weights and aggregates the results to a combined prediction outcome. Moreover, I provide an outlook on the next steps of my research project.
This work thus contributes to several important streams of IS and management research. First, I provide a taxonomy of potential predictors that can be generalized for modelling start-up success predictions (e.g. Böhm et al. 2017). Second, this research adds to literature on predictive research in IS and data analytics (e.g. Chen et al. 2012) by introducing a new method for predicting uncertain outcomes under limited information and unknowable risk by combining collective and machine intelligence in a Hybrid Intelligence Method. This approach allows to complement formal analysis of “hard” information and intuitive predictions based on “soft” information. Consequently, my research offers prescriptive knowledge in this vein (Gregor and Jones 2007). Third, I contribute to previous work on collective intelligence (e.g. Malone et al. 2009; Wooley et al. 2010) by proposing novel applications of the crowd. Finally, I provide a practical solution that offers entrepreneurial decision makers a useful way to support their innovation decisions.
One way towards understanding predictions in uncertain situations is to examine the mental processes that underlies the cognitive decision-making process. A theory that is particularly helpful in this context is the dual process theory of decision-making. The underlying assumption of this theory is that people make use of two cognitive modes, one is characterized by intuition (system 1) and one by deliberate analytical predictions (system 2) (Tversky and Kahneman 1983; Kahneman 2011).
Predicting the success of early stage ventures is extremely complex and uncertain because frequently just vague ideas are prevalent, prototypes do not yet exist and thus the proof of concept is still pending. Moreover, such ideas might even not have a market yet, but offer great potential of growth in the future (Alvarez and Barney 2007). Consequently, the decision-making context is highly uncertain as neither possible outcomes nor the probability of such are known. This fact can be explained through two concepts: information asymmetry and unknowable risk (Alvarez and Barney 2007; Huang and Pearce 2015).
Information asymmetry describes situation, in which forecasters have incomplete information to decide. When perfect information is absent, decision makers tend to search for various indicators that signal the likeliness of future outcomes (Morris 1987). In my context, such signals include both “hard” signals that can be easily quantified and categorized (e.g. industry, technology, team size) as well as “soft” signals (e.g. innovativeness, personality of entrepreneur). Humans then try to apply formal analysis to gather signals that support them in making deliberate, rule-based system 2 decisions (Kahneman 2011). On the other hand, unknowable risk defines situations in which a decision maker cannot gather information that signal a potential outcome or make decisions based on formal analysis because the simply not exist. This may be best compared to the error term of a statistical Bayesian model. Unknowable risk covers unexpected events that describe a deviation from status quo (Diebold et al. 2010); (Kaplan and Garrick 1981). In my context, this means for instance identifying a unicorn start-up that gains enormous return that only few would have expected. Formal analytics are not working in these contexts, as representative cases might be missing in previous experience. In such situations, where humans “don’t know what they don’t know”, decision-making is mainly based on intuition (system 1) rather than formal analysis (Tversky and Kahneman 1983; Huang and Pearce 2015). Thus, predicting the success of early stage start-ups is a challenging task and the costs of misclassification are high as they might lead to disastrous decisions or missing valuable chances for return (Attenberg et al. 2015). Previous research in the context of early stage ventures provides strong evidence the best performance in terms of accuracy are provided by combining both types of predictions: analytical (system 2) and intuitive (system 1) (Huang and Pearce 2015).
For analytically predicting future events, computational statistical methods become particularly valuable due to progresses in ML and AI. They can identify, extract and process various forms of data from different sources (Böhm et al. 2017; Carneiro et al. 2017). Statistical modelling allows to make highly accurate and consistent predictions in the context of financing decisions (Yuan et al. 2016b), financial return ,or bankruptcy of firms (Olson et al. 2012) by identifying patterns in the prior distribution of data and thus predict future events. Machine intelligence is, thus, particularly valuable as biases or limited capacity of human decision makers does not taint it. Statistical models are unbiased, free of social or affective contingence, consistently integrate empirical evidence and weigh them optimally and they are not constraint by cognitive resource limitations (Blattberg and Hoch 1990). Consequently, machine intelligence is a suitable approach for making statistical inference based on prior data and they can learn as the data input grows (Jordan and Mitchell 2015). While such machine intelligence approaches are superior in analytically predicting uncertain outcomes by minimizing the problems of information asymmetries and bounded rationality based on prior distributions of “hard” objective variables (e.g. firm age, team size), they are neither able to explain the remaining random error term of such distributions, which I conceptualized through unknowable risk (Aldrich 1999) nor the “soft” and subjective signals of new ventures such as innovativeness, the vision or the fit of the team, or the overall consistency of a new venture (Petersen 2004). Both limitations of machines might lead to costly misclassifications (Attenberg et al. 2015). While progresses in the field of AI provide evidence for the applicability of machines in making subjectivity decisions intuitive predictions remain the advantage of humans and require the completion of machine capabilities.
In this vein, the benefits of human decision makers come into play. Humans are still the “gold standard” for assessing “soft” signals that cannot easily be quantified into models such as creativity and innovativeness (Baer and McKool 2014). Humans are talented at making intuitive predictions by providing subjective judgement of information that is difficult to measure objectively through statistical models (Einhorn 1974). Moreover, human decision makers can have highly organized domain knowledge that enables them to recognize and interpret very rare information. Such information might lead to outcomes that are difficult to predict and would rather represent outliers in a statistical model (Blattberg and Hoch 1990). Consequently, using human intuition proved to be a valuable strategy for anticipating start-up success at an early stage (Huang and Pearce 2015).
However, individual decision makers make errors due to their bounded rationality (Simon 1997; (Kahneman 2003). This assumption considers the capacity of the human mind for solving complex problems as rather constraint. Instead of optimizing every decision, individuals tend to engage in limited information accessing to reduce cognitive effort (Hoenig and Henkel 2015). Consequently, they use cognitive heuristics (i.e. mental shortcuts) and simplifying knowledge structures for reducing information-processing demands. One is for example drawing conclusions from a small amount of information or using easily accessible signals (Tversky and Kahneman 1974). Moreover, humans have several biases (cognitive processes that involve erroneous assumptions) that guide the interpretation of information to make predictions (Busenitz and Barney 1997).
Research on human computation provides a solution for these problems (Quinn and Bederson 2011). Collective intelligence leverages the “wisdom of crowds” to aggregate the evaluations of a large group of humans, thereby, reducing the noise and biases of individual predictions (Atanasov et al. 2016; Cowgill and Zitzewitz 2015). The value of crowds compared to individuals underlies two basic principles: error reduction and knowledge aggregation (Larrick and Feiler 2015; Mellers et al. 2015). Error reduction is because although individual decision makers might be prone to biases and errors, the principle of statistical aggregation minimizes such errors by combining multiple perspectives (Armstrong 2001). Second, knowledge aggregation describes the diversity of knowledge that can be aggregated by combining the experience of multiple decision makers. Such knowledge aggregation enables to capture a fuller understanding of a certain context (Keuschnigg and Ganser 2016). Thus, collective intelligence can assess the probability of uncertain outcomes by accessing more diverse signals and reduce the threat of biased interpretation.
Using collective intelligence enables to complement a machine model by assessing unknowable risk, which cannot be explained through prior distribution but rather from the combined intuition of humans. ML provides advantages in making analytical predictions based on “hard” information while collective intelligence offers benefits in making intuitive predictions taking also “soft” information into account. Previous work emphasizes the complementary nature of humans and statistical models in making predictions about future events in various settings such as sports, politics, economy, or medicine (Nagar and Malone 2011; Meehl 1954; Dawes et al. 1989; Einhorn 1972; Ægisdóttir et al. 2006). I thus argue that a Hybrid Intelligence Method that combines the complementary capabilities of analytical and intuitive predictions is most accurate for predicting the success of early stage start-ups.
Automated Access to High Volume of Hard Information
Consistent Processing of Hard Information
Identifying Pattern in Prior Distribution of Hard Information
Aggregation of Diverse Hard and Soft Information
Aggregated Processing of Hard and Soft Information
Error Reduction of Individual Bias in Interpreting Hard and Soft Information
To develop a method that capitalizes the benefits of both machine and collective intelligence, I followed a DSR approach (Hevner et al. 2004; Gregor and Hevner 2013) guided by schematic steps of (Shmueli and Koppius 2011). The result of this design science project constitutes a hybrid intelligence method as a new and innovative artefact that helps to solve a real-world problem. Following Hevner’s (2007) three cycle view of relevance and rigor I combine inputs from the practical problem domain (relevance) with the existing body of knowledge (rigor). Abstract theoretical knowledge thus has a dual role. First, it addresses the suggestions for a potential solution. Second, the abstract learning from my design serves as blueprint to provide prescriptive knowledge for solving similar problems in the future (Gregor and Jones 2007).
As outlined above, both collective and machine intelligence approaches have specific benefits to predict uncertainty future events. To develop a novel method to predict the future success of early stage start-ups, I decided to combine an ensemble of machine and collective intelligence due to several reasons. First, combining multi methods has a long tradition in research on forecasting as combined methods and sources should be at least similar or better than the best individual prediction method (Armstrong 2001). Second, previous research considered the strengths of human decision makers and statistical models and anticipated their complementary capabilities in making predictions (Blattberg and Hoch 1990; Nagar and Malone 2011). Third, I argue that for contexts of extreme uncertainty due to both information asymmetry and unknowable risk, a hybrid intelligence method consisting of collective and machine intelligence, is superior to one or the other approach alone. A hybrid intelligence method for predicting the success of new ventures enables to statistically model a prediction by consistently accessing and processing “hard” signals based on Bayesian inference, while collective intelligence allows to access more diverse “soft” signals through aggregation and reducing the systematic errors of individual humans, which also helps to reduce the random error term of predictions that reflects unknowable risk through the benefits of human intuition.
A fully automated system with a large flow of information requires data mining trough web crawling approaches and pre-processing the raw data. Within the scope of this paper I focus on the main part of the hybrid intelligence method by explaining input metrics, the automation process and expert weighting to predict the success of early stage start-ups.
The objective of my solution is to accurately predict the success of early stage start-ups and thus make accurate predictions under extreme uncertainty. To define the success of early stage start-ups, I assessed whether they received Series A funding and used this funding as a proxy. This is the commonly accepted indicator for success in this context (Baum and Silverman 2004; Spiegel et al. 2016). Series A funding defines the first venture capital backed funding that allows new ventures to grow. The objective is thus to provide a probabilistic classification of the binary categorical variable Series A funding.
To identify relevant metrics for the input of the hybrid intelligence automation I first conducted a literature analysis (Webster and Watson 2002) and then conducted interviews (n=15; average duration 60 min) and focus group workshops (n=6; 3-7 participants; average duration 90 min) with experienced angel investors to iteratively combine the findings from the literature review with practical relevant factors following the procedure of Nickerson et a. (2013). Based on these findings, I developed a taxonomy of relevant input metrics for a hybrid intelligence method in the context of predicting the success of early stage ventures (Song et al. 2008) (see Table 15). As I focus on early stage start-ups, I do not include financial metrics, which are barely relevant in this stage (e.g. Maxwell et al. 2011).
To automate the prediction through collective intelligence, the signals need to be converted into crowd-readable format. Therefore, I designed a GUI (graphical user interface) that consist of all “hard” and “soft” signals of my taxonomy (Table 14). This ensures that each human can make a prediction based on a comprehensive representation of the start-up. A graphical approach was chosen to improve the information extraction capabilities that are necessary for making these predictions (e.g. Todd and Benbasat 1999). The GUI is based on a standardized ontology that was developed from my taxonomy of signals (Burton-Jones and Weber 2014).
To apply machine intelligence, I use the above-mentioned metrics as input for probabilistic classification of Series A funding. I therefore choose the following learning algorithms due to their different methodological benefits:
1. Logistic Regression: a very well-known linear regression algorithm used as the baseline algorithm. Frequently applied for binary choice models (Magder and Hughes 1997).
2. Naive Bayes: Bayesian parameter estimation problem based on some known prior distribution (Valle et al. 2012).
3. Support Vector Machine (SVM): classification algorithm based on a linear discriminant function, which uses kernels to find a hyperplane that separates the data into different classes (Vapnik 1998).
4. Artificial Neural Network (ANN): a biologically inspired, non-parametric learning algorithm that can model extremely complex non-linear function (Haykin 2009).
5. Random Forests: a popular ensemble method that minimizes variance without increasing bias by bagging and randomizing input variables (Breiman 2001).
The collective intelligence automation is designed as judgment task (Riedl et al. 2013). This asks each human participant of the crowd to rate the start-up on a multi-dimensional 7-point Likert scale, which provides the most accurate results in collective intelligence prediction (Blohm et al. 2016). Each start-up is then assessed along the most relevant dimensions to indicate success: feasibility, scalability, and desirability. To execute the actual rating process, I use two instances of collective intelligence: a crowd of non-experts and an expert crowd. This is consistent with previous research that indicates that a larger group of non-experts as well as a smaller group of domain experts are both capable to predict uncertain outcomes (Klein and Garcia 2015; Mannes et al. 2014).
1. Crowd: open call to a crowd of non-experts. 16-20 humans judge each start-up to predict its success and leverage the benefits of collective intelligence in non-expert samples (Keuschnigg and Ganser 2017). The non-expert crowd is requested through a crowd working platform (Amazon Mechanical Turk). Language proficiency is ensured through choosing ME residents as my start-up descriptions were generated in English and quality selection is conducted through a minimum HIT approval rate of 95% as selection criteria. Each participant is financially compensated.
2. Expert crowd: restricted open call to humans that have high expertise and domain knowledge (i.e. business angels, start-up mentors, consultants) 5-7 experts judge each start-up to predict its success and leverage the benefits of collective intelligence in expert samples (Mannes et al. 2014). The expert crowd is requested through a start-up mentor network and is not be not financially compensated.
The collective intelligence automation is then calculated as unweighted average that combines each crowd member´s ordinal evaluations into a single score indicating the probabilistic classification of Series A funding success, which provides the most accurate combined predictions (Keuschnigg and Ganser 2016).
To benefit from the hybrid intelligence method, I finally need to aggregate the predictions of each individual human and machine intelligence approach to generate the output of probability of Series A funding. I thus apply two types of aggregation. First, I use simple unweighted averaging as baseline approach which proved to be accurate for many types of predictions (Armstrong 2001; Keuschnigg and Ganser 2016). Second, I follow the idea of combining different individual predictions of the crowd and ML algorithms according to their performance (Archak et al. 2011). For this purpose, I propose three of this weighting algorithms to identify the most accurate one. Machine intelligence performance weighting that is based on the predictions of only ML algorithms. Collective intelligence performance weighting that is based on the prediction of the crowd. I argue that in contexts involving consumer product start-ups, non-expert crowds might be superior as they represent the voice of potential customers and providing demand side knowledge whereas expert crowds are more accurate in B2B settings (e.g. Magnusson et al. 2016). Finally, hybrid intelligence performance weighting combines the predictions of machines and humans to predict the success outcome.
To evaluate my method, I implement the input, automation, and aggregation procedures and evaluate their performance. I started by creating a data set of start-ups by combining information on the signals in my taxonomy from several databases (CrunchBase, Mattermark, and Dealroom). Thereby, I focused on tech start-ups as the high-tech context is particularly tainted with extreme uncertainty. The data sample consists of 1500 start-ups from different industries and all extracted signals from the taxonomy. Part of the sample received Series A funding while the other part did not. All start-ups in my dataset are labelled accordingly. I pre-processed the “hard” signals for the machine algorithms and converted all “hard” and “soft” signals into crowd-readable format (an ontology-based GUI). In the next steps, I standardized the input metrics and test to avoid overfitting (Carneiro et al 2017). The automation process of the “hard” and “soft” signals then be conducted as described in the previous section.
The dataset is then randomly split into two sub-sets, one for training and one for testing. The training data is used to create the prediction models. For my purpose, I choose a 10-fold cross validation approach to split my data set into ten mutually exclusive sub-sets of approximately equal size. The idea behind 10-fold cross validation is to minimize the bias associated with the random sampling of the training and holdout data samples. Each of my proposed prediction approaches (i.e. machine algorithms and the crowd) is then trained and tested ten times with the same ten folds, which means the algorithm is trained on nine folds and tested on the remaining single fold. Cross validation then estimates the overall accuracy of an algorithm by calculating the mean accuracy (e.g. Olson et al. 2012).
The evaluate the performance of my method I use the Matthews correlation coefficient (MCC) that is a well-known balanced performance measure for binary classification, when the classes within the data are of very different sizes (Matthews 1975). I choose this measure as my data set is biased towards successful start-ups although I also identified a large amount of failed ventures, which is a common limitation of such databases (e.g. Böhm et al. 2017). I then use logarithmic regression as baseline algorithm and compare the performance of each individual machine intelligence algorithm, crowd prediction, and weighting algorithm through a two-way analysis of variance (ANOVA) (Bradley 1997). I therefore intend to identify the best combined approach and aim at showing that Hybrid Intelligence approach provides superior results than a machine or human only prediction.
Predicting the success of early stage start-ups is a challenging task and the costs of misclassification is high as it might lead to disastrous funding decisions are missing valuable chances for return. To make predictions in such contexts, I propose to combine the complementary capabilities of machine and human intelligence. While machines are particularly beneficial in consistently processing large number of “hard” signals that indicate the success of a new venture humans are superior in interpreting “soft” signals such as the personality of an entrepreneur or the innovativeness of a new product. Moreover, humans can leverage their intuition to identify valuable start-ups that cannot be find by relying on previous data. To overcome the constraints of bounded rationality of individuals, I thus suggest leveraging collective intelligence. To reach my aim, I developed a preliminary Hybrid Intelligence method that I initially evaluate as I proceed my research. In the next steps, I will then also test its applicability for other outcome variables in the context of start-ups (e.g. growth, survival rate etc.) and other contexts of extreme uncertainty (e.g. innovation in general). Moreover, I intend to assess the relevance of accuracy and transparency with potential users of this method and if they are more willing to take advice when human sources are included (e.g. Önkal et al. 2009). I expect my research to make several contributions to both academia and practice. First, I provide a taxonomy of potential predictors that can be generalized for modelling start-up success predictions (e.g. Böhm et al. 2017). Second, this research adds to literature on predictive research in IS and data analytics (e.g. Chen et al. 2012) by introducing a new method for predicting uncertain outcomes under limited information and unknowable risk by combining collective and machine intelligence in a Hybrid Intelligence method. This approach allows to complement formal analysis of “hard” information and intuitive predictions based on “soft” information. Such hybrid method might be valuable for other settings of extreme uncertainty as well. Consequently, my research offers prescriptive knowledge in this vein that might be generalizable for data science methods in general (Gregor and Jones 2007). Third, I contribute to previous work on collective intelligence (e.g. Malone et al. 2009; Wooley et al. 2010) by proposing novel applications of machines and crowd. I argue that my proposed approach can augment the capabilities of collective intelligence in general. While I use a parallel approach in this paper, further research might explore how machine intelligence might be leveraged as feedback for the crowd and thus point towards more collaborative interactive approaches (Dellermann et al. 2019). Finally, I provide a useful solution for a practical prediction problem that may support entrepreneurs in making decisions and potentially reduce the frequency of failure.
See references
Original paper published in ICIS2017
Identify your use-case
Experience next-generation decisions
Get value from day one
In this 5-minute demo, you’ll learn how vencortex® DeOS can help your team achieve better decisions, faster.
Watch demo
