Ingunn Myrtveit

Empirical Software Engineering 17(1-2) p. 23-33 Doi: https://doi.org/10.1007/s10664-011-9183-7

NOKOBIT: Norsk konferanse for organisasjoners bruk av informasjonsteknologi

NIKT: Norsk IKT-konferanse for forskning og utdanning

Journal of Business Research 60(11) p. 1191-1197

Do Arbitrary Function Approximators Make Sense as Software Prediction Models? p. 3-10

IEEE Transactions on Software Engineering 31(5) p. 380-391

Empirical studies on software prediction models do not converge with respect to the question "which prediction model is best?" The reason for this lack of convergence is poorly understood. In this simulation study, we have examined a frequently used research procedure comprising three main ingredients: a single data sample, an accuracy indicator, and cross validation. Typically, these empirical studies compare a machine learning model with a regression model. In our study, we use simulation and compare a machine learning and a regression model. The results suggest thatit is the research procedure itself that is unreliable. This lack of reliability may strongly contribute to the lack of convergence. Our findings thus cast some doubt on the conclusions of any study of competing software prediction models that used this research procedure as a basis of model comparison. Thus, we need to develop more reliable research procedures before we can have confidence in the conclusions of comparative studies of software prediction models.

Business Strategy and the Environment (BSE) 12 p. 191-203

IEEE Transactions on Software Engineering 29(11) p. 985-995

IEEE Transactions on Software Engineering 29(5) p. 398-416

Empirical Software Engineering 8(2) p. 139-161

Proceedings of the IEEE International Software Metrics Symposium (METRICS) p. 3-12

Magma forskning og viten 4(3) p. 108-113

IEEE Transactions on Software Engineering 27(11) p. 999-1013

Applied Economics 32(3) p. 0-0

IEEE Transactions on Software Engineering 25(4) p. 510-525

Business Strategy and the Environment (BSE) 8 p. 313-321

Informasjonsproblem og økonomisk organisering

Beta – Scandinavian Journal of Business Research

ARCOM 2005 conference

FIBE 2005

34th EMAC Conference

A crucial cornerstone in the market orientation literature is the relationship between market orientation and performance. However, we still lack knowledge whether the most market-oriented firms are the most productive and profitable, as few empirical studies have used objective performance measures. By applying data envelopment analysis (DEA), we develop a measure of relative productivity, and test the market orientation model with productivity as performance measure. Based on data from the hotel industry, our results indicate that market orientation has only modest effect on productivity.

Samfunns- og næringslivsforskning (SNF)

A crucial cornerstone in the market orientation literature is the relationship between market orientation and performance. Products and services should be designed, developed and offered to customers based on market knowledge, and human and physical assets should be combined to satisfy customers. However, we still lack knowledge whether the most market-oriented firms are the most productive and profitable, as few empirical studies have used objective performance measures. By applying data envelopment analysis (DEA), we develop a measure of relative productivity, and test the market orientation model with productivity as performance measure. Based on data from the hotel industry, our results indicate that market orientation has only modest effect on productivity. However, DEA-productivity analyses can be used to identify best practice in an industry, and if used properly, it can be a useful instrument in the process of designing products and services and be a valuable input into the market orientation of the firm.

Workshop on Predictive Software Models (PSM 2004)

8th IASTED International Conference on Software Engineering and Applications (SEA 2004)

2004 Frontiers in Services Conference

Handelshøyskolen BI

The Mean Magnitude of Relative Error, MMRE, is probably the most widely used evaluation criterion for assessing the performance of competing software prediction models. It seems obvious that the purpose of MMRE is to assist us to select the best model. In this paper, we have performed a simulation study demonstrating that MMRE does not select the best model. The consequences are dramatic for a vast body of knowledge in software engineering. The implications of this finding are that the results and conclusions on prediction models over the past 15-25 years are unreliable and may have misled the entire software engineering discipline. We therefore strongly recommend not using MMRE to evaluate and compare prediction models. Instead, we recommend using a combination of theoretical justification of the models we propose together with other metrics proposed in this paper.

Handelshøyskolen BI

Estimating software development cost with high accuracy is still a largely unsolved problem. Consequently, there is ongoing, high activity in this research field; a large number of different estimation models ranging from mathematical functions to arbitrary function approximators (AFA’s) have been proposed over the last 20+ years. Unfortunately, the studies do not converge with respect to the question “which model is best?” when functions and AFA’s are compared. So far, it has not been understood why this is so. In this empirical study, we show that this is due to inappropriate validation methods as far as the validation of AFA’s is concerned. In fact, the de facto validation method, cross-validation combined with MMRE, will give completely arbitrary results for AFA’s. Obviously, other criteria are called for in order to appropriately assess the performance of AFA’s. This should be a topic of future research

Handelshøyskolen BI

Learning from high performance projects is crucial for software process improvement. Therefore, we need to identify outstanding projects that may serve as role models. It is common to measure productivity as an indicator of performance. It is vital that productivity measurements deal correctly with variable returns to scale and multivariate data. Software projects generally exhibit variable returns to scale, and the output from ERP projects is multivariate. We propose to use Data Envelopment Analysis Variable Returns to Scale (DEA VRS) to measure the productivity of software projects. DEA VRS fulfils the two requirements stated above, and to our knowledge, it is the only method complying with them. The results from this empirical study of 30 ERP projects extracted from a benchmarking database in Accenture identified six projects as potential role models. These projects deserve to be studied and probably copied as part of a software process improvement initiative. The results also suggest that there is a 50% potential for productivity improvement, on average. Finally, the results support the assumption of variable returns to scale in ERP projects. We recommend DEA VRS be used as the default technique for appropriate productivity comparisons of software projects. Used together with methods for hypothesis testing, DEA VRS is also a useful technique for assessing the effect of alleged process improvements.

7th International Software Metrics Symposium

12th European Software Control and Metrics conference

12th European Software Control and Metrics Conference (ESCOM 2001)

The European Software Measurement Conference (FESMA 98),May 6-8

utenTitteltekst

utenTitteltekst

5th IEEE International Symposium on Software Metrics, November 20-21

The 1999 International Sustainable Development Research