How Effective is Research Funding? Exploring Research Performance Indicators

This paper deploys bibliometric indices and semantic techniques for understanding to what extent research grants are likely to impact publications, research direction, and co-authorship rate of principal investigators. It includes semantic analysis in the research funding evaluation process to effectively study short-term and long-term funding impact on publication outputs. Our dataset consists of researchers who received research grants from the National ICT Research and Development funding program of Pakistan. Whereas Pakistani researchers’ publications dataset was extracted from Scopus. We show several interesting case studies to conclude that bibliometric-based quantitative assessment combined with semantics can build better sustainable pathways to deploy evaluation frameworks for research funding effectively. The funding data of closed projects from 2007 to 2013 was obtained from ICT R&D public records. The publications dataset was extracted from Scopus data and the details of the statistics were, publications=61,421; researchers=42,376, organizations=213; funded projects=17, funded researchers=23 and funded organizations=10. A significant positive impact (more research output after allocation of funds) has been found for almost all studied organizations. Similarly, a positive funding impact on research output and average co-authorship for the studied cases (investigators under consideration) was found. However, no funding impact was found on the research focus of investigators, i.e., research focus remained almost unchanged after grant allocation. Also, the study suggested the best possible match candidates for collaboration or potential reviewers against the selected project by semantically analyzing the executive summary. Most funded researchers and research organizations have found a positive funding impact on research output (i.e., number of publications). Using semantics along with bibliometric indicators (relating to funding and impacts) can be constructive in making funding programs more effective and for better impacts evaluation; it is recommended for funding agencies to use it in formal framework formation and proposal evaluation process.