| NSF Org: |
CNS Division Of Computer and Network Systems |
| Recipient: |
|
| Initial Amendment Date: | July 9, 2020 |
| Latest Amendment Date: | July 9, 2020 |
| Award Number: | 2016604 |
| Award Instrument: | Standard Grant |
| Program Manager: |
Sol Greenspan
sgreensp@nsf.gov (703)292-7841 CNS Division Of Computer and Network Systems CSE Direct For Computer & Info Scie & Enginr |
| Start Date: | September 1, 2020 |
| End Date: | February 28, 2023 (Estimated) |
| Total Intended Award Amount: | $28,000.00 |
| Total Awarded Amount to Date: | $28,000.00 |
| Funds Obligated to Date: |
|
| History of Investigator: |
|
| Recipient Sponsored Research Office: |
1 UTSA CIR SAN ANTONIO TX US 78249-1644 (210)458-4340 |
| Sponsor Congressional District: |
|
| Primary Place of Performance: |
One UTSA Circle TX US 78249-1644 |
| Primary Place of Performance Congressional District: |
|
| Unique Entity Identifier (UEI): |
|
| Parent UEI: |
|
| NSF Program(s): | CCRI-CISE Cmnty Rsrch Infrstrc |
| Primary Program Source: |
|
| Program Reference Code(s): |
|
| Program Element Code(s): |
|
| Award Agency Code: | 4900 |
| Fund Agency Code: | 4900 |
| Assistance Listing Number(s): | 47.070 |
ABSTRACT
A key goal of software engineering and programming languages research is to create new environments, tools, languages, and techniques through which software engineers create software. Evaluating claims with direct evidence about the expected value of a tool or language for developers requires conducting user studies with developers. Yet researchers looking to conduct these studies face a number of significant barriers, including recruiting participants, administering tasks, and reusing materials. As a result, conducting user studies of software engineering tools and programming languages remains infrequent. Reducing these barriers may increase the frequency of these studies, increasing the ability of tools and languages to support developers and enabling developers to be more productive and effective in their work.
This project encompasses planning activities to construct a shared community infrastructure for conducting user studies of developers. Planning activities will be conducted to gather community input to better understand the nature of the problem and obtain feedback on ways in which a future community infrastructure for conducting studies might reduce the barriers researchers face. Community feedback will be obtained on potential configurable components addressing key barriers. A survey and interviews of software engineering and programming languages researchers will be conducted to understand the barriers researchers face and how infrastructure might reduce these barriers. A workshop will facilitate exchange of best practices, and a working group will explore the detailed design of the infrastructure with researchers experienced in user studies. Finally, a summative survey will assess the ability of potential infrastructure to support the studies researchers conduct.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
PROJECT OUTCOMES REPORT
Disclaimer
This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.
A key goal of software engineering and programming languages research is to create new environments, tools, languages, and techniques through which software engineers create software. Evaluating claims about the expected value of a software tool for developers requires human developers for direct evidence. Yet these studies today are infrequent due to the substantial barriers that researchers face in conducting these studies.
This planning project explored the potential for a future community infrastructure for conducting programming user studies. By engaging with potential researchers who might use such a platform, as well as developers who might participate in studies conducted using the platform, we worked to better understand the barriers researchers face and the ways in which new infrastructure might help to reduce these barriers. We conducted interviews with senior and junior researchers as well as developers to examine barriers and best practices in programming user studies. We then conducted 3 working groups with 12 researchers focused on three specific barriers: recruiting participants, knowledge needed, and effort required. From this, we identified 18 barriers researchers encounter, 23 solution strategies that experienced researchers used to address these barriers today, and ways in which future infrastructure might help more broadly reduce these barriers. Finally, we conducted a 1 day in-person workshop bringing together software engineering, human-computer interaction, programming languages researchers, and behavioral science researchers to explore the potential for future infrastructure.
To examine the potential for specific infrastructure, we developed two early prototypes. Building on top of the successful JsPsych platform for conducting psychology user studies, we developed a JsPysch-SE-Plugin which enables a participant to view, edit and run code and answer study questions from anywhere in the world as long as she has access to a browser. And we developed AWS StudyHosts, which creates a virtual machine with a pre-installed IDE, system monitors, and programming tool which is being studied. This prototype supports remote deployment of heavy-weight tasks which require a traditional desktop IDE, such as evaluations of IDE plugins.
Interactions with researchers revealed the potential for future programming user study infrastructure to greatly expand the number of user studies of programming which are conducted. Among 58 researchers responding to our survey, 95% (55) reported they would definitely (28, 48%) or probably (27, 47%) consider using community infrastructure to conduct programming user studies.
Last Modified: 05/24/2023
Modified by: Xiaoyin Wang
Please report errors in award information by writing to: awardsearch@nsf.gov.
