In open source software repositories such as GitHub — an online platform for software development where developers store their projects and collaborate with other developers — the quality of a coder’s technical contributions is undoubtedly important. Such communities have long seen themselves as meritocracies where code is king and decisions are based solely on technical excellence.
But in a recent study at the Cheriton School of Computer Science, researchers found that whether a developer’s contributions on GitHub are accepted by a project manager is just as likely to hinge on key aspects of a developer’s personality and even more so on social factors.
Study researchers (L to R): Cheriton School of Computer Science master’s candidate Alex Yun and Professors Jesse Hoey and Meiyappan Nagappan. Lead researcher, Rahul Iyer, a recent master’s graduate in computer science and now a software engineer at Amazon, was unavailable for the photo.
The researchers began their study by expanding the dataset that was used in a 2014 study conducted by Jason Tsay and his colleagues at Carnegie Mellon University, which found that the strength of the social connection between a software project contributor and project manager influenced the likelihood of a contribution’s acceptance.
“Tsay’s group had conducted research on the influence of social and technical factors on contributions in GitHub,” said Meiyappan Nagappan, a professor at the Cheriton School of Computer Science who specializes in software engineering. “They had collected a lot of data from GitHub at the pull request level. We took their dataset and updated it to 2018 by adding more projects to assess the influence of social and technical factors.”
Using this expanded dataset, the Cheriton researchers reconfirmed that the degree of separation between software project contributors and project managers and whether contributors and managers worked together previously were the two most significant elements determining whether a contribution to an open source project on GitHub is accepted or rejected.
“Our expanded dataset reproduced Tsay’s findings that social factors are very important,” said Professor Nagappan. “If I know and trust a developer because they have made a strong contribution to my project in the past, it’s reasonable to think that the same developer’s current contribution will also be of good quality.”
But the novel twist is that Cheriton researchers also looked at the personality traits of both developers and project managers to see if that has a bearing on whether a contributor’s work is accepted or rejected. To evaluate this, they analyzed the personality traits of 16,935 active developers from 1,860 projects on GitHub, each of which had at least 250 pull requests — a mechanism developers use to notify other open source contributors working on the same project that they have completed a task.
The researchers analyzed their textual data using IBM Watson Personality Insights — a linguistic analytics service that can infer an individual’s personality from email and forum posts they have written — to determine the personality traits of software developer contributors (known as requesters on GitHub) and software project managers (known as closers).
“One of the most influential personality models is the Big Five, which is comprised of several traits that collectively determine a person’s disposition or behaviour,” said Alex Yun, a master’s student at the Cheriton School of Computer Science and a co-author of the study. “IBM Watson Personality Insights provides a measure of a person’s Big Five personality traits — the person’s openness, conscientiousness, extraversion, agreeableness and neuroticism.”
Openness is a measure of a person’s creativity and willingness to try new things. Conscientiousness is a measure of industriousness and orderliness or, in other words, if a person is detail-oriented, reliable and hard working. Extraversion is a measure of gregariousness and the tendency of a person to take charge in social situations. Agreeableness is a measure of compassion and politeness, and people who score highly on this personality trait tend to be nurturing and trusting. Neuroticism is a measure of volatility and withdrawal, and those who score highly on it tend to express negative emotions and sentiment.
The researchers found that requesters who scored as more open and conscientious had a greater chance that their pull request would be accepted by a closer. Extraversion had a small negative effect on pull request acceptance, but the influence was statistically insignificant. And pull requests had a high likelihood of being accepted if the closers scored as more conscientious, extroverted and neurotic.
“The finding that requester and closer do not have similar personalities might seem unusual,” said Professor Nagappan. “But if you dig a little deeper you see a natural explanation. The requester — the person submitting a software contribution in an open source project — and the closer — the person accepting or rejecting it — have very different roles. The requester is trying to get the contribution accepted while the closer is acting as a gatekeeper. In this dynamic, the contributor needs to be open and conscientious. It is not their project so they have to be amenable. If I’m the one requesting and Alex is the one closing, it’s Alex’s project so he sets the terms for what’s acceptable. And he may need to be more neurotic than I am. He might say, ‘Look, you can’t do this’ or ‘you really need to be doing that.’ He needs to be nitpicky, critical and negative, and this personality trait is scored as neurotic by IBM Watson.”
People who contribute to open source projects should be aware of how open they are to discussing new ideas and how conscientious they are, as these personality traits affect how willing people are to work with them and to accept their contributions, Professor Nagappan said. “Often when doing open source work, people think that only their ability to code matters. But we found that that certain personality factors are important because that’s how your behaviour is expressed when you interact with other developers.”
“It’s important to point out that the takeaway from our study isn’t that project closers shouldn’t take into account the quality of code when they’re accepting or rejecting a contribution,” Alex Yun added. “It’s more that developers should realize that project closers are making decisions on more than just technical factors. There might be biases involved that influence the acceptance or rejection of their contributions.”
To learn more about this research, please see Effects of Personality Traits on Pull Request Acceptance, a study conducted by Rahul N. Iyer, S. Alex Yun, Meiyappan Nagappan and Jesse Hoey and published recently in the journal IEEE Transactions on Software Engineering.