Abstract

Collaborative work often benefits from having teams or organizations with heterogeneous members. In this paper, we present a method to form such diverse teams from people arriving sequentially over time. We define a monotone submodular objective function that combines the diversity and quality of a team and proposes an algorithm to maximize the objective while satisfying multiple constraints. This allows us to balance both how diverse the team is and how well it can perform the task at hand. Using crowd experiments, we show that, in practice, the algorithm leads to large gains in team diversity. Using simulations, we show how to quantify the additional cost of forming diverse teams and how to address the problem of simultaneously maximizing diversity for several attributes (e.g., country of origin and gender). Our method has applications in collaborative work ranging from team formation, the assignment of workers to teams in crowdsourcing, and reviewer allocation to journal papers arriving sequentially. Our code is publicly accessible for further research.

References

1.
Stirling
,
A.
,
2007
, “
A General Framework for Analysing Diversity in Science, Technology and Society
,”
J. R. Soc. Interface
,
4
(
15
), pp.
707
719
. 10.1098/rsif.2007.0213
2.
Hewlett
,
S. A.
,
Marshall
,
M.
, and
Sherbin
,
L.
,
2013
, “
How Diversity Can Drive Innovation
,”
Harvard Bus. Rev.
,
91
(
12
), pp.
30
30
.
3.
Hunt
,
V.
,
Layton
,
D.
, and
Prince
,
S.
,
2015
,
Diversity Matters
, Vol.
1
McKinsey Company
,
New York
, pp.
15
29
.
4.
Paulus
,
P. B.
,
van der Zee
,
K. I.
, and
Kenworthy
,
J.
,
2016
, “Cultural Diversity and Team Creativity,”
The Palgrave Handbook of Creativity and Culture Research
,
Springer
,
New York
, pp.
57
76
.
5.
Horton
,
J. J.
,
2017
, “
The Effects of Algorithmic Labor Market Recommendations: Evidence From a Field Experiment
,”
J. Labor Econ.
,
35
(
2
), pp.
345
385
. 10.1086/689213
6.
Kurata
,
R.
,
Hamada
,
N.
,
Iwasaki
,
A.
, and
Yokoo
,
M.
,
2017
, “
Controlled School Choice With Soft Bounds and Overlapping Types
,”
J. Artif. Intell. Res.
,
58
(
1
), pp.
153
184
. 10.1613/jair.5297
7.
Drummond
,
J.
,
Perrault
,
A.
, and
Bacchus
,
F.
,
2015
, “
Sat Is an Effective and Complete Method for Solving Stable Matching Problems With Couples.
,”
Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)
,
Buenos Aires, Argentina
,
July
, pp.
518
525
.
8.
Charlin
,
L.
, and
Zemel
,
R. S.
,
2013
, “
The Toronto Paper Matching System: An Automated Paper-Reviewer Assignment System
,”
International Conference on Machine Learning (ICML) 2013, Workshop on Peer Reviewing and Publishing Models
,
Atlanta, GA
,
June
.
9.
Liu
,
X.
,
Suel
,
T.
, and
Memon
,
N.
,
2014
, “
A Robust Model for Paper Reviewer Assignment
,”
Proceedings of the 8th ACM Conference on Recommender Systems
,
Foster City, CA
,
October
, pp.
25
32
.
10.
Bertsimas
,
D.
,
Farias
,
V. F.
, and
Trichakis
,
N.
,
2013
, “
Fairness, Efficiency, and Flexibility in Organ Allocation for Kidney Transplantation
,”
Oper. Res.
,
61
(
1
), pp.
73
87
. 10.1287/opre.1120.1138
11.
Dickerson
,
J. P.
, and
Sandholm
,
T.
,
2015
, “
FutureMatch: Combining Human Value Judgments and Machine Learning to Match in Dynamic Environments
,”
AAAI Conference on Artificial Intelligence (AAAI)
,
Austin, TX
,
January
, pp.
622
628
.
12.
Benabbou
,
N.
,
Chakraborty
,
M.
,
Xuan
,
V. H.
,
Sliwinski
,
J.
, and
Zick
,
Y.
,
2018
, “
Diversity Constraints in Public Housing Allocation
,”
Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems
,
Stockholm, Sweden
, International Foundation for Autonomous Agents and Multiagent Systems, pp.
973
981
.
13.
Cummings
,
T.
,
2009
,
Organization Development and Change
,
Wiley Online Library
,
Hoboken, NJ
.
14.
Clutterbuck
,
D.
,
2011
,
Coaching the Team at Work
,
Nicholas Brealey Publishing
,
London
.
15.
Lenhardt
,
V.
,
2004
,
Coaching for Meaning: The Culture and Practice of Coaching and Team Building
,
Insep Editions
,
New York
.
16.
Levi
,
D.
,
2015
,
Group Dynamics for Teams
,
Sage Publications
,
Thousand Oaks, CA
.
17.
Holpp
,
L.
,
1999
,
Managing Teams
,
McGraw Hill Professional
,
New York
.
18.
Humphrey
,
S. E.
,
Morgeson
,
F. P.
, and
Mannor
,
M. J.
,
2009
, “
Developing a Theory of the Strategic Core of Teams: A Role Composition Model of Team Performance
,”
J. Appl. Psychol.
,
94
(
1
), p.
48
. 10.1037/a0012997
19.
Sassenberg
,
K.
,
Jonas
,
K. J.
,
Shah
,
J. Y.
, and
Brazy
,
P. C.
,
2007
, “
Why Some Groups Just Feel Better: The Regulatory Fit of Group Power
,”
J. Personal. Soc. Psychol.
,
92
(
2
), p.
249
. 10.1037/0022-3514.92.2.249
20.
Barrick
,
M. R.
,
Stewart
,
G. L.
,
Neubert
,
M. J.
, and
Mount
,
M. K.
,
1998
, “
Relating Member Ability and Personality to Work-Team Processes and Team Effectiveness
,”
J. Appl. Psychol.
,
83
(
3
), p.
377
. 10.1037/0021-9010.83.3.377
21.
Jordan
,
P. J.
,
Ashkanasy
,
N. M.
,
Härtel
,
C. E.
, and
Hooper
,
G. S.
,
2002
, “
Workgroup Emotional Intelligence: Scale Development and Relationship to Team Process Effectiveness and Goal Focus
,”
Hum. Resour. Manage. Rev.
,
12
(
2
), pp.
195
214
. 10.1016/S1053-4822(02)00046-3
22.
Ross
,
J.
,
Irani
,
L.
,
Silberman
,
M.
,
Zaldivar
,
A.
, and
Tomlinson
,
B.
,
2010
, “
Who Are the Crowdworkers? Shifting Demographics in Mechanical Turk
,”
CHI’10 Extended Abstracts on Human Factors in Computing Systems
,
Atlanta, GA
,
April
,
ACM
, pp.
2863
2872
.
23.
Østergaard
,
C. R.
,
Timmermans
,
B.
, and
Kristinsson
,
K.
,
2011
, “
Does a Different View Create Something New? The Effect of Employee Diversity on Innovation
,”
Res. Policy
,
40
(
3
), pp.
500
509
. 10.1016/j.respol.2010.11.004
24.
Horwitz
,
S. K.
, and
Horwitz
,
I. B.
,
2007
, “
The Effects of Team Diversity on Team Outcomes: A Meta-Analytic Review of Team Demography
,”
J. Manage.
,
33
(
6
), pp.
987
1015
.
25.
Janis
,
I. L.
,
1971
, “
Groupthink
,”
Psychol. Today
,
5
(
6
), pp.
43
46
.
26.
Williams
,
K. Y.
, and
O’Reilly III
,
C. A.
,
1998
, “Demography and Diversity in Organizations: A Review of 40 Years of Research,”
Research in Organizational Behavior
,
B. M.
Staw
, and
L. L.
Cummings
, eds.,
JAI Press
,
Greenwich, CT
, pp.
77
140
.
27.
Harrison
,
D. A.
,
Price
,
K. H.
,
Gavin
,
J. H.
, and
Florey
,
A. T.
,
2002
, “
Time, Teams, and Task Performance: Changing Effects of Surface- and Deep-Level Diversity on Group Functioning
,”
Acad. Manage. J.
,
45
(
5
), pp.
1029
1045
. 10.5465/3069328
28.
Pelled
,
L. H.
,
Eisenhardt
,
K. M.
, and
Xin
,
K. R.
,
1999
, “
Exploring the Black Box: An Analysis of Work Group Diversity, Conflict and Performance
,”
Administr. Sci. Q.
,
44
(
1
), pp.
1
28
. 10.2307/2667029
29.
Cox
,
T. H.
, and
Blake
,
S.
,
1991
, “
Managing Cultural Diversity: Implications for Organizational Competitiveness
,”
Executive
,
5
(
3
), pp.
45
56
.
30.
Bryne
,
D.
,
Clore
,
G.
, and
Worchel
,
P.
,
1966
, “
The Effect of Economic Similarity–Dissimilarity as Determinants of Attraction
,”
J. Personal. Soc. Psychol.
,
4
(
2
), pp.
220
224
. 10.1037/h0023559
31.
Lin
,
H.
, and
Bilmes
,
J.
,
2011
, “
A Class of Submodular Functions for Document Summarization
,”
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies–Volume 1
,
Portland, OR
,
June
, Association for Computational Linguistics, pp.
510
520
.
32.
Lin
,
H.
, and
Bilmes
,
J.
,
2012
, “
Learning Mixtures of Submodular Shells With Application to Document Summarization
,”
Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence
,
Catalina Island, CA
,
August
, pp.
479
490
.
33.
Carbonell
,
J.
, and
Goldstein
,
J.
,
1998
, “
The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries
,”
Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
,
Melbourne, Australia
,
August
,
ACM
, pp.
335
336
.
34.
Zhu
,
X.
,
Goldberg
,
A. B.
,
Van Gael
,
J.
, and
Andrzejewski
,
D.
,
2007
, “
Improving Diversity in Ranking Using Absorbing Random Walks
,”
HLT-NAACL
,
Rochester, NY
,
April
,
Citeseer
, pp.
97
104
.
35.
Zhai
,
C. X.
,
Cohen
,
W. W.
, and
Lafferty
,
J.
,
2003
, “
Beyond Independent Relevance: Methods and Evaluation Metrics for Subtopic Retrieval
,”
Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
,
Toronto, Canada
,
July
,
ACM
, pp.
10
17
.
36.
Kulesza
,
A.
, and
Taskar
,
B.
,
2012
, “
Determinantal Point Processes for Machine Learning
,”
Found. Trends ® Mach. Learn.
,
5
(
2–3
), pp.
123
286
.
37.
Ahmed
,
F.
, and
Fuge
,
M.
,
2018
, “
Ranking Ideas for Diversity and Quality
,”
ASME J. Mech. Des.
,
140
(
1
), p.
011101
. 10.1115/1.4038070
38.
Wilde
,
D. J.
,
2008
,
Teamology: The Construction and Organization of Effective Teams
,
Springer
,
London
.
39.
Cruz
,
W. M.
, and
Isotani
,
S.
,
2014
, “
Group Formation Algorithms in Collaborative Learning Contexts: A Systematic Mapping of the Literature
,”
CYTED-RITOS International Workshop on Groupware
,
Santiago, Chile
,
September
,
Springer
, pp.
199
214
.
40.
Anagnostopoulos
,
A.
,
Becchetti
,
L.
,
Castillo
,
C.
,
Gionis
,
A.
, and
Leonardi
,
S.
,
2012
, “
Online Team Formation in Social Networks
,”
Proceedings of the 21st international conference on World Wide Web
,
Lyon, France
,
April
,
ACM
, pp.
839
848
.
41.
Ahmed
,
F.
,
Dickerson
,
J. P.
, and
Fuge
,
M.
,
2017
, “
Diverse Weighted Bipartite B-Matching
,”
Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)
,
Melbourne, Australia
,
August
, pp.
35
41
.
42.
Ahmadi
,
S.
,
Ahmed
,
F.
,
Dickerson
,
J. P.
,
Fuge
,
M.
, and
Khuller
,
S.
,
2019
, “
An Algorithm for Multi-Attribute Diverse Matching
,” preprint .
43.
Cohen
,
S.
, and
Yashinski
,
M.
,
2017
, “
Crowdsourcing With Diverse Groups of Users
,”
Proceedings of the 20th International Workshop on the Web and Databases, WebDB’17
,
Chicago, IL
,
May
,
ACM
, pp.
7
12
.
44.
Basu Roy
,
S.
,
Lykourentzou
,
I.
,
Thirumuruganathan
,
S.
,
Amer-Yahia
,
S.
, and
Das
,
G.
,
2015
, “
Task Assignment Optimization in Knowledge-Intensive Crowdsourcing
,”
Int. J. Very Large Data Bases
,
24
(
4
), pp.
467
491
. 10.1007/s00778-015-0385-2
45.
Schmitz
,
H.
, and
Lykourentzou
,
I.
,
2018
, “
Online Sequencing of Non-Decomposable Macrotasks in Expert Crowdsourcing
,”
ACM Trans. Soc. Comput.
,
1
(
1
), p.
1
. 10.1145/3140459
46.
Karp
,
R. M.
,
Vazirani
,
U. V.
, and
Vazirani
,
V. V.
,
1990
, “
An Optimal Algorithm for On-Line Bipartite Matching
,”
Proceedings of the Annual Symposium on Theory of Computing (STOC)
,
Baltimore, MD
,
May
, pp.
352
358
.
47.
Mirzasoleiman
,
B.
,
Jegelka
,
S. S.
, and
Krause
,
A.
,
2018
, “
Streaming Non-Monotone Submodular Maximization: Personalized Video Summarization on the Fly
,”
Proceedings of the 32nd AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18)
,
New Orleans, LA
,
Feb. 2–7
,
Association for the Advancement of Artificial Intelligence (AAAI)
, pp.
1379
1386
.
48.
Devanur
,
N. R.
, and
Jain
,
K.
,
2012
, “
Online Matching With Concave Returns
,”
Proceedings of ACM Symposium on Theory of Computing (STOC)
,
New York, NY
,
May
,
ACM
, pp.
137
144
.
49.
Agrawal
,
S.
, and
Devanur
,
N. R.
,
2014
, “
Fast Algorithms for Online Stochastic Convex Programming
,”
Proceedings of the ACM-SIAM Symposium on Discrete Algorithms (SODA)
,
Portland, OR
,
January
,
SIAM
, pp.
1405
1424
.
50.
Yu
,
Q.
,
Xu
,
E. L.
, and
Cui
,
S.
,
2016
, “
Submodular Maximization With Multi-Knapsack Constraints and Its Applications in Scientific Literature Recommendations
,”
2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP)
,
Washington, DC
,
December
, pp.
1295
1299
.
51.
Badanidiyuru
,
A.
, and
Vondrák
,
J.
,
2014
, “
Fast Algorithms for Maximizing Submodular Functions
,”
Annual ACM-SIAM Symposium on Discrete Algorithms (SODA)
,
Portland, OR
,
January
, pp.
1497
1514
.
52.
Ahmed
,
F.
,
Ramachandran
,
S. K.
,
Fuge
,
M.
,
Hunter
,
S.
, and
Miller
,
S.
,
2019
, “
Measuring and Optimizing Design Variety Using Herfindahl Index
,”
Proceedings of the ASME 2019 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Volume 7: 31st International Conference on Design Theory and Methodology
,
Anaheim, CA
,
August 18–21
,
American Society of Mechanical Engineers Digital Collection
, p.
V007T06A007
. https://doi.org/10.1115/DETC2019-97778
53.
Yu
,
Q.
,
Xu
,
L.
, and
Cui
,
S.
,
2018
, “
Streaming Algorithms for News and Scientific Literature Recommendation: Monotone Submodular Maximization With a D-Knapsack Constraint
,”
IEEE Access
,
6
, pp.
53736
53747
. 10.1109/ACCESS.2018.2871668
54.
Lubin
,
M.
,
Yamangil
,
E.
,
Bent
,
R.
, and
Vielma
,
J. P.
,
2018
, “
Polyhedral Approximation in Mixed-Integer Convex Optimization
,”
Math. Program.
,
172
(
1–2
), pp.
139
168
. 10.1007/s10107-017-1191-y
55.
Di Noia
,
T.
,
Ostuni
,
V. C.
,
Rosati
,
J.
,
Tomeo
,
P.
, and
Di Sciascio
,
E.
,
2014
, “
An Analysis of Users’ Propensity Toward Diversity in Recommendations
,”
ACM Conference on Recommender Systems (RecSys)
,
Foster City, CA
,
October
, pp.
285
288
.
56.
Difallah
,
D.
,
Filatova
,
E.
, and
Ipeirotis
,
P.
,
2018
, “
Demographics and Dynamics of Mechanical Turk Workers
,”
Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)
,
Los Angeles, CA
,
February
,
ACM
, pp.
135
143
.
57.
Levay
,
K. E.
,
Freese
,
J.
, and
Druckman
,
J. N.
,
2016
, “
The Demographic and Political Composition of Mechanical Turk Samples
,”
SAGE Open
,
6
(
1
), p.
2158244016636433
. 10.1177/2158244016636433
58.
Siangliulue
,
P.
,
Arnold
,
K. C.
,
Gajos
,
K. Z.
, and
Dow
,
S. P.
,
2015
, “
Toward Collaborative Ideation at Scale: Leveraging Ideas From Others to Generate More Creative and Diverse Ideas
,”
Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing
,
Vancouver, BC, Canada
,
March
,
ACM
, pp.
937
945
.
59.
Kim
,
J.
,
Cheng
,
J.
, and
Bernstein
,
M. S.
,
2014
, “
Ensemble: Exploring Complementary Strengths of Leaders and Crowds in Creative Collaboration
,”
Proceedings of the 17th ACM conference on Computer Supported Cooperative Work & Social Computing
,
Baltimore, MD
,
February
,
ACM
, pp.
745
755
.
60.
Ye
,
T.
, and
Robert
,
L. P.
, Jr.,
2017
, “
Does Collectivism Inhibit Individual Creativity? The Effects of Collectivism and Perceived Diversity on Individual Creativity and Satisfaction in Virtual Ideation Teams
,”
Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing
,
Portland, OR
,
February
,
ACM
, pp.
2344
2358
.
61.
Dong
,
W.
,
Ehrlich
,
K.
,
Macy
,
M. M.
, and
Muller
,
M.
,
2016
, “
Embracing Cultural Diversity: Online Social Ties in Distributed Workgroups
,”
Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing
,
San Francisco, CA
,
February
,
ACM
, pp.
274
287
.
62.
Allahbakhsh
,
M.
,
Benatallah
,
B.
,
Ignjatovic
,
A.
,
Motahari-Nezhad
,
H. R.
,
Bertino
,
E.
, and
Dustdar
,
S.
,
2013
, “
Quality Control in Crowdsourcing Systems: Issues and Directions
,”
IEEE Internet Comput.
,
17
(
2
), pp.
76
81
. 10.1109/MIC.2013.20
63.
Fuge
,
M.
,
Tee
,
K.
,
Agogino
,
A.
, and
Maton
,
N.
,
2014
, “
Analysis of Collaborative Design Networks: A Case Study of Openideo
,”
ASME J. Comput. Inf. Sci. Eng.
,
14
(
2
), p.
021009
. 10.1115/1.4026510
64.
Fuge
,
M.
, and
Agogino
,
A.
,
2014
, “
How Online Design Communities Evolve Over Time: The Birth and Growth of OpenIDEO
,”
Proceedings of the ASME 2014 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. Volume 7: 2nd Biennial International Conference on Dynamics for Design; 26th International Conference on Design Theory and Methodology
,
Buffalo, NY
,
Aug. 17–20
,
ASME
, p. V007T07A038. https://doi.org/10.1115/DETC2014-35243
65.
Moreno
,
P. G.
,
Artés-Rodríguez
,
A.
,
Teh
,
Y. W.
, and
Perez-Cruz
,
F.
,
2015
, “
Bayesian Nonparametric Crowdsourcing
,”
J. Mach. Learn. Res.
,
16
(
1
), pp.
1607
1627
.
You do not currently have access to this content.