Déjà vu? slacking and hacking in santa fe

 

Alessandro Acquisti & Ritirupa Samanta

Santa Fe Institute Computational Economics Workshop, July 2001

 

Preliminary, not to be distributed

 

1. Introduction

 

A group of 20 students are assigned a homework problem once a week for fifteen weeks. Students can group and regroup as they wish in order to complete their homeworks. Which parameters will become crucial in their interactions, and what dynamics will emerge?

In this paper we propose a computational analysis of this scenario grounded on the use of Brahms - an agent-oriented, activity-based language developed to “model knowledge in situated action and learning in human activities” (Sierhuis and Clancey [1997]).  In what follows, we define a model of students’ motivations and behaviors; we then simulate students’ activities as they form and reform different groups week after week in the course of completing their assignments according to those motivations.  The setup of the model, together with the underlying assumptions and the solution algorithm on which is based, are discussed first. Then, we discuss the methodology that we adopted for the scenario, and present an overview of its actual implementation and initial results. While the results are preliminary, the model’s flexibility allows us to discuss various computational experiments and social scientific applications that would extend the scenario and its implications. 

 

 

2. Model setup

 

How to model the interaction of students who are free to group and regroup week after week in order to complete homework assignments? We approached this problem through an agent-based computational model, where students are agents defined by the ability and effort spent in completing team homework.    Each team is made of n students. N can vary across different runs of the scenario but is fixed throughout the fifteen iterations (weeks) of a single run.  Along with the agents and their teams, a ‘Professor’ acts as a meta-agent that grades the homework and facilitates the matching between agents and teams, hence regulating the formation and dissolution of teams through the rounds.

 

 

3. Model Structure

 

Each student i is defined by a combination of two parameters: an ability a and an effort e.  Both parameters are distributed randomly for each student:

 

 

 

Each student i is a member of a team s at a time t:

 



 

The grade g for a team’s home work is assigned by the ‘Professor’ on the basis of a function g(.) over the function f(.) of all team members’ performances.  For example, g(.) could be a simple average of f(.), which would be on its turn a sum of the arguments of a students’ performances, given by combinations of ability and effort.  The interaction of these two parameters within the f(.) function presents some interesting modeling alternatives that will be explored further in the paper. 

 

 

At the end of the week, each student receives the same grade as that of the team she was member of during that week. Students receive utility over time from the grades they get and from the free time they enjoy, which is inversely proportional to their effort levels. Hence, students try to maximize:

 

 

and where alpha and beta

are weights.

After grades are made public, some students will be dropped by their teams and some others will voluntarily leave their team. A process of new matching is facilitated by the ‘Professor’ and implemented through an algorithm discussed further below.

 

 

 

 

4. Model assumptions

 

The values of each agent’s ability and willingness to work (effort) are fixed. These parameters are private information, even though the performance f that results from them is known to all members inside the team. The only public information available outside the team is the grade of each agent (and team) at the end of each round. However, there is (initially) no learning: students who happened to be in the same teams do not make assumptions on whether the effort and ability levels of their colleagues have remained the same or varied over time.  

We restrict the analysis to cases where the number of components of each group is fixed. We consider the following explicit forms for g and f:

 

 

 

This grade is calculated at the end of each round and a cumulative grade is maintained for each agent that is updated through each round.

 

 

5. The algorithm

 

 

Our computational approach implements the following steps for the game:

 

  1. The game starts with a random group of students assigned to each team. The size of the team remains fixed for the rest of the game.

 

  1. Based on each student’s combination of ability and effort (her contribution to the overall group performance), a grade is calculated for each team. This is also the grade for each student in the team at the end of this period.

 

  1. If a student’s performance is lower than x standard deviations away from the team’s average performance, that agent is asked to leave the team.

 

  1. If no student is dropped in Step 3, the agent with the highest performance in the group can still opt to leave the team, depending on whether her performance is higher than x standard deviations away from the team’s average performance.

 

  1. All teams with vacant positions are then matched with the free agents. Students try to reach the teams with the highest grade in the latest round, and teams try to get the students coming from the team with the highest payoff – the only publicly known information. However, a student cannot return to the team she was with in the preceding round if she dropped it or was evicted from it - unless this is the only possible matching.

 

  1. The next iteration follows.

 

 

6. Hypotheses

 

This model aims to address first the following hypotheses:

H1.  Over time in the basic model we will observe emerging homogeneity in the composition of the groups.  The rationale is that if we do not allow for learning we will expect that depending on the exit rule, at the end of a run, the performance levels for instance may be the same across the team.  Depending on the details of the rules, this would have implications for the payoff of the team if the payoff function relied on complementarity and not substitutability between skills.

H2.  If we include memory and parameter changes through learning, more complex dynamics will emerge.

We will also consider the following questions:

1.       If the parameters are fixed in the payoff metric then does ability outweigh effort in the overall payoff at the end of the run? Do agents consistently do better if they are smarter or can hard work ultimately overcome an initial inequality in talent within this model?

2.       When learning is considered, will agents converge to some level of effort, and how will this level be related to the range of effort at the initial state?  (for example, will a few extremely hardworking members in the class pull the entire class up to a higher equilibrium level of effort?)

3.       Finally, are the results path dependant?

 

 

7. Methodology and preliminary results

 

The above model structure and algorithm have been implemented in Brahms, an agent-oriented, activity-based language developed to “model knowledge in situated action and learning in human activities” (Sierhuis and Clancey [1997]). Brahms can be used to simulate the behavior of people and artifacts when collaborating together.

Brahms links knowledge-based models of cognition with discrete simulation and the behavior-based subsumption architecture. In Brahms, agents’ behaviors are organized into activities, inherited from the groups they belong to. Activities are located in time and space – therefore issues like resource availability and human participation are considered.

More specifically, the architecture includes the following constructs (cf. Sierhuis and Clancey [1997]):

 

Groups of groups containing

       Agents who are located and have

              Beliefs that lead them to engage in

                                    Activities that are specified by

                           Workframes that consist of

                                  Preconditions of beliefs that lead to

                                  Actions, consisting of

                                         Communication Actions

                                         Movement actions

                                         Primitive Actions

                                         Other composite activities

                                  Consequences of new beliefs and facts

                           Thoughtframes that consist of

                                  Preconditions and

                                  Consequences              

 

Brahms is used at NASA to analyze existing work practices and design models of new work systems. In our scenario, we design the Students and the Professor as agents with goals, attributes and beliefs. For instance, a Professor might grade a homework and then communicate the grade to the team’s members with the following (highly stylized) construct:

 

workframe wf_completePayoffTeamA {

      

       repeat: true;

     

       variables:

       foreach(Student) std;

             

       when(knownval(std ismemberof Team))

                    

       do {

       CompletePayoffTeam();

       conclude((Team.latest_team_score = current.tempscoreTeam / n), bc:100, fc:100);

       AnnounceGrades(std);

       }

}

 

A view of one of the simulations we run is presented in the following Figure:

 

 

 

 

 

where students are sending in their homeworks (through communication lines) and the Professor (on the top line) is returning the grades. Analysis of the numerical outputs (see next Figure) currently shows a quick adjustment to an equilibrium where groups become homogeneous in terms of the performance of their members. No cyclical or chaotic dynamics emerge.  These results are ongoing as weight changes and other computational experiments (discussed below) are also being performed.

 

 

 

 

8. Computational experiments

 

We identified four dimensions along which the model can be varied in order to compare different computation experiments. Some of these variations are implicit in the parameters’ structure we have proposed. Some require extensions to the model.

 

A. Who is the decision maker: the agent or the team?

 

B. How is the exit option determined? Is it based on the observed performance or on the level of effort?

 

C. What is each agent’s memory process? Do agents remember the revealed abilities and effort levels of agents with whom they have worked in the past, to what extent is their knowledge imperfect and for how long do they retain this information?

 

D. To what extent can agents alter their initial parameters of ability and effort? Can they become for example harder workers over time?

 

We comment below on each of the above 4 points.

 

A. Decision Maker

In the current model the decision process about who will exit the team and in what way is very team-centric: we assume a simple metric, where those whose performance is below the team average will be asked to leave the team.  This approach can be significantly adjusted in two directions:

1.      Leaving the decision to the team, but introducing more complex decision procedures. For example, in each team a voting process at the end of the round would determine by majority ruling the individual(s) to be forced out. The team would then be endowed with a performance measure of its own, which it will try to maximize.  If we were also willing to adjust the assumption of fixed team size, the team then would reorganize according to its ideal composition and size. Uncertainty would be determined by the fact that the team’s expected payoff would be a weighted combination of the signals it receives from its members, but the information about actual ability and willingness to work remains private.

2.      In a different approach, the decision makers could be the students themselves. This would be the most fluid and complex form of the model along this dimension. Each agent would maintain an array of beliefs about the expected performance of every other agent in the model.  He would then try to maximize her expected payoff from all possible combinations of a fixed team size within the group. She will then go about attempting to form these groups.

 

B. Exit Rule

The exit rule has several interesting implications for the model.  It is possible to introduce learning in this model by making public the information about the effort levels.  By setting the exit rule such that those with a level of effort below a certain team effort threshold are evicted, interesting dynamics emerge should emerge (this is further discussed in the final case). An interesting case here is that of the “weakest link” exit rules, where teams vote off the people that they believe are contributing the least.  However, instead of measuring contribution on the basis of each agent’s own performance only (), it would be interesting to consider interactions such as:, that include the abilities of all other agents in the team. This idea of complementarity might cause people to be voted off the team simply because their levels of effort and ability do not successfully enhance those of others on the team.

 

C. Agent Memory

The case where agents have the ability to remember the features of agents that they have interacted with in the last n rounds complements the scenario in which agents become the primary decision makers.  In this case, we might let the agents learn each other’s effort levels with certainty but receive noisy signals about abilities.  Depending on the weights given to the parameters, incentives might emerge for the hardest working members to rapidly select into stable teams with an above average but consistent payoff.  If memory lengths could have varying degrees, those with a longer memory may be able to do better in identifying and forming successful teams across the time periods.

 

D. Capacity for Change

The final case follows from the previous discussion.  Given memory and agent based decision making, agents might be allowed to alter their behavior and become harder working as they become more aware of the benefits of being identified as such over time. On the other hand, based on the weights given to the parameters, we may find overall average grades declining across teams as harder working but less smart people start to dominate the space.  This represents an interesting application of the principal/agent problem and the effort to form incentive compatible contracts.  In this case, if the principal observes only face effort with certainty and intelligence with wider margins of error, then he may be forced to use some inadequate metric such as face time to determine who should be kept and who should be removed from a team. This may shift the group toward hard working but not necessarily smarter members, and hence a lower than potential average payoff for the whole team.

 

 

9. Conclusions

This paper presents a flexible model that is easily amended to explore various aspects of the problem of team formation and payoff maximization. Agents can be programmed to adapt their level of effort over time and to become the primary decision makers in the model.  By altering their parameters of ability and effort, it is possible to verify whether patterns of homogeneity emerge in teams across time, and the implications this has for the overall payoff structure.  Our preliminary results from the basic programmed version of this model show that such homogeneity does in fact begin to emerge and that the model quickly arrives at equilibrium.  This version will be further developed along the lines presented throughout this discussion.

 

  

10. References

 

Sierhuis, M. and W. Clancey (1997). “Knowledge, Practice, Activities and People”, Presented at the AAAI Spring Symposium Artificial Intelligence in Knowledge Management Stanford University, March 24-26, 1997.