Shilei Tian

PhD Student at Stony Brook University

In order to understand how OpenMP works internally, in addition to the essential part of functions, we also need to learn some basic data structures. This series of posts are going to give you a basic impression on some OpenMP essential data structures in runtime library. Let’s get started from the most basic and also one of the most complicated one: kmp_base_info.

typedef struct kmp_base_info {
  kmp_desc_t th_info;
  kmp_team_p *th_team; /* team we belong to */
  kmp_root_p *th_root; /* pointer to root of task hierarchy */
  kmp_info_p *th_next_pool; /* next available thread in the pool */
  kmp_disp_t *th_dispatch; /* thread's dispatch data */
  int th_in_pool; /* in thread pool (32 bits for TCR/TCW) */

  int th_team_nproc; /* number of threads in a team */
  kmp_info_p *th_team_master; /* the team's master thread */
  int th_team_serialized; /* team is serialized */
  microtask_t th_teams_microtask; /* save entry address for teams construct */
  int th_teams_level; /* save initial level of teams construct */

  kmp_uint64 th_team_bt_intervals;
  omp_allocator_handle_t th_def_allocator; /* default allocator */
  /* The data set by the master at reinit, then R/W by the worker */
  int th_set_nproc; /* if > 0, then only use this request for the next fork */
  kmp_hot_team_ptr_t *th_hot_teams; /* array of hot teams */
      th_set_proc_bind; /* if != proc_bind_default, use request for next fork */
      th_teams_size; /* number of teams/threads in teams construct */
  int th_prev_level; /* previous level for affinity format */
  int th_prev_num_threads; /* previous num_threads for affinity format */
  kmp_local_t th_local;
  struct private_common *th_pri_head;

  kmp_team_p *th_serial_team; /*serialized team held in reserve*/
  /* The following are also read by the master during reinit */
  struct common_table *th_pri_common;
  volatile kmp_uint32 th_spin_here; /* thread-local location for spinning */
  /* while awaiting queuing lock acquire */
  volatile void *th_sleep_loc; // this points at a kmp_flag
  ident_t *th_ident;
  unsigned th_x; // Random number generator data
  unsigned th_a; // Random number generator data

  /* Tasking-related data for the thread */
  kmp_task_team_t *th_task_team; // Task team struct
  kmp_taskdata_t *th_current_task; // Innermost Task being executed
  kmp_uint8 th_task_state; // alternating 0/1 for task team identification
  kmp_uint8 *th_task_state_memo_stack; // Stack holding memos of th_task_state
  // at nested levels
  kmp_uint32 th_task_state_top; // Top element of th_task_state_memo_stack
  kmp_uint32 th_task_state_stack_sz; // Size of th_task_state_memo_stack
  kmp_uint32 th_reap_state; // Non-zero indicates thread is not
  // tasking, thus safe to reap

  /* More stuff for keeping track of active/sleeping threads (this part is
     written by the worker thread) */
  kmp_uint8 th_active_in_pool; // included in count of #active threads in pool
  int th_active; // ! sleeping; 32 bits for TCR/TCW
  struct cons_header *th_cons; // used for consistency check

  /* Add the syncronizing data which is cache aligned and padded. */
  kmp_balign_t th_bar[bs_last_barrier];

  volatile kmp_int32
      th_next_waiting; /* gtid+1 of next thread on lock wait queue, 0 if none */
  kmp_cond_align_t th_suspend_cv;
  kmp_mutex_align_t th_suspend_mx;
  std::atomic th_suspend_init_count;
  std::atomic th_blocking;
  kmp_cg_root_t *th_cg_roots; // list of cg_roots associated with this thread
} kmp_base_info_t;

In order to make it more understandable, I’ve removed some unimportant fields controlled by macros, like OMPT. I also remove OS dependent part, only focusing on Linux. However, they just have different types on different OS. It does not affect the essential concept.

(To be continued…)