Monday, August 1, 2016
Morning Sessions, 10:30am-12:00pm
Balboa
Technology in selection: When is it appropriate?
Paper Presentation (45 minutes)
Allysa Singer & Megan Krueger
The Personnel Board of Jefferson County This paper discusses the effects computer-based testing can have on performance on selection exercises. Results of written work sample exercises were compared between hand written responses and responses composed on a computer, leading to the discussion of factors that should be considered when determining the appropriateness of computer-based assessments in selection.
Ongoing monitoring of human capital decisions: Tools you can use.
Paper Presentation (45 minutes)
Martha Hennen, Ph.D.
Consumer Financial Protection Bureau Presentation describes analytic tools to support ongoing/periodic monitoring of human capital data for fairness and equal opportunity. Discussion covers a compliance monitoring framework developed to engage management in self-evaluation of decisions impacting equal opportunity and diversity. Attendees will gain specific recommendations for methods they can implement in their own organization.
Calavaras
Judging the testing situation: An overview of situational judgment tests
Paper Presentation (45 minutes)
Kelsey Stephens
PAN This presentation will provide a holistic view of situational judgment tests (SJTs)—including use cases, best practices, and other areas of consideration. The audience will leave the presentation with the knowledge necessary to understand the advantages and disadvantages of SJTs and equipped with the ability to develop and score SJTs.
Situational Judgment Tests: Applications and Advances
Symposium (45 minutes)
John Weiner, Clinton Kelly, & Cyrus Mirza, Ph.D.
PSI
Situational judgment tests have been used as an alternative to traditional paper-and-pencil tests for decades. Numerous studies and meta-analyses have demonstrated that SJTs have useful levels of validity as predictors of job performance. SJTs have been shown to have advantages over more traditional “high-fidelity” practical assessments, such as lower cost and the elimination of rater issues such as training and inter-rater reliability. Due to its popularity, the academic literature and case uses of SJTs in personnel selection are extensive. This symposium will provide a background of SJTs and will review various SJT methods that are available. The symposium will also discuss how technology has helped evolve these assessments and will finish with a discussion of potential directions for future research and application. Within these areas we will discuss specific case studies of SJTs in order to provide real world context and applications.
California
Equal Employment Opportunity in Pay: Regulations, Obligations, and Analysis Strategies
Tutorial (90 minutes)
Michael Aamodt, Ph.D. & Kayo Sady, Ph.D.
DCI Consulting This tutorial is focused on equal employment opportunity (EEO) regulations and practices that focus on compensation. The presenters review historical and contemporary regulations, commonly implemented analytic strategies, and areas within the compensation EEO landscape in which industrial/organizational psychologists can offer expertise.
Diablo
Banding, Adverse Impact and Utility
Paper Presentation (45 minutes)
Frank Igou, Ph.D.
Louisiana Tech University
Test score banding methods define a range of scores in which as scores are considered statistically equivalent. In this presentation, data from a 2015 law enforcement promotional procedure will be examined using standard error of difference bands, based and standard on error of measurement (SEM) and standard error of estimate (SEE). Three different score use rules will be compared: (1) Strict top-down selection, (2) fixed bands, diversity-based selection within bands and (3) sliding bands, diversity-based selection. These methods will be compared for occurrences of adverse impact using the 4/5ths rule and Fisher’s Exact Probability Test. Utility of these methods will also be compared using mean Z scores and the Cronbach-Gleser Utility Formula. Current legal status of banding will also be discussed.
Smith v Boston (2015): The Decision and an Insider's Perspective
Paper Presentation (45 minutes)
Joel Wiesen, Ph.D.
Applied Personnel Research
The decision in this Title VII case takes novel, sometimes reasonable stances on content validation and adverse impact. It both raises and lowers the bar for validation. It broadens the definition of adverse impact. The court rejected a claim of validity generalization for this promotional exam for Police Lieutenant.
El Dorado
The Job Analysis is Complete, Now What?
Tutorial (90 minutes)
Elizabeth Reed
City of Columbus Civil Service Commission As personnel assessment practitioners we need to apply the science in real world terms. We are charged with devising and implementing meaningful methods of predicting who will be successful in a myriad of jobs. We complete the job analysis and identify the critical knowledge, skills, and abilities for successful performance in the job. We understand the strengths and weakness of various selection tools. But, how do we take this information and determine the best selection process for the job—within the reality of our organization? The selection tools we opt to use depend upon a number of factors. It’s these factors that will drive our decisions. In this session, we’ll briefly discuss strengths and weaknesses various selection tools then focus on the factors that impact our assessment decisions in a real world context. As time permits, we’ll discuss how to guide a change to the context in which we work.
back to top
Monday, August 1, 2016
Student Poster Session, 12:00pm – 1:00pm
Balboa
Revisiting and Refining Interactions: The Effect of a Personality-General Mental Ability Interaction on Job Performance
Alexander Harris & Nathan Carter, Ph.D.
University of Georgia
Anthony Boyce, Ph.D.
Aon Consulting
Michael Tocci
Proctor & Gamble Despite increasing research on interactionist explanations of job performance, few studies have investigated a personality-general mental ability (GMA) interaction. Support for a personality-GMA interaction has differed and the traits considered have been limited. This study revisits a personality-GMA interaction for predicting performance and considers all Five Factor Model traits.
What do we miss from our nested data? An evaluation of three multilevel models
Li Guan & Nathan Carter, Ph.D.
University of Georgia
Multilevel modeling has gained popularity among organizational researchers. Several measurement approaches are introduced to deal with nested data, but limited research attention is received. To that end, this proposed study compares three multilevel approaches with an aim to suggest an effective approach to better capture nesting and multilevel effects.
Sources of Employer Resistance to Unproctored Internet Testing
Sami Nesnidol & Scott Highhouse, Ph.D.
Bowling Green State University
Assessment practitioners were surveyed regarding reasons for resistance to using unproctored internet testing (UIT) for pre-employment tests. Concerns about data quality and diffusion in the field were significant predictors of resistance. Public sector practitioners were found to be significantly less likely to choose UIT than private sector practitioners.
back to top
Monday, August 1, 2016
Afternoon Sessions, 1:30pm – 3:00pm
Calavaras
Fits and Starts: The Evolution of Testing for the State of California
Invited Session (60 minutes)
Adria Jenkins-Jones, Chief, Selection Division & Bryan Baldwin, Chief, Personnel Management Division
Organization: California Department of Human Resources In this lively discussion, the presenters will discuss the current state of employment testing for the state of California, including significant recent and upcoming changes to our examination software. Using an interview format, the presenters will discuss the massive changes envisioned for statewide testing, and how the California Department of Human Resources is attempting to collaborate with stakeholders to change the traditional paradigm, and significantly improve automation, while maintaining the commitment to merit. The presenters will engage in open dialogue and there will be time for audience members to ask questions about current and future directions of testing for the state.
California
Contemporary Content-Oriented Validation Research: Same Old Song and Dance?
Panel (90 minutes)
Michael Aamodt, Ph.D. & Emilee B. Tison, Ph.D.
DCI Consulting Group
Kathryn Niles-Jolly, Ph.D.
U.S. Office of Personnel Management
Heather Patchell, M.A.
Biddle Consulting Group
Scott C. Whiteford, Ph.D.
Talent Plus, Inc Organizations often employ selection tools to make personnel decisions. The value of these tools depends on whether or not accurate inferences about applicant or employee characteristics can be made based on the selection procedure scores. Content-oriented validation research is a commonly used and potentially useful organizational strategy to support the use of selection procedures, both from organizational value and defensibility perspectives. However, I/O Psychologists have recently debated how useful content-oriented strategies are, particularly relative to other validation strategies. Additionally, recent court cases (e.g., Lopez v City of Lawrence, 2014) have put content-oriented research in the spotlight. Although content-oriented validation strategies are recognized as legitimate and legally defensible methods for test development and evaluation, this panel will provide an updated discussion of content validation in today’s organizational and legal environments. Particular emphasis will be on modern challenges, practical constraints, defensibility considerations, and value of this approach relative to other strategies.
Diablo
Transitioning to computer-based testing: Lessons learned and recommendations
Paper Presentation (30 minutes)
Emily Nguyen & Tomeca Turner
U.S. Secret Service
This paper presentation will discuss the U.S. Secret Service’s migration from wholly paper and pencil testing to computer-based testing for its entry-level selection process. This presentation will address expected and unforeseen challenges to implementing computer-based testing as well as practitioner-focused strategies and recommendations for data management and safeguarding applicant PII.
A Moving Target: Test Security in Temporary Facilities
Paper Presentation (30 minutes)
Brian Ross & Asia Johnson
U.S. Secret Service
This presentation will provide views on how the U.S. Secret Service ensures the security of entry level and promotional exams both domestically and internationally. It will also discuss the potential repercussions of inadequate test security protocols and unsecure testing environments.
El Dorado
Job Analysis: Methods & Implementation
Tutorial (30 minutes)
Tomer Gotlib, Ph.D. & Julie Sampson, Ph.D.
Ergometrics & Applied Personnel Research, Inc.
This tutorial will cover methods and best practices for job analysis with the intent of educating attendees on how to effectively and efficiently collect information about a job. The tutorial will also give attendees a chance to practice writing test questions with the knowledge gained during the tutorial.
Developing Physical Ability Work Simulations
Tutorial (30 minutes)
Oscar Spurlin, Ph.D.
Ergometrics & Applied Personnel Research, Inc.
In our many years of work in design of physical ability tests for industrial and public safety occupations, our methodology has come to stress work simulation tests over standard exercise test formats. This paper will present reasons for this preference, describe the steps in test design, and discuss some of the pitfalls in trying to create effective simulations.
back to top
Tuesday, August 2, 2016
Morning Sessions, 10:30am – 12:00pm
Calavaras
The Challenges of Measuring Interpersonal Communication Skills
Symposium (45 minutes)
James Kuthy, Ph.D. & Heather Patchell, M.A.
Biddle Consulting Group
Interpersonal communication skills are frequently found to be an important factor for success for many jobs. However, there appear to be potential limitations to the validity and reliability of some of the frequently used tests assessing these skills. The presenters will discuss real-world issues they’ve encountered implementing valid and reliable measures of interpersonal communication.
Using Exit Survey Data in Organizational Health Assessment
Paper Presentation (45 minutes)
Martha Hennen, Ph.D. & Tanisha Robinson
Consumer Financial Protection Bureau
This presentation will describe compiling, reporting and using Exit Survey data across time to inform assessment of organizational health. Comparison of exiting employee perceptions by demographic and functional variables is used to inform human capital business partnership, risk assessment and continuous improvement.
California
Perspectives on Orlando's Mass Shooting: What can I/O Psychology do?
Invited Session (90 minutes)
Deonda Scott
City of Orlando
The City of Orlando was recently rocked to its core by the nation’s deadliest mass shooting to date. The horror that unfolded is strikingly similar to our most recent police promotional emergency incident command exercises. This session will explore how the City of Orlando and other organizations have used interactive emergency incident command simulations to select leaders who will be best able to save lives during times of crisis.
Diablo
This job analysis is tasking: A different approach to conducting job analysis to support physical fitness and medical standards
Paper Presentation (45 minutes)
Kathlea Vaughn & Kathy Stewart
U.S. Customs and Border Protection
CBP was faced with the challenge of conducting job analyses of three law enforcement occupations to support the development of medical and physical fitness standards with the goal that the physical standards be equivalent. The presenters will describe the job analysis model developed and specific challenges faced and solutions implemented.
Teaching an Old Dog New Tricks: Moving from Face-to-Face Job Analysis to a Distance Approach
Paper Presentation (45 minutes)
Brooke Parker, M.S., Dennis Pribe, M.A. & Bob Mahlamn, M.A. PMP
The Ohio State University
Our legacy job analysis approach has been highly effective, but not without drawbacks for geographically dispersed organizations and those who cannot afford staff to be offline for extended periods of time. Presenters will compare and contrast the legacy and distance approaches, sharing challenges and design considerations encountered along the way.
El Dorado
Strengthening the HR foundation: Creating career tracks for your HR Workforce
Paper Presentation (45 minutes)
George Johnston
U.S. Secret Service
The U.S. Secret Service sought to improve the career development opportunities for its Administrative, Professional, and Technical (APT) workforce through its APT Career Track Initiative. A thorough look at the creation of the HR Specialist and Instructional Systems Specialist pilot programs will provide insight on the lessons learned.
back to top
Tuesday, August 2, 2016
Afternoon Sessions, 1:30pm – 3:00pm
Calavaras
The Trainer Matters: Cross-Classified Models of Trainee Reactions.
Student Award (45 minutes)
David Glerum
University of Central Florida
A vast number of human resources professionals have been utilizing trainee reactions (Kirkpatrick, 1959) as part of their training evaluation systems (American Society for Training & Development, 2009). Although prior research (Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997; Brown, 2005) has examined the factor structure of trainee reactions, little is known about whether or not variability in trainee reactions reflects differences in trainers (i.e., who conducts the training) or training content (i.e., what is being trained). Using cross-classified random effects models (CCREM; Rasbash & Goldstein, 1994; Raudenbush, 1993) of trainee reactions from over 10,000 employees enrolled in professional development programs, we found that variability across trainers is the primary source of variation in trainee reactions (in comparison to variability in training content), suggesting the trainer influences reactions more than what is being trained. Based on our results, we provide practical and research implications as well as directions for future research.
Capital One & Quintela Group eInterview System
Innovation Award (45 minutes)
Julia Bayless
Capital One Adam Vassar Quintela Group
In response to the clear need for behavioral interviewing to enter the HR technology age, Capital One and Quintela Group embarked upon an online interview system project to digitize the Capital One interview program. The resulting eInterview system represents a significant technology innovation for interviewing practices. The system includes: a Builder & Guides Portal and a Candidates & eGuide Portal. This presentation will discuss the system development and implementation, lessons learned and results achieved.
California
What if and what now: A critical look at the use of content validation methods
Panel (90 minutes)
Erin Wood
PAN
Elizabeth Reed
City of Columbus Civil Service Commission
Kathlea Vaughn
U.S. Customs and Border Protection
Heather Patchell, M.A.
Biddle Consulting Group
Kelsey Stephens
PAN The ability to validate the use of assessment tools is paramount for the ability to create fair and defensible employment programs, and one method frequently used to validate tools used in assessment programs is content validation. Although content validation has been supported as appropriate for use in validating employment measures by both the SIOP Principles and the Uniform Guidelines, in the field of Human Resources and Industrial and Organizational Psychology, content validation standards have not become as uniform as those related to criterion-related validation. As such, the goal of this panel will be to examine content validation as a method through a critical lens, identifying weaknesses and concerns associated with the method, and how experienced practitioners address those issues. The panel intends to create an open discussion with the audience about the realities and limitations associated with conducting content validation studies, and discuss methods for addressing difficult questions and decisions.
Diablo
Improving Candidate Experience: Robust Tactics for Implementing Video Interviewing
Tutorial (90 minutes)
Daniel Schmeling, Ph.D., Anne Scaduto, Ph.D., Jen Harvel, Ph.D. & Maureen Costello, M.A.
Capital One Technology has been leveraged in the hiring process recently by using one-way video interviews where candidates record themselves responding to pre-established sets of questions that are viewed by evaluators at a later time. Research has shown the effectiveness of this technology by comparing one-way video interviewing to traditional phone and in-person interviewing. One-way video interviewing has been shown to be a superior option in terms of the candidate experience, hiring manager experience, efficiencies in the hiring process, cost savings, and quality hiring decisions (Scaduto, Schmerling, & Harvel, 2016). This tutorial will provide a step-by-step guided set of examples and instructions on how to use one-way video interviewing best practices to implement this technology in an organization. The authors will provide role-play exercises and materials that explain and show to session attendees how one-way video interviewing can be efficiently and effectively introduced into any organization large or small.
El Dorado
Affective Assessments - The Missing Link in Employee Engagement
Tutorial (90 minutes)
Akeela Davis
Courageous Business Culture Cognitive assessments show what people can do. Conative assessments show how their brains work in challenging situations. Affective assessments, the missing link, are feelings based, and show what drives people to do, or not do. So how can adding an affective assessment, enhance Employee Engagement in organizations? This session discusses three ways it impacts on communications. First, it makes available the true, current, and changing, voice of employees to the organization. Second, because of that voice, it offers managers a conjoint basis to develop employees. Third, because engagement is feelings based, it shows how opposing motivations create tensions between individuals and within teams. We will discuss how these can be diffused, and the results measured in concrete metrics.
back to top
Wednesday, August 3, 2016
Morning Sessions, 8:30am – 10:00am
Calavaras
Validaiton of Pre-Employment Screenings
Tutorial (60 minutes)
Kathleen Walker
City of Little Rock
Program Information: This presentation will cover basic content validation techniques to ensure that all pre-employment screenings are both content valid and EEOC compliant. Recent and applicable court cases will be discussed to ensure that best practices are utilized, including the application of the Green Factors for background checks. Finally, participants will be provided a step by step validation process with a sample of the required documentation that they can utilize in their agencies.
Visibility in Interviewing: Video Outperforms Phone Interviewing Outcomes
Paper Presentation (30 minutes)
Anne Scaduto, Ph.D., Daniel Schmerling, Ph.D., Jen Harvel, Ph.D. & Maureen Costello, M.A.
Capital One
Abstract One-way video interviews are being leveraged in the hiring process where candidates record responses to a set of questions viewed by evaluators. This study explores the effectiveness of this technology by comparing one-way video interviewing to traditional interviewing within a large, Fortune 500 Company in the financial services industry.
California
Assessing and Developing Executives’ Business Acumen
Symposium (90 minutes)
Ted Prince, Ph.D.
The Perth Leadership Institute
Traditional competency assessments do not specifically focus on innate financial competencies and business acumen. This presentation discusses these and the nine financial signatures of executives. It shows how each of these impacts financial performance and valuation outcomes of organizations. Based on Dr. Prince’s book “The Three Financial Styles of Very Successful Leaders” (McGraw Hill, 2005) the presentation discusses how to identify and measure financial impact and business acumen using formal assessment instruments and how to use these findings to improve company financial performance and valuation.
Diablo
State of Federal Hiring: Update & Future Directions
Symposium (90 minutes)
Michelle Earley, M.B.A., Margaret Barton, Ph.D.,
Patrick Sharpe, Ph.D., & Jennifer Demarais, M.S.
U.S. Office of Personnel Management
Ryan O’Leary, Ph.D.
PDRI
John Ford, Ph.D.
U.S. Merit Systems Protection Board
Some contend that the process for hiring federal employees is broken – that the government is unable to hire and keep top talent. Given the volume of applicants for federal positions and the high-stakes nature of many mission-critical positions, there is an ongoing need to maximize effectiveness and efficiency, while satisfying a wide range of stakeholders. The purpose of this symposium is to report on what is working and what is not working in federal hiring. Presenters representing government, private sector, systems, research, and practice perspectives will share key updates, challenges and opportunities surrounding this critical hiring sector.
El Dorado
Best Practices for Generating Test Specifications from Job Analysis Results
Symposium (60 minutes)
Jason Schaefer & Clinton Kelly
PSI Services
Howard Forston, Ph.D.
East Bay MUD
The results of job analyses provide the foundation for the development of valid and job related examinations. An important step in this process is the development of test specifications that provide a linkage between the job analysis results and the exam’s content and weighting (i.e., the quantity of items or points assigned to each examination topic area). The presenters will provide a brief overview of the literature for using job analysis data to develop test specifications and their implications for the design and execution of job analysis studies. Additionally, a comparison of the various methods available for mechanically combining job analysis and linkage rating results to generate preliminary KSA weights will be explored. Finally, the presenters will discuss how these preliminary weights can be used to guide and facilitate the committee’s specification development discussions ultimately providing a stronger empirical linkage between the job analysis results and the finalized exam specifications.
back to top
|
|
