Monday, July 20, 2015

Student Poster Session, 12:00pm – 1:30pm

Charleston

Communication Preference Scale

Scott Hines, Mike Knott, & Jason Marks

Louisiana Tech University

With many communication mediums available in today’s workplaces, the possibility exists that conflicting preferences may result. The purpose of this study is to develop a measure of employee communication preference in a sample of approximately 200 working adults. Future research may include examinations of cross-generational communication preferences in the workplace.

Network analysis of psychometric organizational survey data: Using a visualization tool to ease interpretation and target interventions

Rachel Williamson & Nathan Carter, University of Georgia

Michael Zickar, Bowling Green State University

Network analysis has been a groundbreaking method in a variety of scientific disciplines. Recent work extending network analysis to psychometric data has opened endless possibilities. Here we demonstrate the application of network analysis to personnel survey data, highlighting its ability to ease data interpretation and result in highly targeted interventions.

Regional Accent Bias in Hiring Practices

Ellen Lovell, Mitzi Desselles, & Brittani Plaisance

Louisiana Tech University

This paper draws attention to an under-explored contributor to employment discrimination: regional bias. Without evidence that regional accent differences reflect differential job performance, inaccurate and unfair personnel decisions may result. We propose research into the impact of regional bias on validity and suggestions to reduce the negative consequences for practice.

The Cyber Smear Propensity Scale

Olivia Reinecke & Tilman Sheets

Louisiana Tech University

Making use of the Internet to intentionally damage the reputation of one’s employer, sometimes referred to as cyber smearing, is of escalating concern to many of today’s organizations. The purpose of this study is to develop a scale measuring employees’ propensity to engage in cyber smearing against their employers.

How Much Validity is Enough? Information Presentation Affects Validity Thresholds

Chelsea Chatham, Yuyan Zhang, Chris Arnold, Margaret E. Brooks, & Scott Highhouse

Bowling Green State University

When assessing the effectiveness of selection tests, the use of different effect size indicators can impact how easy people perceive to understand the information, and whether people perceive an effect size as minimally acceptable. The poster provides insights for both researchers and practitioners in communicating validity information with different audiences.

Which Psychological Testing Format Should be Trusted: Likert or Forced-Choice?

Li (Ada) Guan & Nathan Carter, University of Georgia

Jeffery Conway & Anthony Boyce, Aon Hewitt

 Alan Mead, Talent Algorithms Inc.

Personality testing is used frequently in the workplace for various purposes, however, there are on-going discussions regarding the credibility of using such testing, especially the Likert format testing. Two of our studies discuss advantages/disadvantages using Forced-Choice (FC) format and the Likert format testing.

Monday, July 20, 2015

Afternoon Sessions, 1:30pm – 3:00pm

 

Salon I, III, V

Video-Based Testing: Steps, Tips, and Guidance on Developing Your Own Video Tests

Tutorial (90 minutes)

Julie Sampson

Ergometrics & Applied Personnel Research, Inc.

The use of videos can enhance the selection and promotional process for public safety departments. Overall, research has demonstrated several advantages to using video-based tests. The session will review the research behind the use of video in the selection process and provide data supporting this research. The main focus of the session will be on the development of video scenarios from pre-shoot logistics to post-production graphics/audio work. After the session, attendees will have a solid understanding of video-based testing and the critical steps needed to develop effective video-based exams.

Salon II

Presenting a Data Driven Business Case for Candidate Assessment

Paper Presentation (45 minutes)

Martha Hennen

Consumer Financial Protection Bureau

This presentation will describe analytic techniques and tips for using available applicant flow information to support your assessment processes. Discussion will cover: data typically available across vacancy announcements, analytic decisions and data presentation tips. Attendees will gain specific recommendations for simple methods they can implement in their own situation.

Conducting Job Analyses in the Real World: Five Challenges I Wish I Had Learned How to Solve in Graduate School

Paper Presentation (45 minutes)

Michael Blair, H&R Block

Jennifer Geimer & Paul DeKoekkoek, CEB

Graduate programs in applied HR disciplines typically educate students on traditional job analysis techniques. Unfortunately, these techniques rarely prepare the student for job analysis in the real world. The authors will discuss real world challenges (e.g., time constraints, lack of incumbents) and potential solutions to these challenges drawn from experience.

Salon VI

An Innovative Succession Planning Program in the Federal Aviation Administration

Paper Presentation (45 minutes)

Anne Marrelli & Michael Patsfall

Federal Aviation Administration

Formal succession planning is rare in Federal agencies. Regulations specific to government pose challenges that the private and non-profit sectors do not face. This paper will describe an innovative succession planning program developed by the Federal Aviation Administration and the structured assessment methods employed to ensure an objective, merit-based process.

Closing the Gaps in Competency Gap Analysis

Paper Presentation (45 minutes)

Rebecca Fraser, Melissa Buford, & Anne Holloway-Lundy

U.S. Office of Personnel Management

For several years, the Federal Government has emphasized assessing and closing skills gaps across the federal workforce, thereby increasing the demand for competency gap analysis services. This presentation offers practitioners various approaches and considerations for conducting gap analysis, and discusses common challenges to success and strategies to address them.

Atlanta

Back Away From the Cliff! Re-evaluating Online Testing Practices after the Lemming Effect

Paper Presentation (45 minutes)

Amy McKee

Pearson VUE

In the last 15 years, online testing and unproctored selection testing has become de rigueur. However, have we all gone over the cliff a bit without fully considering the implications? This paper reviews the expert opinions along with research and employer practices to bring you along on the P&P to CBT to UIT journey. We will revisit what high-stakes tests are in different environments and discuss the best approach for administration, considering the candidate experience as well as test validity and security.

Using Social Networking Websites to Screen Job Candidates: Is it Worth the Risk?

Symposium (45 minutes)

Scott Hines, Lauren Mouton, & Jason Marks

Louisiana Tech University

Social networking websites allow individuals to share information with others over the Internet. Recruiters and hiring managers will often utilize social networking sites to access this information and use it to recruit, screen, and possibly select job candidates. In this symposium, we will discuss the potential advantages and disadvantages of utilizing social networking sites for recruitment, screening, and selection purposes from practical, legal, and ethical standpoints. We believe that the information presented in our symposium will help attendees to think critically about and make informed decisions regarding the utilization of social networking sites at their own organizations for the purposes mentioned above.

Columbia

Total Classification Redesign: From Case for Action to Applications

Symposium (90 minutes)

Wayne Baughman, Shonna Waters, & Tanya Perry

National Security Agency

Chelsa Gurkin & Robyn Trotter

U.S. General Accountability Office

The U.S. General Accountability Office (GAO) has recently recommended the modernization of the classification system used by Federal government organizations (GAO, 2014). We first describe the process used by GAO to evaluate the current state of classification practices and identify eight attributes of a modern, effective classification system. Next, we describe key barriers to implementing modern classification systems and present a framework, principles, and practices that can overcome these barriers and enable more effective classification practices to better drive organizational performance. Finally, we present an example of a promotion process where its redesign involved applying the principles and practices of modern classification. In addition, we will describe the project context, design approach, and the implementation of the revised process. Finally, we will review lessons learned and what we perceive as best practices because of this project.

Tuesday, July 21, 2015

Morning Sessions, 10:30am – 12:00pm

Salon I, III, V

Collecting Multiple Measures in a Single Assessment

Paper Presentation (30 minutes)

Todd Baker & Deborah Gebhardt

Human Performance Systems, Inc.

For all jobs, effectiveness is determined by multiple attributes. For the electrical lineman job effective pole climbing is a function of physical ability, technique, and safety. This paper describes methods used to develop pole climbing assessments measuring multiple attributes. These methods can be used to develop other multiple measure assessments.

Hiring in the Fast Lane

Symposium (60 minutes)

Karen Walker, Brian Ross, & George Johnston

U.S. Secret Service

The U.S. Secret Service utilizes an intricate hurdle hiring system to ensure it selects the most qualified applicants for entry level Special Agent and Uniformed Division Officer positions. This hurdle system has multiple steps and can take up to 12 months for some applicants to endure. Through the implementation of an Entry-level Assessment Center (ELAC) the U.S. Secret Service has expedited its hiring to fill triple the amount of classes in half the amount of time with qualified applicants. The ELACs provide the U.S. Secret Service with an influx of best qualified applicants that can then be further screened for best organizational fit by competency, ability to obtain security clearance, and meet the medical and physical requirements.

Salon II

Applying and Communicating the ROI of Human Capital Initiatives to Drive Change in Organizations

Symposium (60 minutes)

Melinda Seibert, National Security Agency

Eric Sydell, SHAKER

Michael Blair, H&R Block

David Swiderski, University at Albany, State University of New York

Shonna Waters, National Security Agency

The recent recession has caused business leaders in both private and public sectors to tighten budgets in an effort to meet changing economic circumstances. As a result, internal and external consultants alike have been asked to demonstrate the impact of human capital initiatives on the bottom line. This symposium will offer a forum for human resources professionals being asked to translate the value of solutions into metrics that matter to businesses. Presenters will discuss a business-outcome focused approach to demonstrating return on investment (ROI) that can be applied in a variety of industry settings. Presenters will share methods, measures, and tools they have used to affect change in organizations based on demonstrating the economic ROI of assessments. Lessons learned will guide future practitioners as they seek to conduct ROI analyses to help demonstrate the impact of human capital initiatives to affect change in organizations.

Accuracy, not Aspirations: Tips for Identifying and Preventing Applicant Faking

Paper Presentation (30 minutes)

Nivedita Prabhu               , Olivia Reinecke, & Ellen Lovell

Louisiana Tech University

Research has demonstrated that faking in selection can be costly for organizations. Because faking poses a threat to organizations, this proposal focuses on ways to reducing faking likelihood within assessments. More specifically, methods geared towards alleviating the impact of faking in both survey and interview context will be explored further.

Salon VI

The Debate: On-site versus Remote Video Web-based Assessment

Tutorial (90 minutes)     

Allysa Singer & Martinique Alber

Personnel Board of Jefferson County

In the past decade, there have been significant advancements in digital media technology. Specifically, the delivery of test content via video and the recordings of candidate responses on digital media has become more accessible and user friendly. Many organizations and consultants have begun to take advantage of the use of technology in personnel selection and assessment. With fiscal and logistical concerns of the assessment process, it has become more appealing to discover new ways to tackle this challenge. Through a practitioners viewpoint the benefits and drawbacks to both on-site and remote video web-based assessments as solutions to these challenges are thoroughly discussed. Main topics include common assessment practices, assessor training, test security, logistical concerns, and costs.

Atlanta

Batten down the Hatches: Preparing for the Silver Tsunami

Paper Presentation (30 minutes)

Mike Knott, Scott Hines, & Frank Igou

Louisiana Tech University

Massive workforce change is on the horizon as baby-boomers edge towards retirement. More individuals may be leaving the workplace than entering it. To protect cognitive capital in this knowledge-based economy, organizations will need to adapt. This paper examines how training, retention, selection, and ergonomic interventions can address these concerns.

Student Paper Competition Award Winner

Examining the Process Underlying Responses to Personality Measures in High-Stakes Situations: Does the Item Response Process Matter?

Paper Presentation (30 minutes)

Mengyang Cao

University of Illinois

We investigated the underlying response process for personality assessment administered in high- versus low-stakes situations, and whether any differences matter for selection outcomes. In Study 1, we used an experimental design to investigate whether the dominance model or the ideal point model exhibited better model fit to responses to personality scales. Results suggested that respondents adopted a trait shift strategy as opposed to a response model shift strategy. Specifically, individuals continued using an ideal point response process, but inflated their latent trait standings when they responded to personality items in a job application scenario. In Study 2, we used Monte Carlo simulations to further demonstrate that misapplying the dominance model to score ideal point responses may lead to lower predictive validity and lower mean job performance, especially when the selection ratio is low. On the other hand, applying the correct item response model resulted in more fake-resistant selection decisions.

Columbia

Multiple Perspectives on Women in Leadership

Panel Discussion (90 minutes)

Bharati Belwalkar, Louisiana Tech University

Lisa Moore, Florida Institute of Technology

Elizabeth Reed, City of Columbus Civil Service Commission

Mitzi Desselles, Ph.D., AROS Consulting at Louisiana Tech University

Lt. Col. Karen Gregory, USSOCOM and HRSS Consulting Group, LLC

Rae Reed, WordUp, LLC.

 

The present interactive panel session is intended for students and young professionals who desire to climb organizational hierarchy. This panel discussion will revolve around understanding the issue of gender-disparity in leadership. Successful women leaders representing academia, industry, and military will share their thoughts and experiences on this issue. This panel discussion will also attempt to explore ways in which the existing gender-disparity can be reduced.

Tuesday, July 21, 2015

Afternoon Sessions, 1:30pm – 3:00pm

 

Salon I, III, V

Promoting Civil Service

Tutorial (90 minutes)

Elizabeth Reed

City of Columbus Civil Service Commission

Civil Service Commissions are underfunded and misunderstood. Civil Service is not a sexy topic. In reality, Civil Service is just as important today as it was a hundred years ago when Commissions were first being formed in the United States. Over the past 50 years the field of assessment and measurement of performance has evolved and matured. The time is ripe for effective and meaningful Civil Service, but how do we educate stakeholders? How do we promote Civil Service for what it is and what it can be? If Civil Service is not funded and stakeholders are not educated, our critics will be right. How do we change the message? How do we advocate for the right Civil Service reform? How do we campaign for funding? The message may not be sexy, but our voice is critical to success.

 

Salon II

Apples to Oranges? Use and Comparative Scores for Mobile and Non-Mobile Selection Assessments

Paper Presentation (60 minutes)

Erin Wood, Katherine Sliter, & Kelsey Stephens

pan - Performance Assessment Network

Unproctored employment testing using mobile devices has recently emerged as an important issue. This presentation reviews the current literature on mobile assessment and presents the results of an archival review using cognitive, personality, mechanical aptitude, and work attitudes measures. Issues of mobile testing frequency and demographic differences will be discussed.

 

Addressing Common Applicant Assessment Challenges via Internet Based Testing

Paper Presentation (30 minutes)

Charles Thompson, U.S. Office of Personnel Management

Nicholas Martin, Aon-Hewitt

Melissa Buford, Jaron Holmes, & Philip Walmsley

U.S. Office of Personnel Management

This session examines three common challenges associated with designing and implementing personnel selection procedures. We consider 1) the value to an organization, 2) whether and how general competencies can be measured, and 3) user acceptance issues. We discuss our experiences in large-scale online testing and highly encourage audience participation.

Salon VI

Developing Knowledge-Based Written Exams – How to Maximize Reliability and Validity, and Minimize Challenges and Group Differences

Tutorial (60 minutes)

Tomer Gotlib

Ergometrics & Applied Personnel Research, Inc.

This session will cover best practices for the development of knowledge-based multiple-choice written exams with the intent of educating attendees on how to maximize reliability and content validity, while minimizing challenges and group differences. The session will also give attendees a chance to practice writing test questions with the knowledge gained during the session and receive feedback.

 

In-baskets: Development, Validity, and Utility

Paper Presentation (30 minutes)

Deborah Whetzel

Human Resources Research Organization

This presentation will describe steps for developing in-baskets using a content-oriented approach, including examples in-basket items. The latest research on the validity of in-baskets also will be presented. Finally, the improved utility of in-baskets over random selection will be described.

 

Atlanta

Test Construction Guidance in the New (2014) Testing Standards

Paper Presentation (30 minutes)

Joel P. Wiesen

Applied Personnel Research

Late in 2014 a revised edition of the Standards for Educational and Psychological Testing was issued by the American Psychological Association and two other professional societies. This is the first revision of the Standards since 1999. The new Standards contain many rather specific statements about test development and validation. These will be summarized.

 

The Test is Valid because I say it is Valid (and Other Ways to Lose in Court)

Tutorial (60 minutes)

Jim Kuthy & Heather Patchell

Biddle Consulting Group, Inc.

Test developers frequently assume the validation reports they produce demonstrate a practice, procedure, or test is “job related and consistent with business necessity” (Griggs v. Duke Power Co., 1971). However, they are often surprised when their evidence is found to be problematic when challenged in court. For example, we frequently review validation studies that inappropriately rely on techniques related to validity generalization or the “transportability” of content-related evidence, neither of which has been traditionally accepted by the courts. Other reports skip important steps in the validation process, while still others inappropriately rely on the test creator’s reputation as evidence of its validity. During this session we will go over the process we use when reviewing the appropriateness of job analysis and validation studies to determine whether they address the requirements of the federal Uniform Guidelines on Employee Selection Procedures, which is the standard that the EEOC and courts rely on.

Columbia

Optimal selection: Predictive efficacy and diversity fairness using Pareto-optimal selection systems

Paper Presentation (45 minutes)

Jose Valadez, Christoph Gloger, & Frank Igou

Louisiana Tech University

Although advances in the design of more-sophisticated selection systems have been proposed, implementation of these systems in real-world settings has been gradual (De Corte, Sackett, & Lievens, 2011). The purpose of this presentation is to assist practitioners through a stepwise example of a Pareto-optimal selection system that is concurrently efficient and fair.

Development of the Adaptive Employee Personality Test

*** Innovations in Assessment Award Winner ***

Paper Presentation (45 minutes)

Anthony S. Boyce, Jeffrey S. Conway, Pat M. Caputo & Christopher R. Huber

Aon Hewitt

Does the world need another workplace personality assessment? Well, yes, it does, especially if it is adaptive, mitigates faking, and is developed from the ground-up to be globally relevant. This talk will cover the development of the Adaptive Employee Personality Test (ADEPT- 15). Specific focus will be devoted to the development of the personality model, choice of the response format (forced-choice), identification of the IRT model (i.e., the multi-unidimensional pairwise preference model), content development, calibration, validation, and global localization. Additional research focusing on applicant reactions and faking will also be shared.