arXiv 2026 | AI Interviewers Can Probe and Read Faces — Are Their Scores Reliable?
PolyInterview connects job-specific questioning, digital-human probing, and multimodal assessment into one interview-training pipeline. The system has run 1,564 sessions, but expert ratings also expose weak follow-up depth and feedback fidelity.
Traditional self-practice rarely recreates that. Question banks can produce prompts but will not keep probing based on your answer; video playback can show your facial expression but rarely points out why a particular reply lacks a result, or why a stretch of delivery lost its focus. Practicing with a real person comes closer to the real thing, but cost and availability are limited.
PolyInterview tries to connect these otherwise scattered pieces: generate questions from a job description and résumé, run multi-turn spoken interviews with a digital human, assess along answer content, vocal performance, and nonverbal behavior, then ground scores in evidence and revision advice.
The digital human is only the most visible layer. The harder part sits behind it: how automatic interview training moves from being able to ask questions to being able to explain why it evaluates this way.
01 First, connect the interview-training chain
PolyInterview’s entry point is direct. Users fill in the target company, role, and JD, upload a CV, choose an interviewer persona, and set a training length of 8, 15, or 30 minutes. The system allocates a question budget accordingly and organizes five stages: self-introduction, behavioral questions, skills Q&A, situational questions, and candidate questions.
Once the interview starts, the digital human asks by voice while the system records text, audio, and video in parallel. Each answer is sent to four parallel evaluators that check professional performance, delivery style, nonverbal behavior, and spoken expression. Results are then aggregated layer by layer into assessment dimensions, competency tracks, and personalized suggestions.
As Figure 2 shows, this pipeline puts question generation, follow-up probing, assessment, and feedback into one closed loop. That is also its largest difference from an ordinary chatbot: the previous round’s job requirements, candidate experience, and answer content continue to constrain the next questions and later evaluation.
02 Personalization is not stuffing a job title into a prompt
The paper splits personalization into two moments. Before the interview, the Interview Planner Agent extracts job requirements from the JD, then finds related skills, projects, and experience in the CV, aligning that evidence to KSA — knowledge, skills, and abilities. Question types and training length jointly determine the question structure; generation has explicit job grounding and a question budget.
Figure 6 shows this process. The job description is first aligned with assessment dimensions; different question types are also mapped to abilities that are suitable to observe; then a question-generation agent selects target dimensions and evidence within the time budget. The aim is for later assessment criteria and earlier question-generation grounds to share the same competency language as much as possible.
During the interview, the Interviewer Agent checks whether answers contain contradictions, whether expression is clear, and whether content worth expanding appears. Basic, intermediate, and advanced interviewers may probe for at most 1, 2, and 3 rounds respectively; once time runs out, the probe limit is reached, or the issue is resolved, the system moves to the next stage.
That boundary matters. An LLM without stopping conditions can easily turn an interview into endless chat; real interviews are constrained by time, question count, and pacing at once. The probing mechanism in Figure 7 is precisely a trade-off between adaptivity and controllability.
03 It does not only give a total score — it tries to keep the score’s trail
What often unsettles people about automatic assessment is how clean the score looks. A 7.6 looks precise, yet does not tell the user whether the judgment came from answer content, speaking rate, eye contact, or the model’s own preference.
PolyInterview splits assessment into three layers. The bottom layer has 13 behavioral features, including conceptual accuracy, terminology use, problem-solving logic, structural clarity, coherence, word choice, eye contact, facial expression, body posture, gestures, pronunciation accuracy, prosody, and fluency. The middle layer aggregates these features into 10 assessment aspects; the top layer forms two tracks: professional competence and communication competence.
Different question types do not activate every dimension. Skills questions focus on job knowledge, general intelligence, and education/training; behavioral questions are better for observing interpersonal ability, applied thinking, and leadership; candidate questions are not scored. Each assessment aspect uses a 70% primary-feature / 30% secondary-feature weighting. This gating design admits a simple fact: if a question never elicited a behavior, the system should not pretend it measured that behavior.
The interface follows the same evidence chain. Users first see an overview and the two competency tracks, then drill into the 10 aspects and 13 features, and finally return to per-question diagnosis. The STAR framework is used to check whether behavioral answers cover situation, task, action, and result; when Result is missing, the suggestion directly flags that the outcome and its meaning need to be added.
Figure 4 draws this structure more clearly. KSA is used to specify occupational competence; STAR is used to organize behavioral evidence; Big Five only helps select constructs related to social performance. The paper specially notes that the system neither infers nor reports personality traits — that boundary cannot be dropped.
04 The system runs — but platform data is not proof of effect
The paper’s full-account snapshot includes 101 accounts, 1,564 sessions, and 7,665 generated questions spanning 83 job titles. Of these, 1,422 session-question sets fully cover the five stages — 90.9%. Logs also record 1,274 answered main questions, 342 follow-ups, and 1,425 answers that entered the scoring pool.
Job-match analysis supplies another layer of evidence. Median text similarity between question sets and their corresponding JDs is 0.055, versus a cross-role baseline of 0.0058 against other-role JDs — about 9.5× higher. In 93.7% of sessions, similarity to the corresponding JD exceeds the cross-role baseline; in 82.4% of sessions, the corresponding JD ranks first.
These numbers show that generated questions vary with the role rather than giving every position the same generic set. But the paper also does not package this into a stronger claim. Similarity measures lexical correspondence, not question quality, and still less proves that practice users ultimately got jobs.
Total account count also needs careful reading. The snapshot explicitly includes test, internal, placeholder, and team accounts, so it reflects platform activity, not 101 verified external job seekers. That some people reopen the system can be seen in the 61.4% returning-account share; why they returned, and whether ability improved, the data cannot yet answer.
05 Experts rate question generation highest — and worry feedback invents stories for candidates
The study invited 10 experts with backgrounds in human resources, language assessment, and communication to review 12 de-identified materials: 4 job-customized question plans, 4 answer-and-probe sets, and 4 evidence-backed feedback reports. That produced 120 material reviews and 360 metric ratings.
Results were uneven. Question plans scored highest at 4.62/5, with job relevance at 4.80 and stage coverage at 4.83. Follow-ups overall scored 3.68: relevance reached 4.65, but answer dependence was only 3.28 and diagnostic depth 3.13. The system usually knows what to ask next, yet does not always stay close enough to the previous answer to probe what still needs clarifying.
Feedback reports overall scored 3.74. Actionability of suggestions reached 4.70, yet fidelity of polished model answers to the original answers was only 2.75. Experts found that some demonstrations introduced background the candidate had never stated.
This cannot be passed over lightly. Interview feedback can help users complete a STAR structure, but completing a structure and inventing facts are different things. Once a model writes a prettier experience for the candidate, advice looks more complete while training may slide from improving expression into distorting facts.
06 Which boundary should a training tool hold?
PolyInterview is positioned as an assistive practice tool for candidates — it does not take part in hiring, screening, or gatekeeping. Visual assessment only offers suggestions; every score should expose supporting evidence; the system makes no pass/fail decisions. That positioning matches the current strength of its evidence.
The paper also addresses sensitive data directly. The platform processes CVs, answer text, audio, and interview video; it is deployed on campus servers with encrypted transit and storage, access control, and user-level data isolation. Users can delete interview records; raw data is not used to train external models; paper analyses use only de-identified session metadata.
There are still no longitudinal results showing that repeated PolyInterview use improves real interview performance or employment outcomes, and no controlled pre/post test to isolate causal effects. Expert ratings of materials also cannot replace criterion validation between scores and independent career-advisor judgments. Figure 3’s interface comes from different demo sessions and can only illustrate functional form, not typical user performance.
This work does not prove AI already knows how to be an interview coach. Its contribution is more concrete: a runnable, probe-capable, drill-down training infrastructure, plus two hard problems exposed by expert rating. Follow-ups must truly depend on the answer, and feedback must stay strictly faithful to the evidence.
A talking digital human is only the entrance. Whether every judgment has a trail, and whether every suggestion stays inside the factual boundary, is what decides whether systems like this can move from demos to lasting use.
Paper information
Title: PolyInterview: An LLM-based Platform for Immersive Mock Interview Practice with Comprehensive Multimodal Assessment
Authors: Zhiyuan Wen, Jiannong Cao, Kelly Chan, Zijian Wang, Chen Chen, Xiaoyun Liu, Jianing Yin, Zhuo Li
Affiliations: Department of Computing, The Hong Kong Polytechnic University; School of Artificial Intelligence, Chongqing University of Posts and Telecommunications
Version: arXiv:2607.10310v2, 16 July 2026
Paper link: https://arxiv.org/abs/2607.10310