Guide to Creating Scoring Criteria for Virtual Coaching

Sections:

• What Virtual Coaching Effectively Evaluate
• What Virtual Coaches Cannot Effectively Evaluate
• Best Practices for Creating Scoring Criteria
• Creating Scoring Criteria for Virtual Coach Evaluation

 

What Virtual Coaching Can Effectively Evaluate:

1. Language Patterns and Word Choice

Can Evaluate:

• Frequency of specific words or phrases
• Professional vs. casual language usage
• Technical terminology accuracy
• Filler words and verbal crutches
• Inclusive language patterns

Example Metrics:

1. "Uses industry-specific terminology correctly at least 3 times"
2. "Reduces filler words ('um,' 'like,' 'you know') to less than 5% of total words"
3. "Employs inclusive language ('we,' 'our') in at least 60% of statements"

2. Conversation Structure

Can Evaluate:

• Question types and frequency
• Turn-taking patterns
• Topic transitions
• Conversation flow
• Time management through text indicators

Example Metrics:

• "Asks at least 5 open-ended questions during discovery"
• "Maintains balanced speaking turns (neither dominating nor passive)"
• "Uses clear transitional phrases between topics"

3. Active Listening Indicators

Can Evaluate:

• Verbal acknowledgments
• Information recall
• Follow-up questions
• Paraphrasing accuracy
• Reference to previously mentioned points

Example Metrics:

• "References client's previous statements at least 3 times"
• "Uses acknowledgment phrases appropriately ('I understand,' 'I see what you mean')"
• "Asks relevant follow-up questions based on client's responses"

4. Sales Technique Application

Can Evaluate:

• Value proposition clarity
• Objection handling approaches
• Feature-benefit connections
• Trial closes
• Commitment-seeking language

Example Metrics:

• "Connects each feature mentioned to a specific client benefit"
• "Uses at least 3 different trial closing techniques"
• "Addresses objections using established frameworks (Feel-Felt-Found, etc.)"

5. Discourse Analysis

Can Evaluate:

• Turn-taking patterns
• Response relevance and timing
• Conversation flow and direction
• Interruption patterns
• Discussion thread maintenance
• Topic transition effectiveness
• Conversation repair techniques (clarification, correction)

Example Metrics:

• "Maintains logical progression between topics without abrupt changes"
• "Responds directly to client questions within 1-2 conversational turns"
• "Successfully redirects off-topic discussions back to relevant points"
• "Uses appropriate conversation repair techniques when misunderstandings occur"
• "Maintains cohesive discussion threads without losing context"

6. Content Analysis

Can Evaluate:

• Story structure and coherence
• Example relevance and effectiveness
• Solution alignment with stated needs
• Problem-solution mapping
• Value proposition clarity

Example Metrics:

• "Presents solutions that directly address each stated problem"
• "Uses relevant examples to illustrate key points"
• "Maintains clear connection between features and benefits”

 

 

What Virtual Coaches Cannot Effectively Evaluate

1. Non-Verbal Elements

Cannot Evaluate:

• Body language
• Facial expressions
• Eye contact
• Physical gestures
• Visual engagement

Do Not Include Metrics Like:

• "Maintains appropriate eye contact"
• "Uses confident body language"
• "Shows engagement through nodding”

2. Audio Elements

Cannot Evaluate:

• Voice tone
• Speaking pace
• Volume
• Emotional inflection
• Pronunciation

Do Not Include Metrics Like:

• "Speaks with appropriate pace"
• "Uses confident tone of voice"
• "Shows enthusiasm through vocal variation”

3. Visual Materials

Cannot Evaluate:

• Slide presentations
• Physical demonstrations
• Visual aids
• Written materials
• Physical environment

Do Not Include Metrics Like:

• "Effectively uses presentation slides"
• "Demonstrates product features clearly"
• "Uses whiteboard effectively”

 

 

Best Practices for Creating Scoring Criteria:

1. Focus on Measurable Elements

   • Use specific, countable instances (e.g., "number of open-ended questions asked")
   • Define clear verbal indicators (e.g., "uses phrases that demonstrate active listening")
   • Include examples of what constitutes successful demonstration of skills

2. Use Clear, Objective Language

   • Avoid subjective terms like "good," "effective," or "appropriate"
   • Replace with specific, observable behaviors
   • Provide concrete examples of what meets criteria

3. Structure Scoring Components

   • Break down each criterion into observable elements
   • Assign clear percentage weights
   • Include specific examples of what constitutes different score levels

4. Consider LLM Capabilities

   • Focus on text-based indicators
   • Include pattern recognition elements
   • Allow for context analysis
   • Enable comparison against best practices

5. Provide Clear Success Metrics

   • Define what constitutes different levels of performance
   • Include specific examples of good and poor performance
   • Create clear scoring rubrics

 

 

Creating Scoring Criteria for Virtual Coach Evaluation

1. Suggested Format for Each Criterion

Criterion Name: [Name] Weight: [X%] 

Behaviors: [Behavior 1]. [Behavior 2]. [Behavior 3].

Scoring: 5: [X+ instances], 4: [X-Y instances], 3: [X-Y instances], 2: [X-Y instances], 1: [0-X instances]

2. Guidelines for Behaviors

DO Write:

• Use specific, countable actions
• Focus on verbal indicators
• Include clear examples

DON'T Write:

• Subjective descriptions
• Multiple behaviors in one line
• Non-transcript behaviors

3. Examples of Strong Criteria

Example 1: Active Listening

Behaviors: Uses acknowledgment phrases ("I see," "I understand"). References client's previous statements. Asks follow-up questions based on responses.
Scoring: 5: 8+ instances 4: 6-7 instances 3: 4-5 instances 2: 2-3 instances 1: 0-1 instances

Example 2: Value Articulation

Behaviors: Links features to specific client needs. States quantifiable benefits. Uses client's business context.

Scoring: 5: 6+ value statements, 4: 4-5 value statements, 3: 2-3 value statements, 2: 1 value statement, 1: No clear statements

4. Converting Subjective to Objective

Instead of: "Shows good listening skills" Write: "Uses verbal acknowledgments"

Instead of: "Effectively handles objections" Write: "Addresses each objection with specific solution"

Instead of: "Builds good rapport" Write: "References client's statements in responses”

5. Key Reminders

• Keep behaviors countable
• Use specific numbers in scoring
• Focus only on what appears in transcript
• Avoid subjective descriptors
• Make each behavior distinct and measurable

6. Tips for Character Count

• Use concise language
• Remove unnecessary words
• Use clear abbreviations if needed
• Focus on essential elements only
• Test your criterion in the 512-character field before finalizing

7. Final Check

Your criterion should: 

✓ Fit within 512 characters 

✓ Include all required elements 

✓ Use only observable behaviors 

✓ Have clear scoring metrics 

✓ Be consistently measurable