Skip to main content

Lesson 5: Human Interaction & Summary Generation

Ready to make your AI workflows truly interactive? In this lesson, you'll learn to build workflows that seamlessly integrate human input with AI processing, creating collaborative systems that leverage both human intelligence and AI capabilities.

Learning Objectives​

By the end of this lesson, you will:

  • ✅ Integrate human interaction points into AI workflows
  • ✅ Build intelligent summary generation systems
  • ✅ Create interactive approval and feedback mechanisms
  • ✅ Design human-in-the-loop quality assurance
  • ✅ Master the HumanAgent and SummaryAgent
  • ✅ Build adaptive workflows that learn from human input

Overview: What We're Building​

We'll create an Interactive Content Review System that:

  1. Analyzes documents and extracts key information
  2. Summarizes findings for human review
  3. Requests human feedback and approval
  4. Adapts based on human input
  5. Generates final reports incorporating human insights

Step 1: Download the Interactive System​

Let's get all the files for our human-AI collaborative workflow:

Main Workflow File​

Sample Content for Review​

Custom Agent: Summary Agent​

Step 2: Understanding Human-AI Collaboration​

The HumanAgent​

The HumanAgent is a special agent type that pauses workflow execution to request human input:

RequestHumanReview,Get human feedback on analysis and summary,human,ProcessFeedback,ErrorHandler,"analysis_result,executive_summary",human_feedback,"[prompt for human]","{""timeout"": 300, ""required_fields"": [""accuracy_assessment"", ""approval_decision""]}"

Key Configuration Options:

  • timeout: Maximum wait time for human response (seconds)
  • required_fields: Fields that must be provided by human
  • optional_fields: Additional fields human can provide
  • validation_rules: Rules for validating human input

The SummaryAgent​

Our custom SummaryAgent provides intelligent summarization:

Configuration Options:

{
  "summary_type": "executive|technical|bullet_points|standard",
  "focus_areas": ["key_findings", "recommendations", "risks"],
  "max_length": 500,
  "include_metrics": true
}

Summary Types:

  • Executive: Strategic insights for leadership
  • Technical: Implementation details for developers
  • Bullet Points: Quick scan format
  • Standard: Balanced overview

Step 3: Interactive Workflow Patterns​

Pattern 1: Approval Workflow​

Pattern 2: Iterative Refinement​

Pattern 3: Quality Assurance​

Human reviewers act as quality gates:

  • Accuracy Validation: Verify AI analysis accuracy
  • Completeness Check: Ensure nothing important is missed
  • Context Addition: Provide domain expertise
  • Priority Adjustment: Rerank importance based on business needs

Step 4: Running the Interactive System​

Setup and Execution​

  1. Create Working Directory:
mkdir -p human-ai-collaboration/data
cd human-ai-collaboration

  1. Download All Files:

    • lesson5.csv (interactive workflow)
    • review_content.txt (sample business review)
    • summary_agent.py (custom summary agent)

  2. Set Environment:

export ANTHROPIC_API_KEY="your-api-key"
  1. Run Interactive Workflow:
agentmap run lesson5.csv

What to Expect​

The workflow will:

  1. Analyze the business review content with AI
  2. Generate an executive summary
  3. Pause and ask for your feedback
  4. Wait for your input on accuracy and approval decision
  5. Adapt based on your feedback
  6. Generate final report incorporating your insights

Sample Human Interaction​

When the workflow pauses, you'll see:

Please review the AI analysis and summary below:

## 🤖 AI Analysis Results
[Comprehensive analysis of the business review]

## 📋 Executive Summary
[Generated executive summary]

**Please provide your feedback:**

1. **Accuracy Assessment** (1-10): How accurate is the AI analysis?
2. **Completeness Review**: What important points did the AI miss?
3. **Priority Ranking**: Which findings are most/least important?
4. **Action Items**: What specific actions should be taken?
5. **Additional Context**: What context should be added?
6. **Approval Decision**: APPROVE/REVISE/REJECT

Enter your response:

Sample Human Response​

Accuracy Assessment: 8
The AI correctly identified most key metrics and trends.

Completeness Review: 
- Missing analysis of competitive threats from the new $50M funded startup
- Should emphasize the critical nature of the hiring plan
- Infrastructure costs scaling faster than revenue is a bigger concern

Priority Ranking:
- HIGHEST: Operational scaling challenges (affects customer satisfaction)
- HIGH: Infrastructure limitations (affects all users) 
- MEDIUM: Competitive pressure (manageable with differentiation)

Action Items:
1. Emergency hiring committee to accelerate the 12-person hiring plan
2. Immediate infrastructure assessment and upgrade timeline
3. Customer communication plan about improvements underway

Additional Context:
This is our first quarter showing operational strain. Previous quarters managed growth smoothly. The 18-hour support response time is already affecting customer satisfaction scores.

Approval Decision: REVISE
Need to emphasize operational urgency and adjust priority ranking

Step 5: Advanced Human-AI Patterns​

Dynamic Routing Based on Human Input​

def route_based_on_human_feedback(feedback):
    decision = feedback.get('approval_decision', 'REVISE')
    
    if decision == 'APPROVE':
        return 'GenerateFinalReport'
    elif decision == 'REVISE':
        return 'EnhancedAnalysis'
    elif decision == 'REJECT':
        return 'AlternativeAnalysis'
    else:
        return 'RequestClarification'

Confidence-Adjusted Processing​

def adjust_processing_based_on_confidence(ai_confidence, human_confidence):
    """Adjust workflow based on confidence levels."""
    
    if ai_confidence > 0.9 and human_confidence > 8:
        return 'fast_track'  # Both AI and human are confident
    elif ai_confidence < 0.6 or human_confidence < 5:
        return 'detailed_review'  # Low confidence, need thorough review
    else:
        return 'standard_review'  # Normal processing

Learning from Human Feedback​

class LearningWorkflow:
    def __init__(self):
        self.feedback_history = []
    
    def process_human_feedback(self, feedback):
        # Store feedback for learning
        self.feedback_history.append({
            'timestamp': datetime.now(),
            'feedback': feedback,
            'ai_confidence': self.last_ai_confidence,
            'outcome': 'success' if feedback.get('approval_decision') == 'APPROVE' else 'revision'
        })
        
        # Adjust future processing based on patterns
        self.update_processing_parameters()
    
    def update_processing_parameters(self):
        """Update AI processing based on human feedback patterns."""
        recent_feedback = self.feedback_history[-10:]  # Last 10 interactions
        
        # Analyze patterns
        common_missing_areas = self.analyze_missed_areas(recent_feedback)
        common_priority_adjustments = self.analyze_priority_patterns(recent_feedback)
        
        # Update processing focus
        self.adjust_analysis_focus(common_missing_areas)
        self.adjust_priority_weights(common_priority_adjustments)

Step 6: Building Custom Interactive Agents​

Exercise 1: Create Validation Agent​

Build an agent that validates AI output before human review:

class ValidationAgent(BaseAgent):
    def process(self, inputs):
        ai_analysis = inputs.get('analysis_result', {})
        
        validation_results = {
            'completeness_score': self._check_completeness(ai_analysis),
            'consistency_score': self._check_consistency(ai_analysis),
            'confidence_assessment': self._assess_confidence(ai_analysis),
            'flagged_areas': self._identify_concerns(ai_analysis),
            'validation_passed': False
        }
        
        # Overall validation decision
        if (validation_results['completeness_score'] > 0.7 and 
            validation_results['consistency_score'] > 0.8 and
            len(validation_results['flagged_areas']) < 3):
            validation_results['validation_passed'] = True
        
        return validation_results

Exercise 2: Implement Feedback Learning​

Create a system that learns from human feedback:

class FeedbackLearningAgent(BaseAgent):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.feedback_db = {}  # In practice, use proper database
        
    def process(self, inputs):
        human_feedback = inputs.get('human_feedback', {})
        ai_analysis = inputs.get('analysis_result', {})
        
        # Analyze feedback patterns
        feedback_insights = {
            'accuracy_trend': self._calculate_accuracy_trend(),
            'common_gaps': self._identify_common_gaps(),
            'priority_misalignments': self._find_priority_issues(),
            'improvement_suggestions': self._generate_improvements()
        }
        
        # Store for future learning
        self._store_feedback_data(human_feedback, ai_analysis)
        
        return feedback_insights
    
    def _calculate_accuracy_trend(self):
        """Calculate accuracy improvement over time."""
        recent_scores = [feedback.get('accuracy_assessment', 5) 
                        for feedback in self.feedback_db.values()]
        
        if len(recent_scores) >= 2:
            return {
                'current_average': sum(recent_scores) / len(recent_scores),
                'trend': 'improving' if recent_scores[-1] > recent_scores[-2] else 'declining'
            }
        return {'current_average': 5, 'trend': 'insufficient_data'}

Exercise 3: Multi-Human Consensus​

Build a workflow that gets input from multiple humans:

workflow,node,description,type,next_node,error_node,input_fields,output_field,prompt,context
MultiReview,GetReviewer1,Get feedback from first reviewer,human,GetReviewer2,Error,analysis,reviewer1_feedback,"Review as Domain Expert...","{""timeout"": 300}"
MultiReview,GetReviewer2,Get feedback from second reviewer,human,GetReviewer3,Error,analysis,reviewer2_feedback,"Review as Business Stakeholder...","{""timeout"": 300}"
MultiReview,GetReviewer3,Get feedback from third reviewer,human,BuildConsensus,Error,analysis,reviewer3_feedback,"Review as Technical Lead...","{""timeout"": 300}"
MultiReview,BuildConsensus,Build consensus from multiple reviews,consensus_builder,FinalDecision,Error,"reviewer1_feedback,reviewer2_feedback,reviewer3_feedback",consensus_result,,

Step 7: Best Practices for Human-AI Collaboration​

Design Principles​

1. Clear Human Instructions​

  • Provide specific questions rather than open-ended requests
  • Use structured input formats for consistency
  • Include examples of good feedback

2. Appropriate Timeout Settings​

  • Complex reviews: 10-30 minutes (600-1800 seconds)
  • Simple approvals: 2-5 minutes (120-300 seconds)
  • Emergency decisions: 30-60 seconds

3. Graceful Fallbacks​

  • Default actions when humans don't respond
  • Alternative workflows for different response types
  • Clear error messages and recovery options

4. Context Preservation​

  • Maintain conversation history
  • Show relevant previous decisions
  • Preserve human preferences across sessions

Common Pitfalls and Solutions​

Pitfall 1: Information Overload​

Problem: Presenting too much information to human reviewers Solution: Use progressive disclosure and focused summaries

Pitfall 2: Unclear Decision Points​

Problem: Humans unsure what decision is needed Solution: Provide clear decision frameworks and examples

Pitfall 3: Lost Context​

Problem: Humans lose track of workflow state Solution: Include workflow progress indicators and relevant history

Pitfall 4: Inconsistent Feedback​

Problem: Different humans provide conflicting guidance Solution: Implement consensus mechanisms and feedback validation

Performance Optimization​

1. Async Human Input​

async def handle_human_input_async(workflow_id, prompt, timeout=300):
    """Handle human input asynchronously."""
    
    # Store prompt and create notification
    await store_human_request(workflow_id, prompt)
    await notify_human_reviewer(workflow_id)
    
    # Wait for response with timeout
    response = await wait_for_human_response(workflow_id, timeout)
    
    if response:
        return response
    else:
        return await handle_timeout(workflow_id)

2. Intelligent Batching​

def batch_human_requests(pending_requests):
    """Batch multiple requests for efficiency."""
    
    # Group by reviewer type
    batched = {}
    for request in pending_requests:
        reviewer_type = request.get('reviewer_type', 'general')
        if reviewer_type not in batched:
            batched[reviewer_type] = []
        batched[reviewer_type].append(request)
    
    return batched

3. Smart Routing​

def route_to_appropriate_human(task_complexity, domain, urgency):
    """Route tasks to humans based on expertise and availability."""
    
    if urgency == 'HIGH':
        return get_available_expert(domain)
    elif task_complexity > 0.8:
        return get_senior_reviewer(domain)
    else:
        return get_available_reviewer()

Key Concepts Mastered​

1. Human-AI Collaboration Patterns​

  • Interactive approval workflows
  • Iterative refinement processes
  • Quality assurance integration
  • Consensus building mechanisms

2. Smart Summary Generation​

  • Configurable summary types and lengths
  • Focus area customization
  • Content metrics analysis
  • Adaptive compression ratios

3. Workflow Adaptation​

  • Dynamic routing based on human input
  • Confidence-adjusted processing
  • Learning from feedback patterns
  • Context-aware decision making

4. Interaction Design​

  • Clear human instruction patterns
  • Structured feedback collection
  • Timeout and fallback handling
  • Progress indication and context preservation

Troubleshooting Human-AI Workflows​

Common Issues​

Issue: Human input timeouts​

Symptoms: Workflows hanging waiting for human response Solutions:

  • Set reasonable timeout values
  • Implement automatic fallback actions
  • Send reminders before timeout
  • Provide easy response interfaces

Issue: Inconsistent human feedback​

Symptoms: Different humans giving conflicting guidance Solutions:

  • Provide clear feedback frameworks
  • Use structured input formats
  • Implement consensus mechanisms
  • Train reviewers on consistent standards

Issue: Low human engagement​

Symptoms: Humans not providing detailed feedback Solutions:

  • Make feedback requests specific and actionable
  • Show impact of human input on final results
  • Provide easy-to-use interfaces
  • Recognize valuable human contributions

Congratulations!​

You've mastered human-AI collaboration! You can now build workflows that:

  • ✅ Integrate Human Expertise - Seamlessly incorporate human knowledge
  • ✅ Generate Intelligent Summaries - Create focused, actionable summaries
  • ✅ Adapt Based on Feedback - Learn and improve from human input
  • ✅ Handle Complex Interactions - Manage approval, revision, and consensus processes

What's Next?​

Ready for more advanced topics? Let's explore:

🤝 Human-AI collaboration represents the future of intelligent automation - combining the analytical power of AI with human wisdom, creativity, and domain expertise. You're now ready to build systems that are truly collaborative!