Higher-Order Functions and Collection Operations

Introduction

Higher-order functions are the backbone of functional programming in Scala. They allow you to write concise, expressive code by treating functions as first-class values. Scala's collections library is built around higher-order functions, providing powerful tools for data transformation, filtering, and aggregation.

In this lesson, you'll master the essential collection operations that make Scala programming so elegant and productive. You'll learn how to chain operations together to create data processing pipelines that are both readable and efficient.

Core Collection Operations

Map: Transforming Elements

The map operation applies a function to every element in a collection, creating a new collection with the transformed elements.

// Basic map operations
val numbers = List(1, 2, 3, 4, 5)

val doubled = numbers.map(_ * 2)
val squared = numbers.map(x => x * x)
val stringified = numbers.map(_.toString)

println(doubled)      // List(2, 4, 6, 8, 10)
println(squared)      // List(1, 4, 9, 16, 25)
println(stringified)  // List(1, 2, 3, 4, 5)

// Map with more complex transformations
case class Person(name: String, age: Int)

val people = List(
  Person("Alice", 25),
  Person("Bob", 30),
  Person("Charlie", 35)
)

val names = people.map(_.name)
val ages = people.map(_.age)
val greetings = people.map(p => s"Hello, ${p.name}!")
val ageGroups = people.map { person =>
  if (person.age < 30) "Young"
  else if (person.age < 50) "Middle-aged"
  else "Senior"
}

println(names)      // List(Alice, Bob, Charlie)
println(ages)       // List(25, 30, 35)
println(greetings)  // List(Hello, Alice!, Hello, Bob!, Hello, Charlie!)
println(ageGroups)  // List(Young, Middle-aged, Middle-aged)

// Map works with different collection types
val array = Array(1, 2, 3, 4, 5)
val vector = Vector(1, 2, 3, 4, 5)
val set = Set(1, 2, 3, 4, 5)

println(array.map(_ * 3).toList)   // List(3, 6, 9, 12, 15)
println(vector.map(_ + 10))        // Vector(11, 12, 13, 14, 15)
println(set.map(_.toString))       // Set(1, 2, 3, 4, 5) as strings

// Chaining map operations
val result = numbers
  .map(_ + 1)        // Add 1: List(2, 3, 4, 5, 6)
  .map(_ * 2)        // Multiply by 2: List(4, 6, 8, 10, 12)
  .map(x => s"[$x]") // Format: List([4], [6], [8], [10], [12])

println(result)

// Map with Option
val maybeNumbers = List(Some(1), None, Some(3), Some(4), None)
val incrementedOptions = maybeNumbers.map(_.map(_ + 1))
println(incrementedOptions)  // List(Some(2), None, Some(4), Some(5), None)

// Working with nested structures
val words = List("hello", "world", "scala")
val characters = words.map(_.toList)
val upperCaseChars = words.map(_.toList.map(_.toUpper))

println(characters)     // List(List(h, e, l, l, o), List(w, o, r, l, d), List(s, c, a, l, a))
println(upperCaseChars) // List(List(H, E, L, L, O), List(W, O, R, L, D), List(S, C, A, L, A))

Filter: Selecting Elements

The filter operation selects elements that satisfy a given predicate (boolean function).

val numbers = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

// Basic filtering
val evens = numbers.filter(_ % 2 == 0)
val odds = numbers.filter(_ % 2 != 0)
val greaterThanFive = numbers.filter(_ > 5)
val lessThanOrEqualThree = numbers.filter(_ <= 3)

println(evens)                 // List(2, 4, 6, 8, 10)
println(odds)                  // List(1, 3, 5, 7, 9)
println(greaterThanFive)       // List(6, 7, 8, 9, 10)
println(lessThanOrEqualThree)  // List(1, 2, 3)

// Filtering with complex predicates
val words = List("apple", "banana", "cherry", "date", "elderberry", "fig")

val longWords = words.filter(_.length > 5)
val startsWithVowel = words.filter(word => "aeiou".contains(word.head.toLower))
val containsA = words.filter(_.contains('a'))
val shortWordsWithE = words.filter(word => word.length <= 4 && word.contains('e'))

println(longWords)        // List(banana, cherry, elderberry)
println(startsWithVowel)  // List(apple, elderberry)
println(containsA)        // List(apple, banana, date)
println(shortWordsWithE)  // List(date)

// Filtering objects
case class Product(name: String, price: Double, category: String, inStock: Boolean)

val products = List(
  Product("Laptop", 999.99, "Electronics", true),
  Product("Coffee", 12.99, "Food", true),
  Product("Book", 29.99, "Education", false),
  Product("Phone", 699.99, "Electronics", true),
  Product("Tea", 8.99, "Food", false)
)

val inStockProducts = products.filter(_.inStock)
val expensiveProducts = products.filter(_.price > 50)
val electronics = products.filter(_.category == "Electronics")
val affordableInStock = products.filter(p => p.price < 100 && p.inStock)

println("In stock:")
inStockProducts.foreach(p => println(s"  ${p.name} - $${p.price}"))

println("Expensive (>$50):")
expensiveProducts.foreach(p => println(s"  ${p.name} - $${p.price}"))

println("Electronics:")
electronics.foreach(p => println(s"  ${p.name} - $${p.price}"))

println("Affordable and in stock:")
affordableInStock.foreach(p => println(s"  ${p.name} - $${p.price}"))

// Combining filter with other operations
val processedNumbers = numbers
  .filter(_ % 2 == 0)     // Keep even numbers: List(2, 4, 6, 8, 10)
  .filter(_ > 4)          // Keep those > 4: List(6, 8, 10)
  .map(_ * 3)             // Multiply by 3: List(18, 24, 30)

println(processedNumbers)

// FilterNot - opposite of filter
val notEvens = numbers.filterNot(_ % 2 == 0)  // Same as filter(_ % 2 != 0)
val notExpensive = products.filterNot(_.price > 100)

println(notEvens)  // List(1, 3, 5, 7, 9)
println(s"Not expensive: ${notExpensive.map(_.name)}")

FlatMap: Transforming and Flattening

FlatMap combines map and flatten operations. It's particularly useful when working with nested collections or when a transformation function returns a collection.

// Basic flatMap with collections
val words = List("hello", "world", "scala")
val characters = words.flatMap(_.toList)
println(characters)  // List(h, e, l, l, o, w, o, r, l, d, s, c, a, l, a)

// Compare with map (creates nested structure)
val charactersNested = words.map(_.toList)
println(charactersNested)  // List(List(h, e, l, l, o), List(w, o, r, l, d), List(s, c, a, l, a))

// FlatMap with numbers
val numbers = List(1, 2, 3, 4)
val expanded = numbers.flatMap(n => List(n, n * 10))
println(expanded)  // List(1, 10, 2, 20, 3, 30, 4, 40)

// More complex flatMap examples
val sentences = List("Hello world", "Scala is great", "Functional programming rocks")
val allWords = sentences.flatMap(_.split(" "))
println(allWords.toList)  // List(Hello, world, Scala, is, great, Functional, programming, rocks)

// FlatMap with ranges
val ranges = List(1 to 3, 4 to 6, 7 to 9)
val flatNumbers = ranges.flatMap(_.toList)
println(flatNumbers)  // List(1, 2, 3, 4, 5, 6, 7, 8, 9)

// FlatMap with Option - filtering out None values
val maybeNumbers = List(Some(1), None, Some(3), Some(4), None, Some(6))
val actualNumbers = maybeNumbers.flatMap(_.toList)  // Converts Some(x) to List(x), None to List()
println(actualNumbers)  // List(1, 3, 4, 6)

// Alternative using flatten
val onlyDefinedNumbers = maybeNumbers.flatten
println(onlyDefinedNumbers)  // List(1, 3, 4, 6)

// Practical example: Processing file paths
case class Directory(name: String, files: List[String])

val directories = List(
  Directory("src", List("Main.scala", "Utils.scala", "Config.scala")),
  Directory("test", List("MainTest.scala", "UtilsTest.scala")),
  Directory("docs", List("README.md", "CHANGELOG.md"))
)

val allFiles = directories.flatMap(_.files)
val scalaFiles = directories.flatMap(_.files.filter(_.endsWith(".scala")))
val withPaths = directories.flatMap(dir => 
  dir.files.map(file => s"${dir.name}/$file")
)

println(allFiles)   // All files
println(scalaFiles) // Only .scala files
println(withPaths)  // Files with directory paths

// FlatMap for error handling patterns
def safeDivide(x: Double, y: Double): Option[Double] = {
  if (y != 0) Some(x / y) else None
}

val numerators = List(10.0, 20.0, 30.0)
val denominators = List(2.0, 0.0, 5.0)

val divisions = numerators.zip(denominators).flatMap { case (num, den) =>
  safeDivide(num, den)
}
println(divisions)  // List(5.0, 6.0) - skips the division by zero

// Advanced flatMap with multiple collections
val colors = List("red", "green", "blue")
val sizes = List("small", "large")

val combinations = colors.flatMap { color =>
  sizes.map { size =>
    s"$size $color"
  }
}
println(combinations)  // List(small red, large red, small green, large green, small blue, large blue)

// Using for-comprehension (syntactic sugar for flatMap/map)
val combinationsFor = for {
  color <- colors
  size <- sizes
} yield s"$size $color"

println(combinationsFor)  // Same result as above

Fold and Reduce: Aggregating Elements

Fold and reduce operations combine all elements in a collection into a single value using a binary operation.

val numbers = List(1, 2, 3, 4, 5)

// Reduce operations
val sum = numbers.reduce(_ + _)
val product = numbers.reduce(_ * _)
val maximum = numbers.reduce(math.max)
val minimum = numbers.reduce(math.min)

println(s"Sum: $sum")        // Sum: 15
println(s"Product: $product") // Product: 120
println(s"Max: $maximum")     // Max: 5
println(s"Min: $minimum")     // Min: 1

// Fold operations with initial values
val foldSum = numbers.fold(0)(_ + _)       // Same as reduce for sum
val foldProduct = numbers.fold(1)(_ * _)   // Same as reduce for product
val sumWith100 = numbers.fold(100)(_ + _)  // Start with 100

println(s"Fold sum: $foldSum")           // Fold sum: 15
println(s"Fold product: $foldProduct")   // Fold product: 120
println(s"Sum + 100: $sumWith100")       // Sum + 100: 115

// FoldLeft and foldRight for order-dependent operations
val words = List("Hello", "World", "Scala")

val concatenatedLeft = words.foldLeft("")(_ + " " + _).trim
val concatenatedRight = words.foldRight("")(_ + " " + _).trim

println(s"Left fold: '$concatenatedLeft'")   // Left fold: 'Hello World Scala'
println(s"Right fold: '$concatenatedRight'") // Right fold: 'Hello World Scala'

// Better example showing the difference
val letters = List("A", "B", "C", "D")

val leftToRight = letters.foldLeft("Start")((acc, elem) => s"($acc + $elem)")
val rightToLeft = letters.foldRight("End")((elem, acc) => s"($elem + $acc)")

println(s"Left: $leftToRight")   // Left: ((((Start + A) + B) + C) + D)
println(s"Right: $rightToLeft")  // Right: (A + (B + (C + (D + End))))

// Practical examples
case class Transaction(amount: Double, transactionType: String)

val transactions = List(
  Transaction(100.0, "deposit"),
  Transaction(-50.0, "withdrawal"),
  Transaction(75.0, "deposit"),
  Transaction(-25.0, "withdrawal"),
  Transaction(200.0, "deposit")
)

val finalBalance = transactions.foldLeft(0.0)((balance, transaction) => 
  balance + transaction.amount
)

val depositTotal = transactions
  .filter(_.transactionType == "deposit")
  .foldLeft(0.0)((sum, transaction) => sum + transaction.amount)

val withdrawalTotal = transactions
  .filter(_.transactionType == "withdrawal")
  .foldLeft(0.0)((sum, transaction) => sum + transaction.amount)

println(f"Final balance: $$${finalBalance}%.2f")      // Final balance: $300.00
println(f"Total deposits: $$${depositTotal}%.2f")    // Total deposits: $375.00
println(f"Total withdrawals: $$${withdrawalTotal}%.2f") // Total withdrawals: $-75.00

// Working with complex data structures
case class Student(name: String, grades: List[Int])

val students = List(
  Student("Alice", List(85, 92, 78, 95)),
  Student("Bob", List(78, 85, 88, 82)),
  Student("Charlie", List(92, 95, 98, 90))
)

// Calculate class average
val allGrades = students.flatMap(_.grades)
val classAverage = allGrades.foldLeft(0.0)(_ + _) / allGrades.length

// Find student averages
val studentAverages = students.map { student =>
  val average = student.grades.foldLeft(0.0)(_ + _) / student.grades.length
  (student.name, average)
}

// Count grades by letter grade
val letterGradeCounts = allGrades.foldLeft(Map[String, Int]()) { (acc, grade) =>
  val letterGrade = grade match {
    case g if g >= 90 => "A"
    case g if g >= 80 => "B"
    case g if g >= 70 => "C"
    case g if g >= 60 => "D"
    case _ => "F"
  }
  acc + (letterGrade -> (acc.getOrElse(letterGrade, 0) + 1))
}

println(f"Class average: ${classAverage}%.1f")
println("Student averages:")
studentAverages.foreach { case (name, avg) => 
  println(f"  $name: ${avg}%.1f")
}
println("Letter grade distribution:")
letterGradeCounts.foreach { case (grade, count) =>
  println(s"  $grade: $count")
}

// Scan operations - like fold but keeps intermediate results
val scanSum = numbers.scanLeft(0)(_ + _)
val scanProduct = numbers.scanLeft(1)(_ * _)

println(s"Scan sum: $scanSum")       // List(0, 1, 3, 6, 10, 15)
println(s"Scan product: $scanProduct") // List(1, 1, 2, 6, 24, 120)

// Running totals example
val dailySales = List(100, 150, 80, 200, 120)
val runningTotals = dailySales.scanLeft(0)(_ + _).tail  // Remove initial 0

println(s"Daily sales: $dailySales")
println(s"Running totals: $runningTotals")  // List(100, 250, 330, 530, 650)

Advanced Collection Operations

GroupBy: Organizing Data

// Basic groupBy examples
val words = List("apple", "banana", "cherry", "apricot", "blueberry", "avocado")

val groupedByFirstLetter = words.groupBy(_.head)
val groupedByLength = words.groupBy(_.length)

println("Grouped by first letter:")
groupedByFirstLetter.foreach { case (letter, wordList) =>
  println(s"  $letter: ${wordList.mkString(", ")}")
}

println("Grouped by length:")
groupedByLength.toList.sortBy(_._1).foreach { case (length, wordList) =>
  println(s"  $length: ${wordList.mkString(", ")}")
}

// Practical example with objects
case class Employee(name: String, department: String, salary: Double, level: String)

val employees = List(
  Employee("Alice", "Engineering", 75000, "Senior"),
  Employee("Bob", "Engineering", 65000, "Mid"),
  Employee("Charlie", "Marketing", 60000, "Mid"),
  Employee("David", "Engineering", 85000, "Senior"),
  Employee("Eve", "Marketing", 55000, "Junior"),
  Employee("Frank", "Sales", 70000, "Senior"),
  Employee("Grace", "Sales", 45000, "Junior")
)

val byDepartment = employees.groupBy(_.department)
val byLevel = employees.groupBy(_.level)
val bySalaryRange = employees.groupBy { emp =>
  emp.salary match {
    case s if s < 50000 => "Low"
    case s if s < 70000 => "Medium"
    case _ => "High"
  }
}

println("By Department:")
byDepartment.foreach { case (dept, empList) =>
  val avgSalary = empList.map(_.salary).sum / empList.length
  println(f"  $dept: ${empList.length} employees, avg salary: $$${avgSalary}%.0f")
}

println("By Level:")
byLevel.foreach { case (level, empList) =>
  val depts = empList.map(_.department).distinct
  println(s"  $level: ${empList.length} employees across departments: ${depts.mkString(", ")}")
}

// Complex grouping with multiple criteria
val departmentLevelGroups = employees.groupBy(emp => (emp.department, emp.level))

println("By Department and Level:")
departmentLevelGroups.toList.sortBy(_._1).foreach { case ((dept, level), empList) =>
  val names = empList.map(_.name)
  println(s"  $dept $level: ${names.mkString(", ")}")
}

// GroupBy with aggregations
val departmentStats = employees
  .groupBy(_.department)
  .view.mapValues { empList =>
    val salaries = empList.map(_.salary)
    (
      empList.length,           // count
      salaries.sum,             // total salary
      salaries.sum / empList.length, // average
      salaries.min,             // minimum
      salaries.max              // maximum
    )
  }.toMap

println("Department Statistics:")
departmentStats.foreach { case (dept, (count, total, avg, min, max)) =>
  println(f"  $dept:")
  println(f"    Employees: $count")
  println(f"    Total Salary: $$${total}%.0f")
  println(f"    Average: $$${avg}%.0f")
  println(f"    Range: $$${min}%.0f - $$${max}%.0f")
}

Partition and Span: Splitting Collections

val numbers = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

// Partition splits into two collections based on a predicate
val (evens, odds) = numbers.partition(_ % 2 == 0)
println(s"Evens: $evens")  // List(2, 4, 6, 8, 10)
println(s"Odds: $odds")    // List(1, 3, 5, 7, 9)

// Span splits at the first element that doesn't match the predicate
val (lessThanFive, fiveAndUp) = numbers.span(_ < 5)
println(s"Less than 5: $lessThanFive")  // List(1, 2, 3, 4)
println(s"5 and up: $fiveAndUp")        // List(5, 6, 7, 8, 9, 10)

// More examples with strings
val sentence = "The quick brown fox jumps over the lazy dog"
val words = sentence.split(" ").toList

val (shortWords, longWords) = words.partition(_.length <= 4)
val (beforeFox, fromFox) = words.span(_ != "fox")

println(s"Short words: ${shortWords.mkString(", ")}")
println(s"Long words: ${longWords.mkString(", ")}")
println(s"Before fox: ${beforeFox.mkString(" ")}")
println(s"From fox: ${fromFox.mkString(" ")}")

// Practical example
case class Task(name: String, priority: Int, completed: Boolean)

val tasks = List(
  Task("Write report", 1, false),
  Task("Reply emails", 2, true),
  Task("Code review", 1, false),
  Task("Team meeting", 3, true),
  Task("Fix bug", 1, false),
  Task("Update docs", 2, false)
)

val (completed, pending) = tasks.partition(_.completed)
val (highPriority, otherPriority) = pending.partition(_.priority == 1)

println("Completed tasks:")
completed.foreach(task => println(s"  ${task.name}"))

println("High priority pending tasks:")
highPriority.foreach(task => println(s"  ${task.name}"))

println("Other pending tasks:")
otherPriority.foreach(task => println(s"  ${task.name} (priority ${task.priority})"))

Zip and Unzip: Combining Collections

val numbers = List(1, 2, 3, 4, 5)
val letters = List("a", "b", "c", "d", "e")
val booleans = List(true, false, true, false, true)

// Basic zip operations
val numbersAndLetters = numbers.zip(letters)
val numbersAndBooleans = numbers.zip(booleans)

println(numbersAndLetters)  // List((1,a), (2,b), (3,c), (4,d), (5,e))
println(numbersAndBooleans) // List((1,true), (2,false), (3,true), (4,false), (5,true))

// Zip with different lengths (shorter collection determines result length)
val shortList = List(1, 2, 3)
val longList = List("a", "b", "c", "d", "e", "f")
val zipped = shortList.zip(longList)
println(zipped)  // List((1,a), (2,b), (3,c))

// Zip with index
val withIndex = letters.zipWithIndex
println(withIndex)  // List((a,0), (b,1), (c,2), (d,3), (e,4))

// Triple zip using multiple zip operations
val triple = numbers.zip(letters).zip(booleans).map { case ((num, letter), bool) =>
  (num, letter, bool)
}
println(triple)  // List((1,a,true), (2,b,false), (3,c,true), (4,d,false), (5,e,true))

// Unzip operations
val pairs = List((1, "one"), (2, "two"), (3, "three"))
val (nums, words) = pairs.unzip
println(s"Numbers: $nums")  // List(1, 2, 3)
println(s"Words: $words")   // List(one, two, three)

// Practical examples
case class Student(name: String, id: Int)
case class Grade(studentId: Int, subject: String, score: Int)

val students = List(
  Student("Alice", 1),
  Student("Bob", 2),
  Student("Charlie", 3)
)

val grades = List(
  Grade(1, "Math", 85),
  Grade(2, "Math", 78),
  Grade(3, "Math", 92)
)

// Create student-grade pairs
val studentGradePairs = students.zip(grades).map { case (student, grade) =>
  (student.name, grade.score)
}

println("Student grades:")
studentGradePairs.foreach { case (name, score) =>
  println(s"  $name: $score")
}

// Working with parallel arrays (not recommended, but sometimes necessary)
val productNames = List("Laptop", "Mouse", "Keyboard")
val productPrices = List(999.99, 29.99, 79.99)
val productStock = List(5, 50, 25)

val products = productNames.zip(productPrices).zip(productStock).map {
  case ((name, price), stock) => (name, price, stock)
}

println("Product inventory:")
products.foreach { case (name, price, stock) =>
  println(f"  $name: $$${price}%.2f (${stock} in stock)")
}

// ZipAll for collections of different lengths
val list1 = List(1, 2, 3)
val list2 = List("a", "b", "c", "d", "e")

val zippedAll = list1.zipAll(list2, 0, "?")
println(zippedAll)  // List((1,a), (2,b), (3,c), (0,d), (0,e))

Complex Data Processing Pipelines

Real-World Example: E-commerce Analytics

case class Order(id: Int, customerId: Int, products: List[OrderItem], date: String, status: String)
case class OrderItem(productId: Int, name: String, category: String, price: Double, quantity: Int)
case class Customer(id: Int, name: String, segment: String, country: String)

// Sample data
val customers = List(
  Customer(1, "Alice", "Premium", "USA"),
  Customer(2, "Bob", "Standard", "Canada"),
  Customer(3, "Charlie", "Premium", "USA"),
  Customer(4, "Diana", "Standard", "UK"),
  Customer(5, "Eve", "Premium", "Canada")
)

val orders = List(
  Order(1, 1, List(
    OrderItem(101, "Laptop", "Electronics", 999.99, 1),
    OrderItem(102, "Mouse", "Electronics", 29.99, 2)
  ), "2024-01-15", "Completed"),

  Order(2, 2, List(
    OrderItem(103, "Book", "Education", 19.99, 3),
    OrderItem(104, "Pen", "Office", 4.99, 5)
  ), "2024-01-16", "Completed"),

  Order(3, 3, List(
    OrderItem(101, "Laptop", "Electronics", 999.99, 1),
    OrderItem(105, "Keyboard", "Electronics", 79.99, 1)
  ), "2024-01-17", "Shipped"),

  Order(4, 1, List(
    OrderItem(106, "Coffee", "Food", 12.99, 2)
  ), "2024-01-18", "Completed"),

  Order(5, 4, List(
    OrderItem(102, "Mouse", "Electronics", 29.99, 1),
    OrderItem(107, "Notebook", "Office", 9.99, 3)
  ), "2024-01-19", "Processing")
)

// Complex analytics pipeline
def analyzeOrders(orders: List[Order], customers: List[Customer]) = {
  // Create customer lookup map
  val customerMap = customers.map(c => c.id -> c).toMap

  // Calculate order totals
  val orderTotals = orders.map { order =>
    val total = order.products.map(item => item.price * item.quantity).sum
    (order, total)
  }

  // Revenue by customer segment
  val revenueBySegment = orderTotals
    .filter(_._1.status == "Completed")  // Only completed orders
    .flatMap { case (order, total) =>
      customerMap.get(order.customerId).map { customer =>
        (customer.segment, total)
      }
    }
    .groupBy(_._1)
    .view.mapValues(_.map(_._2).sum)
    .toMap

  // Top products by revenue
  val productRevenue = orders
    .filter(_.status == "Completed")
    .flatMap(_.products)
    .groupBy(_.name)
    .view.mapValues(items => items.map(item => item.price * item.quantity).sum)
    .toList
    .sortBy(-_._2)
    .take(5)

  // Customer spending analysis
  val customerSpending = orderTotals
    .filter(_._1.status == "Completed")
    .groupBy(_._1.customerId)
    .view.mapValues(_.map(_._2).sum)
    .toList
    .flatMap { case (customerId, totalSpent) =>
      customerMap.get(customerId).map { customer =>
        (customer.name, customer.segment, totalSpent)
      }
    }
    .sortBy(-_._3)

  // Category performance
  val categoryStats = orders
    .filter(_.status == "Completed")
    .flatMap(_.products)
    .groupBy(_.category)
    .view.mapValues { items =>
      val revenue = items.map(item => item.price * item.quantity).sum
      val quantity = items.map(_.quantity).sum
      val avgPrice = items.map(_.price).sum / items.length
      (revenue, quantity, avgPrice)
    }
    .toMap

  (revenueBySegment, productRevenue, customerSpending, categoryStats)
}

val (revenueBySegment, productRevenue, customerSpending, categoryStats) = 
  analyzeOrders(orders, customers)

println("Revenue by Customer Segment:")
revenueBySegment.foreach { case (segment, revenue) =>
  println(f"  $segment: $$${revenue}%.2f")
}

println("\nTop Products by Revenue:")
productRevenue.foreach { case (product, revenue) =>
  println(f"  $product: $$${revenue}%.2f")
}

println("\nCustomer Spending (Top to Bottom):")
customerSpending.foreach { case (name, segment, spending) =>
  println(f"  $name ($segment): $$${spending}%.2f")
}

println("\nCategory Performance:")
categoryStats.foreach { case (category, (revenue, quantity, avgPrice)) =>
  println(f"  $category:")
  println(f"    Revenue: $$${revenue}%.2f")
  println(f"    Quantity Sold: $quantity")
  println(f"    Average Price: $$${avgPrice}%.2f")
}

// Advanced filtering and transformations
val premiumCustomerOrders = orders
  .filter { order =>
    customers.find(_.id == order.customerId).exists(_.segment == "Premium")
  }
  .filter(_.status == "Completed")
  .map { order =>
    val customer = customers.find(_.id == order.customerId).get
    val total = order.products.map(item => item.price * item.quantity).sum
    (customer.name, order.id, total)
  }
  .sortBy(-_._3)

println("\nPremium Customer Completed Orders:")
premiumCustomerOrders.foreach { case (name, orderId, total) =>
  println(f"  $name (Order #$orderId): $$${total}%.2f")
}

// Multi-step transformation pipeline
val electronicsPurchases = orders
  .flatMap(_.products)                           // Flatten all products
  .filter(_.category == "Electronics")          // Electronics only
  .groupBy(_.name)                              // Group by product name
  .view.mapValues { items =>                     // Calculate stats per product
    val totalQuantity = items.map(_.quantity).sum
    val totalRevenue = items.map(item => item.price * item.quantity).sum
    val avgPrice = totalRevenue / totalQuantity
    (totalQuantity, totalRevenue, avgPrice)
  }
  .toList
  .sortBy(-_._2._2)                             // Sort by revenue descending

println("\nElectronics Products Performance:")
electronicsPurchases.foreach { case (product, (qty, revenue, avgPrice)) =>
  println(f"  $product:")
  println(f"    Quantity: $qty")
  println(f"    Revenue: $$${revenue}%.2f")
  println(f"    Avg Price: $$${avgPrice}%.2f")
}

Performance Considerations

Lazy vs Eager Evaluation

// View creates lazy collections for chained operations
val largeList = (1 to 1000000).toList

// Eager evaluation - each operation creates intermediate collections
val eagerResult = largeList
  .map(_ * 2)
  .filter(_ % 3 == 0)
  .map(_ + 1)
  .take(10)

// Lazy evaluation - operations are fused together
val lazyResult = largeList.view
  .map(_ * 2)
  .filter(_ % 3 == 0)
  .map(_ + 1)
  .take(10)
  .toList  // Materializes the result

println(s"Eager result: $eagerResult")
println(s"Lazy result: $lazyResult")

// Demonstrating the difference with timing
def timeOperation[T](name: String)(operation: => T): T = {
  val start = System.nanoTime()
  val result = operation
  val end = System.nanoTime()
  println(f"$name took ${(end - start) / 1e6}%.2f ms")
  result
}

val numbers = (1 to 1000000).toList

timeOperation("Eager evaluation") {
  numbers
    .map(_ * 2)
    .filter(_ % 100 == 0)
    .map(_.toString)
    .take(100)
}

timeOperation("Lazy evaluation") {
  numbers.view
    .map(_ * 2)
    .filter(_ % 100 == 0)
    .map(_.toString)
    .take(100)
    .toList
}

// Iterator for memory-efficient processing
def processLargeFile(filename: String): Unit = {
  // Simulated file processing
  val lines = (1 to 1000000).map(i => s"Line $i: Some data here")

  // Memory efficient - processes one line at a time
  val result = lines.iterator
    .filter(_.contains("100"))
    .map(_.toUpperCase)
    .take(10)
    .toList

  println(s"Processed ${result.length} lines")
}

processLargeFile("large.txt")

Summary

In this lesson, you've mastered Scala's essential higher-order functions and collection operations:

✅ Map: Transform every element in a collection
✅ Filter: Select elements based on predicates
✅ FlatMap: Transform and flatten nested structures
✅ Fold/Reduce: Aggregate elements into single values
✅ GroupBy: Organize data by categories
✅ Partition/Span: Split collections based on conditions
✅ Zip/Unzip: Combine and separate related collections
✅ Complex Pipelines: Chain operations for sophisticated data processing
✅ Performance: Use lazy evaluation for efficiency

These operations form the foundation of functional data processing in Scala and enable you to write concise, expressive code for complex transformations.

What's Next

In the next lesson, we'll explore pattern matching in depth, learning how to deconstruct data structures, match on types, and create powerful, readable conditional logic that goes far beyond simple if-else statements.