Scala Performance Lazy Streams LazyList View Evaluation Infinite Data Memory

August 31, 2025

Lazy Evaluation and Streams: Efficient Data Processing

Harness the power of lazy evaluation and streams to process large datasets efficiently, work with infinite sequences, and optimize memory usage in your Scala applications.

Lesson 27 🚀 Practice

Lazy Evaluation and Streams: Efficient Data Processing

Introduction

Lazy evaluation is a powerful technique where computations are deferred until their results are actually needed. Scala provides several mechanisms for lazy evaluation, including lazy values, views, and streams (LazyList). This approach can significantly improve performance and memory usage, especially when working with large datasets or infinite sequences.

This lesson will teach you to leverage lazy evaluation effectively, understand when to use different lazy constructs, and build efficient data processing pipelines.

Understanding Lazy Evaluation

Lazy Values

// Eager evaluation (default)
val eagerValue = {
  println("Computing eager value...")
  42 * 2
}
println("Eager value created")
println(s"Using eager value: $eagerValue")

// Lazy evaluation
lazy val lazyValue = {
  println("Computing lazy value...")
  42 * 2
}
println("Lazy value declared")
println(s"Using lazy value: $lazyValue")
println(s"Using lazy value again: $lazyValue")

// Lazy evaluation with expensive computation
def expensiveComputation(x: Int): Int = {
  println(s"Performing expensive computation for $x")
  Thread.sleep(100)  // Simulate expensive operation
  x * x
}

lazy val expensiveResult = expensiveComputation(10)
println("Expensive computation declared")

// Only computed when first accessed
println(s"Result: $expensiveResult")
println(s"Result again: $expensiveResult")  // Not computed again

// Lazy initialization pattern
class DatabaseConnection {
  private lazy val connection = {
    println("Establishing database connection...")
    // Simulate connection setup
    "connection-handle-123"
  }

  def query(sql: String): String = {
    s"Executing '$sql' on $connection"
  }
}

val db = new DatabaseConnection()
println("Database instance created")
println(db.query("SELECT * FROM users"))  // Connection established here

// Lazy fields in case classes (using lazy val)
case class Person(name: String, age: Int) {
  lazy val isAdult: Boolean = {
    println(s"Checking if $name is adult...")
    age >= 18
  }

  lazy val category: String = {
    println(s"Determining category for $name...")
    if (age < 13) "child"
    else if (age < 18) "teenager"
    else if (age < 65) "adult"
    else "senior"
  }
}

val person = Person("Alice", 25)
println(s"Person created: ${person.name}")
println(s"Is adult: ${person.isAdult}")     // Computed here
println(s"Category: ${person.category}")    // Computed here
println(s"Is adult again: ${person.isAdult}")  // Not computed again

// Conditional lazy evaluation
def createExpensiveResource(condition: Boolean): Option[String] = {
  if (condition) {
    lazy val resource = {
      println("Creating expensive resource...")
      "expensive-resource"
    }
    Some(resource)
  } else {
    None
  }
}

println("Testing conditional lazy evaluation:")
val resource1 = createExpensiveResource(false)  // No computation
val resource2 = createExpensiveResource(true)   // Computation deferred
println(s"Resource2: $resource2")  // Computation happens here

Lazy Function Parameters

// By-name parameters (lazy evaluation)
def logAndReturn[T](message: String)(value: => T): T = {
  println(s"Log: $message")
  value  // Evaluated here
}

def expensiveFunction(): String = {
  println("Executing expensive function...")
  Thread.sleep(50)
  "result"
}

// By-value parameter (eager evaluation)
def eagerLog[T](message: String, value: T): T = {
  println(s"Eager log: $message")
  value
}

println("Testing by-name parameters:")
logAndReturn("About to compute")(expensiveFunction())

println("\nTesting by-value parameters:")
eagerLog("About to compute", expensiveFunction())

// Conditional execution with by-name parameters
def ifThenElse[T](condition: Boolean)(thenBranch: => T)(elseBranch: => T): T = {
  if (condition) thenBranch else elseBranch
}

def sideEffect1(): String = {
  println("Side effect 1")
  "result1"
}

def sideEffect2(): String = {
  println("Side effect 2")
  "result2"
}

println("\nConditional execution:")
val result1 = ifThenElse(true)(sideEffect1())(sideEffect2())  // Only executes sideEffect1
val result2 = ifThenElse(false)(sideEffect1())(sideEffect2()) // Only executes sideEffect2

// Implementing lazy OR and AND operators
def lazyOr(a: => Boolean, b: => Boolean): Boolean = {
  if (a) {
    println("Short-circuiting OR: first condition is true")
    true
  } else {
    println("Evaluating second condition in OR")
    b
  }
}

def lazyAnd(a: => Boolean, b: => Boolean): Boolean = {
  if (!a) {
    println("Short-circuiting AND: first condition is false")
    false
  } else {
    println("Evaluating second condition in AND")
    b
  }
}

def expensiveCheck1(): Boolean = {
  println("Expensive check 1")
  true
}

def expensiveCheck2(): Boolean = {
  println("Expensive check 2")
  false
}

println("\nLazy operators:")
println(s"Lazy OR: ${lazyOr(expensiveCheck1(), expensiveCheck2())}")
println(s"Lazy AND: ${lazyAnd(expensiveCheck2(), expensiveCheck1())}")

// Memoization with lazy evaluation
class Fibonacci {
  private val cache = scala.collection.mutable.Map[Int, BigInt]()

  def apply(n: Int): BigInt = {
    cache.getOrElseUpdate(n, {
      println(s"Computing fibonacci($n)")
      if (n <= 1) n
      else apply(n - 1) + apply(n - 2)
    })
  }
}

val fib = new Fibonacci()
println("\nFibonacci with memoization:")
println(s"fib(10) = ${fib(10)}")
println(s"fib(15) = ${fib(15)}")  // Uses cached values from fib(10)

// Lazy factory pattern
trait LazyFactory[T] {
  lazy val instance: T = createInstance()
  protected def createInstance(): T
}

class DatabaseService extends LazyFactory[String] {
  protected def createInstance(): String = {
    println("Initializing database service...")
    "database-service-instance"
  }

  def getData(): String = s"Data from $instance"
}

val service = new DatabaseService()
println("Service created")
println(service.getData())  // Instance created here

Views: Lazy Collections

Creating and Using Views

// View creates a lazy version of a collection
val numbers = (1 to 1000000).toList
println(s"List created with ${numbers.length} elements")

// Eager evaluation
def timeOperation[T](name: String)(operation: => T): T = {
  val start = System.nanoTime()
  val result = operation
  val end = System.nanoTime()
  println(f"$name: ${(end - start) / 1e6}%.2f ms")
  result
}

timeOperation("Eager operations") {
  val result = numbers
    .filter(_ % 2 == 0)
    .map(_ * 2)
    .filter(_ > 1000)
    .take(100)
  result.length
}

// Lazy evaluation with view
timeOperation("Lazy operations with view") {
  val result = numbers.view
    .filter(_ % 2 == 0)
    .map(_ * 2)
    .filter(_ > 1000)
    .take(100)
    .toList
  result.length
}

// View transformations are lazy
val numberView = numbers.view
  .filter { x =>
    println(s"Filtering $x")
    x % 2 == 0
  }
  .map { x =>
    println(s"Mapping $x")
    x * 2
  }

println("View created - no operations executed yet")

// Operations execute when materialized
val firstFive = numberView.take(5).toList
println(s"First five: $firstFive")

// Working with infinite views
val infiniteNumbers = LazyList.from(1)
val evenSquares = infiniteNumbers.view
  .filter(_ % 2 == 0)
  .map(x => x * x)

println(s"First 10 even squares: ${evenSquares.take(10).toList}")

// Practical view examples
case class LogEntry(timestamp: Long, level: String, message: String)

val logEntries = (1 to 100000).map(i =>
  LogEntry(
    System.currentTimeMillis() - i * 1000,
    if (i % 10 == 0) "ERROR" else if (i % 5 == 0) "WARN" else "INFO",
    s"Log message $i"
  )
).toList

// Efficient log processing with views
def processLogs(entries: List[LogEntry]): List[String] = {
  entries.view
    .filter(_.level == "ERROR")
    .map(entry => s"${entry.timestamp}: ${entry.message}")
    .take(10)
    .toList
}

timeOperation("Log processing with view") {
  processLogs(logEntries).length
}

// Without view (less efficient)
def processLogsEager(entries: List[LogEntry]): List[String] = {
  entries
    .filter(_.level == "ERROR")
    .map(entry => s"${entry.timestamp}: ${entry.message}")
    .take(10)
}

timeOperation("Log processing without view") {
  processLogsEager(logEntries).length
}

// Chaining multiple view operations
val complexView = (1 to 1000).view
  .filter(_ % 3 == 0)
  .map(_ * 2)
  .zipWithIndex
  .filter { case (value, index) => index % 2 == 0 }
  .map { case (value, index) => s"$index: $value" }

println("Complex view operations:")
complexView.take(5).foreach(println)

// Views with groupBy (materializes intermediate results)
val groupedView = numbers.view
  .filter(_ < 100)
  .groupBy(_ % 10)
  .view
  .mapValues(_.sum)

println(s"Grouped sums: ${groupedView.toMap}")

// Sliding window with views
val slidingView = (1 to 20).view
  .sliding(3)
  .map(_.sum)

println("Sliding window sums:")
slidingView.take(5).foreach(println)

LazyList (Streams)

Creating and Working with Streams

// LazyList (formerly Stream in Scala 2.12)
val lazyNumbers = LazyList.from(1)
println(s"Lazy list created: ${lazyNumbers}")

// Taking elements forces evaluation
val firstTen = lazyNumbers.take(10).toList
println(s"First 10 numbers: $firstTen")

// Infinite sequences
val fibonacciStream: LazyList[BigInt] = {
  def fib(a: BigInt, b: BigInt): LazyList[BigInt] = 
    a #:: fib(b, a + b)
  fib(0, 1)
}

println(s"First 20 Fibonacci numbers:")
fibonacciStream.take(20).foreach(println)

// Prime number sieve
def sieve(numbers: LazyList[Int]): LazyList[Int] = {
  numbers match {
    case head #:: tail =>
      head #:: sieve(tail.filter(_ % head != 0))
    case _ => LazyList.empty
  }
}

val primes = sieve(LazyList.from(2))
println(s"First 20 primes: ${primes.take(20).toList}")

// Generating sequences with LazyList.iterate
val powersOfTwo = LazyList.iterate(1)(_ * 2)
println(s"First 10 powers of 2: ${powersOfTwo.take(10).toList}")

val collatzSequence = LazyList.iterate(13)(n => if (n % 2 == 0) n / 2 else 3 * n + 1)
val collatzSteps = collatzSequence.takeWhile(_ != 1).toList
println(s"Collatz sequence starting from 13: $collatzSteps")

// Custom stream generators
def randomStream(seed: Long): LazyList[Int] = {
  val random = new scala.util.Random(seed)
  LazyList.continually(random.nextInt(100))
}

val randomNumbers = randomStream(42)
println(s"First 10 random numbers: ${randomNumbers.take(10).toList}")

// Date sequence stream
import java.time.LocalDate

def dateStream(start: LocalDate): LazyList[LocalDate] = 
  LazyList.iterate(start)(_.plusDays(1))

val dates = dateStream(LocalDate.of(2024, 1, 1))
val weekends = dates
  .filter(date => date.getDayOfWeek.getValue >= 6)
  .take(10)

println("First 10 weekends of 2024:")
weekends.foreach(println)

// Combining streams
val naturalNumbers = LazyList.from(1)
val evenNumbers = naturalNumbers.filter(_ % 2 == 0)
val oddNumbers = naturalNumbers.filter(_ % 2 == 1)

// Merge two streams alternately
def mergeStreams[T](s1: LazyList[T], s2: LazyList[T]): LazyList[T] = {
  (s1, s2) match {
    case (h1 #:: t1, h2 #:: t2) => h1 #:: h2 #:: mergeStreams(t1, t2)
    case (LazyList(), s2) => s2
    case (s1, LazyList()) => s1
  }
}

val alternating = mergeStreams(evenNumbers, oddNumbers)
println(s"Alternating even/odd: ${alternating.take(20).toList}")

// Stream of streams (nested lazy evaluation)
val matrixStream = LazyList.from(1).map(row => 
  LazyList.from(1).map(col => row * col)
)

println("3x3 multiplication table:")
matrixStream.take(3).foreach { row =>
  println(row.take(3).toList)
}

// Working with file streams (conceptual)
def fileLineStream(filename: String): LazyList[String] = {
  // Simulate reading lines lazily
  LazyList.from(1).map(i => s"Line $i from $filename")
}

val logLines = fileLineStream("app.log")
val errorLines = logLines
  .filter(_.contains("ERROR"))
  .take(5)

println("Simulated error lines:")
errorLines.foreach(println)

// Memoized streams
case class MemoizedStream[T](private val underlying: LazyList[T]) {
  private val cache = scala.collection.mutable.ArrayBuffer[T]()

  def apply(index: Int): T = {
    if (index >= cache.length) {
      val newElements = underlying.drop(cache.length).take(index - cache.length + 1)
      cache ++= newElements
    }
    cache(index)
  }

  def take(n: Int): List[T] = (0 until n).map(apply).toList
}

val memoizedFib = MemoizedStream(fibonacciStream)
println(s"Memoized fib(30): ${memoizedFib(30)}")
println(s"Memoized fib(25): ${memoizedFib(25)}")  // Uses cached values

// Stream comprehensions
val pythagoreanTriples = for {
  c <- LazyList.from(1)
  a <- LazyList.range(1, c)
  b <- LazyList.range(a, c)
  if a * a + b * b == c * c
} yield (a, b, c)

println("First 10 Pythagorean triples:")
pythagoreanTriples.take(10).foreach(println)

Practical Applications

Large Data Processing

// Processing large datasets efficiently
case class SalesRecord(id: Int, product: String, amount: Double, region: String, date: String)

// Simulate large dataset
val salesData = (1 to 1000000).map(i =>
  SalesRecord(
    i,
    s"Product${i % 100}",
    math.random() * 1000,
    List("North", "South", "East", "West")(i % 4),
    s"2024-${(i % 12) + 1:02d}-${(i % 28) + 1:02d}"
  )
).toList

// Efficient aggregation with views
def salesAnalysis(data: List[SalesRecord]): Map[String, Double] = {
  timeOperation("Sales analysis with view") {
    data.view
      .filter(_.amount > 500)
      .groupBy(_.region)
      .view
      .mapValues(_.map(_.amount).sum)
      .toMap
  }
}

val regionSales = salesAnalysis(salesData)
regionSales.foreach { case (region, total) =>
  println(f"$region: $$${total}%.2f")
}

// Streaming CSV processing simulation
def processCsvStream(lines: LazyList[String]): LazyList[SalesRecord] = {
  lines
    .filter(_.nonEmpty)
    .filter(!_.startsWith("#"))  // Skip comments
    .map(_.split(","))
    .filter(_.length >= 5)
    .map { fields =>
      try {
        Some(SalesRecord(
          fields(0).toInt,
          fields(1),
          fields(2).toDouble,
          fields(3),
          fields(4)
        ))
      } catch {
        case _: Exception => None
      }
    }
    .collect { case Some(record) => record }
}

// Simulate CSV data stream
val csvLines = LazyList.from(1).map(i =>
  s"$i,Product${i % 50},${math.random() * 1000},${List("North", "South", "East", "West")(i % 4)},2024-01-01"
)

val processedRecords = processCsvStream(csvLines)
println("Sample processed records:")
processedRecords.take(5).foreach(println)

// Pipeline processing with lazy evaluation
class DataPipeline[T] {
  private var transformations: LazyList[T] => LazyList[T] = identity

  def filter(predicate: T => Boolean): DataPipeline[T] = {
    val prevTransform = transformations
    transformations = (stream: LazyList[T]) => prevTransform(stream).filter(predicate)
    this
  }

  def map[U](mapper: T => U): DataPipeline[U] = {
    new DataPipeline[U] {
      transformations = (stream: LazyList[U]) => stream
    }
  }

  def take(n: Int): DataPipeline[T] = {
    val prevTransform = transformations
    transformations = (stream: LazyList[T]) => prevTransform(stream).take(n)
    this
  }

  def process(data: LazyList[T]): LazyList[T] = transformations(data)
}

val pipeline = new DataPipeline[Int]()
  .filter(_ % 2 == 0)
  .filter(_ > 100)
  .take(10)

val pipelineResult = pipeline.process(LazyList.from(1))
println(s"Pipeline result: ${pipelineResult.toList}")

// Real-time data processing simulation
def generateSensorData(): LazyList[Double] = {
  val random = new scala.util.Random()
  LazyList.continually {
    Thread.sleep(10)  // Simulate sensor delay
    20.0 + random.nextGaussian() * 5.0  // Temperature readings
  }
}

def processTemperatureData(data: LazyList[Double]): LazyList[String] = {
  data
    .zipWithIndex
    .map { case (temp, index) =>
      val status = if (temp > 30) "HIGH" else if (temp < 10) "LOW" else "NORMAL"
      f"Reading $index: $temp%.1f°C ($status)"
    }
    .filter(_.contains("HIGH"))  // Only alert on high temperatures
}

// Simulate processing first few readings
val sensorStream = generateSensorData()
val alerts = processTemperatureData(sensorStream)

println("Temperature alerts (first 5):")
alerts.take(5).foreach(println)

// Windowed operations on streams
def slidingAverage(data: LazyList[Double], windowSize: Int): LazyList[Double] = {
  data
    .sliding(windowSize)
    .map(window => window.sum / window.length)
}

val temperatureReadings = LazyList.from(1).map(i => 20.0 + math.sin(i * 0.1) * 10)
val smoothedTemperatures = slidingAverage(temperatureReadings, 5)

println("Original vs smoothed temperatures:")
temperatureReadings.zip(smoothedTemperatures).take(10).foreach {
  case (orig, smooth) => println(f"Original: $orig%.2f, Smoothed: $smooth%.2f")
}

Memory-Efficient Algorithms

// Efficient prime finding with lazy evaluation
def isPrime(n: Int): Boolean = {
  if (n < 2) false
  else if (n == 2) true
  else !(2 to math.sqrt(n).toInt).exists(n % _ == 0)
}

def primesUpTo(limit: Int): LazyList[Int] = {
  LazyList.from(2).takeWhile(_ <= limit).filter(isPrime)
}

timeOperation("Find primes up to 10000") {
  val primeCount = primesUpTo(10000).length
  println(s"Number of primes: $primeCount")
}

// Memory-efficient file processing
def processLargeFile(lines: LazyList[String]): (Int, Int, Map[String, Int]) = {
  val stats = lines
    .view
    .zipWithIndex
    .map { case (line, index) =>
      val words = line.split("\\s+").filter(_.nonEmpty)
      (1, words.length, words.groupBy(identity).view.mapValues(_.length).toMap)
    }
    .foldLeft((0, 0, Map.empty[String, Int])) {
      case ((lineCount, wordCount, wordFreq), (lines, words, freq)) =>
        val newWordFreq = (wordFreq.keySet ++ freq.keySet).map { word =>
          word -> (wordFreq.getOrElse(word, 0) + freq.getOrElse(word, 0))
        }.toMap
        (lineCount + lines, wordCount + words, newWordFreq)
    }

  stats
}

// Simulate file content
val fileContent = LazyList.from(1).map(i => s"line $i with some repeated words and different content")

val (totalLines, totalWords, wordFrequency) = processLargeFile(fileContent.take(1000))
println(s"Lines: $totalLines, Words: $totalWords")
println("Top 5 words:")
wordFrequency.toSeq.sortBy(-_._2).take(5).foreach {
  case (word, count) => println(s"  $word: $count")
}

// Graph traversal with lazy evaluation
case class Node(id: Int, connections: List[Int])

class LazyGraph(nodes: Map[Int, Node]) {
  def breadthFirstSearch(start: Int): LazyList[Int] = {
    def bfs(queue: LazyList[Int], visited: Set[Int]): LazyList[Int] = {
      queue match {
        case current #:: rest if !visited.contains(current) =>
          val neighbors = nodes.get(current).map(_.connections).getOrElse(List.empty)
          val newQueue = rest ++ neighbors.filter(!visited.contains(_))
          current #:: bfs(newQueue, visited + current)
        case current #:: rest =>
          bfs(rest, visited)
        case _ =>
          LazyList.empty
      }
    }

    bfs(LazyList(start), Set.empty)
  }

  def depthFirstSearch(start: Int): LazyList[Int] = {
    def dfs(stack: List[Int], visited: Set[Int]): LazyList[Int] = {
      stack match {
        case current :: rest if !visited.contains(current) =>
          val neighbors = nodes.get(current).map(_.connections).getOrElse(List.empty)
          current #:: dfs(neighbors ++ rest, visited + current)
        case current :: rest =>
          dfs(rest, visited)
        case Nil =>
          LazyList.empty
      }
    }

    dfs(List(start), Set.empty)
  }
}

// Create a sample graph
val graphNodes = Map(
  1 -> Node(1, List(2, 3)),
  2 -> Node(2, List(1, 4, 5)),
  3 -> Node(3, List(1, 6)),
  4 -> Node(4, List(2)),
  5 -> Node(5, List(2, 6)),
  6 -> Node(6, List(3, 5))
)

val graph = new LazyGraph(graphNodes)

println("BFS traversal:")
graph.breadthFirstSearch(1).take(10).foreach(println)

println("DFS traversal:")
graph.depthFirstSearch(1).take(10).foreach(println)

// Lazy tree operations
sealed trait Tree[+T]
case object Empty extends Tree[Nothing]
case class Branch[T](value: T, left: Tree[T], right: Tree[T]) extends Tree[T]

object Tree {
  def fromList[T](list: List[T]): Tree[T] = {
    def build(items: List[T]): Tree[T] = items match {
      case Nil => Empty
      case head :: tail =>
        val (left, right) = tail.splitAt(tail.length / 2)
        Branch(head, build(left), build(right))
    }
    build(list)
  }

  def traverse[T](tree: Tree[T]): LazyList[T] = tree match {
    case Empty => LazyList.empty
    case Branch(value, left, right) =>
      traverse(left) ++ LazyList(value) ++ traverse(right)
  }
}

val tree = Tree.fromList((1 to 15).toList)
val treeTraversal = Tree.traverse(tree)
println(s"Tree traversal: ${treeTraversal.take(10).toList}")

Best Practices and Performance Tips

When to Use Lazy Evaluation

// Guidelines for using lazy evaluation effectively

// 1. Use lazy vals for expensive computations that might not be needed
class ConfigurationManager {
  lazy val databaseConfig = {
    println("Loading database configuration...")
    // Expensive file reading or network call
    Map("host" -> "localhost", "port" -> 5432)
  }

  lazy val cacheConfig = {
    println("Loading cache configuration...")
    Map("ttl" -> 3600, "maxSize" -> 1000)
  }

  def getDatabaseHost: String = databaseConfig("host").toString

  // This might never be called, so cache config won't be loaded
  def getCacheTtl: Int = cacheConfig("ttl").toString.toInt
}

val config = new ConfigurationManager()
println(s"Database host: ${config.getDatabaseHost}")
// Cache config is never loaded

// 2. Use views for chain operations on large collections
def efficientDataProcessing(data: List[Int]): List[Int] = {
  if (data.length > 10000) {
    // Use view for large datasets
    data.view
      .filter(_ % 2 == 0)
      .map(_ * 3)
      .filter(_ > 100)
      .take(50)
      .toList
  } else {
    // Regular operations for small datasets
    data
      .filter(_ % 2 == 0)
      .map(_ * 3)
      .filter(_ > 100)
      .take(50)
  }
}

// 3. Use LazyList for potentially infinite sequences
def generateReports(): LazyList[String] = {
  LazyList.from(1).map { reportId =>
    // Each report is generated only when requested
    s"Report $reportId: ${generateReportContent(reportId)}"
  }
}

def generateReportContent(id: Int): String = {
  s"Content for report $id (generated at ${System.currentTimeMillis()})"
}

val reports = generateReports()
println("First 3 reports:")
reports.take(3).foreach(println)

// 4. Avoid lazy evaluation pitfalls
class LazyPitfalls {
  // Pitfall 1: Lazy vals with side effects
  var counter = 0
  lazy val problematicLazy = {
    counter += 1  // Side effect - unpredictable when it happens
    counter
  }

  // Better: Use def for operations with side effects
  def incrementCounter(): Int = {
    counter += 1
    counter
  }

  // Pitfall 2: Memory leaks with large lazy structures
  lazy val largeData = (1 to 1000000).toList  // Might consume a lot of memory

  // Better: Use streams or views
  def largeDataStream = LazyList.from(1).take(1000000)

  // Pitfall 3: Exception handling with lazy values
  lazy val riskyOperation = {
    val result = 10 / 0  // This will throw an exception
    result
  }

  // Better: Handle exceptions explicitly
  lazy val safeOperation = {
    try {
      Some(10 / 0)
    } catch {
      case _: ArithmeticException => None
    }
  }
}

// 5. Measuring lazy evaluation performance
def measureLazyPerformance(): Unit = {
  val data = (1 to 100000).toList

  // Eager evaluation
  timeOperation("Eager evaluation") {
    val result = data
      .filter(_ % 2 == 0)
      .map(_ * 2)
      .filter(_ > 1000)
      .take(100)
    result.length
  }

  // Lazy evaluation with view
  timeOperation("Lazy evaluation") {
    val result = data.view
      .filter(_ % 2 == 0)
      .map(_ * 2)
      .filter(_ > 1000)
      .take(100)
      .toList
    result.length
  }

  // LazyList
  timeOperation("LazyList") {
    val result = LazyList.from(1)
      .filter(_ % 2 == 0)
      .map(_ * 2)
      .filter(_ > 1000)
      .take(100)
      .toList
    result.length
  }
}

measureLazyPerformance()

// 6. Best practices summary
object LazyEvaluationBestPractices {
  // Use lazy val for expensive initialization
  lazy val expensiveResource = initializeExpensiveResource()

  // Use views for intermediate collection operations
  def processCollection[T](data: List[T], transform: T => T): List[T] = {
    data.view.map(transform).filter(_ != null).toList
  }

  // Use LazyList for infinite or very large sequences
  def infiniteSequence[T](generator: Int => T): LazyList[T] = {
    LazyList.from(0).map(generator)
  }

  // Use by-name parameters for conditional evaluation
  def debug(condition: Boolean)(message: => String): Unit = {
    if (condition) println(s"DEBUG: $message")
  }

  private def initializeExpensiveResource(): String = {
    Thread.sleep(100)
    "expensive-resource"
  }
}

// Usage examples
LazyEvaluationBestPractices.debug(true)("This will be evaluated")
LazyEvaluationBestPractices.debug(false)("This will NOT be evaluated")

val processed = LazyEvaluationBestPractices.processCollection(
  List("a", "b", "c"), 
  _.toUpperCase
)
println(s"Processed: $processed")

val sequence = LazyEvaluationBestPractices.infiniteSequence(i => i * i)
println(s"First 5 squares: ${sequence.take(5).toList}")

Summary

In this lesson, you've mastered lazy evaluation and streams in Scala:

✅ Lazy Values: Deferring expensive computations until needed
✅ By-Name Parameters: Lazy function argument evaluation
✅ Views: Lazy collection operations for performance
✅ LazyList: Working with infinite and large sequences
✅ Practical Applications: Real-world lazy evaluation patterns
✅ Performance: Optimizing memory and computation
✅ Best Practices: When and how to use lazy evaluation effectively

Lazy evaluation is a powerful technique for building efficient, memory-conscious applications that can handle large datasets and infinite sequences gracefully.

What's Next

In the next lesson, we'll explore concurrency and parallelism in Scala, learning how to write concurrent programs using Futures, parallel collections, and actor systems.

🚀 Practice Exercise

Put your knowledge to practice with hands-on coding exercises

Advanced ⏱️ 2h 30min

Instructions

Lazy Evaluation and Streams

Build a sophisticated lazy evaluation framework featuring advanced streaming algorithms, memoized recursive structures, lazy evaluation patterns for performance optimization, and complex lazy data processing pipelines.

Your program should:
1. Implement custom lazy data structures with memoization
2. Build advanced streaming algorithms (sorting, grouping, windowing)
3. Create lazy evaluation patterns for dynamic programming
4. Design memory-efficient processing of large datasets
5. Implement lazy evaluation for recursive algorithms
6. Build a functional reactive programming framework using lazy evaluation

Requirements:
- Implement custom lazy collection types with proper memoization
- Create streaming algorithms that work with infinite data
- Use lazy evaluation for performance optimization in complex algorithms
- Build lazy data processing pipelines for large-scale data
- Implement lazy recursive algorithms (parsing, graph traversal)
- Create reactive streams with lazy event processing

Example usage:
```scala
val processor = LazyDataProcessor(hugeDataset)
val result = processor
.lazyFilter(complexPredicate)
.lazyGroupBy(keyExtractor)
.lazyWindow(1000)
.process()
```

Interactive Playground

⚠️ Spoiler Alert! Try to solve the exercise yourself first. Learning happens through practice!

```scala
import scala.collection.mutable
import java.util.concurrent.atomic.AtomicInteger

object AdvancedLazyEvaluation extends App {
println("=== Advanced Lazy Evaluation Framework ===")

// 1. Custom Lazy Data Structure with Memoization
println("\n1. Memoized Lazy List Implementation")

sealed trait MemoizedLazyList[+A] {
def isEmpty: Boolean
def head: A
def tail: MemoizedLazyList[A]

def #:::[B >: A](elem: B): MemoizedLazyList[B] = MemoizedLazyList.cons(elem, this)

def map[B](f: A => B): MemoizedLazyList[B] = this match {
case MemoizedLazyList.Empty => MemoizedLazyList.Empty
case _ => MemoizedLazyList.cons(f(head), tail.map(f))
}

def filter(p: A => Boolean): MemoizedLazyList[A] = this match {
case MemoizedLazyList.Empty => MemoizedLazyList.Empty
case _ if p(head) => MemoizedLazyList.cons(head, tail.filter(p))
case _ => tail.filter(p)
}

def take(n: Int): MemoizedLazyList[A] = {
if (n <= 0 || isEmpty) MemoizedLazyList.Empty
else MemoizedLazyList.cons(head, tail.take(n - 1))
}

def drop(n: Int): MemoizedLazyList[A] = {
if (n <= 0) this
else if (isEmpty) MemoizedLazyList.Empty
else tail.drop(n - 1)
}

def foldLeft[B](z: B)(f: (B, A) => B): B = this match {
case MemoizedLazyList.Empty => z
case _ => tail.foldLeft(f(z, head))(f)
}

def toList: List[A] = foldLeft(List.empty[A])((acc, elem) => elem :: acc).reverse
}

object MemoizedLazyList {
case object Empty extends MemoizedLazyList[Nothing] {
def isEmpty: Boolean = true
def head: Nothing = throw new NoSuchElementException("Empty list")
def tail: MemoizedLazyList[Nothing] = throw new NoSuchElementException("Empty list")
}

case class Cons[A](
headValue: A,
private var tailThunk: () => MemoizedLazyList[A],
private var tailCache: Option[MemoizedLazyList[A]] = None,
private var isEvaluated: Boolean = false
) extends MemoizedLazyList[A] {

def isEmpty: Boolean = false
def head: A = headValue

def tail: MemoizedLazyList[A] = {
if (!isEvaluated) {
tailCache = Some(tailThunk())
tailThunk = null // Allow GC
isEvaluated = true
}
tailCache.get
}
}

def cons[A](head: A, tail: => MemoizedLazyList[A]): MemoizedLazyList[A] =
Cons(head, () => tail)

def from[A](seq: Seq[A]): MemoizedLazyList[A] = seq match {
case Nil => Empty
case head :: tail => cons(head, from(tail))
}

def continually[A](elem: A): MemoizedLazyList[A] =
cons(elem, continually(elem))

def iterate[A](start: A)(f: A => A): MemoizedLazyList[A] =
cons(start, iterate(f(start))(f))

def unfold[A, B](seed: B)(f: B => Option[(A, B)]): MemoizedLazyList[A] = {
f(seed) match {
case None => Empty
case Some((a, nextSeed)) => cons(a, unfold(nextSeed)(f))
}
}
}

// Demonstrate memoized lazy list
val computationCounter = new AtomicInteger(0)

def expensiveFunction(x: Int): Int = {
val count = computationCounter.incrementAndGet()
println(s"    Computing $x (computation #$count)")
Thread.sleep(10) // Simulate expensive computation
x * x
}

val memoizedSquares = MemoizedLazyList.iterate(1)(_ + 1).map(expensiveFunction)

println("First access to first 5 squares:")
val firstFive = memoizedSquares.take(5).toList
println(s"Result: $firstFive")

println("\nSecond access to first 3 squares (should use cache):")
val firstThree = memoizedSquares.take(3).toList
println(s"Result: $firstThree")

println(s"Total computations performed: ${computationCounter.get()}")

// 2. Advanced Streaming Algorithms
println("\n2. Advanced Streaming Algorithms")

class LazyStream[A] private (private val elements: () => Option[(A, LazyStream[A])]) {

def uncons: Option[(A, LazyStream[A])] = elements()

def isEmpty: Boolean = uncons.isEmpty

def head: Option[A] = uncons.map(_._1)

def tail: LazyStream[A] = uncons.map(_._2).getOrElse(LazyStream.empty)

def map[B](f: A => B): LazyStream[B] = LazyStream {
uncons.map { case (head, tail) => (f(head), tail.map(f)) }
}

def filter(p: A => Boolean): LazyStream[A] = LazyStream {
uncons.flatMap { case (head, tail) =>
if (p(head)) Some((head, tail.filter(p)))
else tail.filter(p).uncons
}
}

def take(n: Int): LazyStream[A] = LazyStream {
if (n <= 0) None
else uncons.map { case (head, tail) => (head, tail.take(n - 1)) }
}

def drop(n: Int): LazyStream[A] = {
if (n <= 0) this
else tail.drop(n - 1)
}

def takeWhile(p: A => Boolean): LazyStream[A] = LazyStream {
uncons.flatMap { case (head, tail) =>
if (p(head)) Some((head, tail.takeWhile(p)))
else None
}
}

def dropWhile(p: A => Boolean): LazyStream[A] = LazyStream {
uncons.flatMap { case (head, tail) =>
if (p(head)) tail.dropWhile(p).uncons
else Some((head, tail))
}
}

// Lazy merge sort
def sorted(implicit ord: Ordering[A]): LazyStream[A] = {
def merge(s1: LazyStream[A], s2: LazyStream[A]): LazyStream[A] = LazyStream {
(s1.uncons, s2.uncons) match {
case (Some((h1, t1)), Some((h2, t2))) =>
if (ord.lteq(h1, h2)) Some((h1, merge(t1, s2)))
else Some((h2, merge(s1, t2)))
case (Some((h1, t1)), None) => Some((h1, t1))
case (None, Some((h2, t2))) => Some((h2, t2))
case (None, None) => None
}
}

def mergeSort(stream: LazyStream[A]): LazyStream[A] = {
val list = stream.take(1000).toList // Process in chunks
if (list.length <= 1) LazyStream.fromList(list.sorted)
else {
val mid = list.length / 2
val (left, right) = list.splitAt(mid)
merge(
mergeSort(LazyStream.fromList(left)),
mergeSort(LazyStream.fromList(right))
)
}
}

mergeSort(this)
}

// Lazy groupBy
def groupBy[K](f: A => K): Map[K, LazyStream[A]] = {
val groups = mutable.Map[K, mutable.ListBuffer[A]]()

def processElement(elem: A): Unit = {
val key = f(elem)
groups.getOrElseUpdate(key, mutable.ListBuffer()) += elem
}

// Process elements lazily
var current = this
while (!current.isEmpty) {
current.head.foreach(processElement)
current = current.tail
}

groups.map { case (k, buffer) => k -> LazyStream.fromList(buffer.toList) }.toMap
}

// Sliding window
def sliding(size: Int): LazyStream[List[A]] = LazyStream {
val window = take(size).toList
if (window.length == size) {
Some((window, tail.sliding(size)))
} else None
}

def toList: List[A] = {
val buffer = mutable.ListBuffer[A]()
var current = this
while (!current.isEmpty) {
current.head.foreach(buffer += _)
current = current.tail
}
buffer.toList
}

def foreach(f: A => Unit): Unit = {
var current = this
while (!current.isEmpty) {
current.head.foreach(f)
current = current.tail
}
}
}

object LazyStream {
def empty[A]: LazyStream[A] = new LazyStream(() => None)

def apply[A](thunk: => Option[(A, LazyStream[A])]): LazyStream[A] =
new LazyStream(() => thunk)

def fromList[A](list: List[A]): LazyStream[A] = list match {
case Nil => empty
case head :: tail => LazyStream(Some((head, fromList(tail))))
}

def continually[A](elem: A): LazyStream[A] =
LazyStream(Some((elem, continually(elem))))

def iterate[A](start: A)(f: A => A): LazyStream[A] =
LazyStream(Some((start, iterate(f(start))(f))))

def range(start: Int, end: Int): LazyStream[Int] = {
if (start >= end) empty
else LazyStream(Some((start, range(start + 1, end))))
}

def fibonacci: LazyStream[BigInt] = {
def fib(a: BigInt, b: BigInt): LazyStream[BigInt] =
LazyStream(Some((a, fib(b, a + b))))
fib(0, 1)
}
}

// Demonstrate advanced streaming
println("Testing lazy merge sort:")
val unsorted = LazyStream.fromList(List(5, 2, 8, 1, 9, 3))
val sortedStream = unsorted.sorted
println(s"Sorted: ${sortedStream.toList}")

println("Testing sliding window:")
val numbers = LazyStream.range(1, 11)
val windows = numbers.sliding(3)
println("Sliding windows of size 3:")
windows.take(5).foreach(window => println(s"  $window"))

// 3. Dynamic Programming with Lazy Evaluation
println("\n3. Dynamic Programming - Fibonacci with Memoization")

class LazyFibonacci {
private val cache = mutable.Map[Int, BigInt]()

lazy val fibSequence: LazyStream[BigInt] = {
def fibStream(n: Int): LazyStream[BigInt] = LazyStream {
Some((fib(n), fibStream(n + 1)))
}
fibStream(0)
}

def fib(n: Int): BigInt = {
cache.getOrElseUpdate(n, {
if (n <= 1) BigInt(n)
else fib(n - 1) + fib(n - 2)
})
}

def getCacheSize: Int = cache.size
}

val lazyFib = new LazyFibonacci()
println("Computing Fibonacci numbers:")
val fibNumbers = lazyFib.fibSequence.take(20).toList
println(s"First 20: ${fibNumbers.take(10)}...")
println(s"Cache size after computation: ${lazyFib.getCacheSize}")

// Test large Fibonacci number
val fib100 = lazyFib.fib(100)
println(s"F(100) = ${fib100}")
println(s"Cache size after F(100): ${lazyFib.getCacheSize}")

// 4. Lazy Graph Traversal
println("\n4. Lazy Graph Traversal")

case class Graph[A](adjacencyList: Map[A, List[A]]) {
def neighbors(node: A): List[A] = adjacencyList.getOrElse(node, Nil)
}

object GraphTraversal {
def lazyBFS[A](graph: Graph[A], start: A): LazyStream[A] = {
def bfsStream(queue: List[A], visited: Set[A]): LazyStream[A] = LazyStream {
queue match {
case Nil => None
case head :: tail =>
if (visited.contains(head)) {
bfsStream(tail, visited).uncons
} else {
val newNeighbors = graph.neighbors(head).filterNot(visited.contains)
Some((head, bfsStream(tail ++ newNeighbors, visited + head)))
}
}
}
bfsStream(List(start), Set.empty)
}

def lazyDFS[A](graph: Graph[A], start: A): LazyStream[A] = {
def dfsStream(stack: List[A], visited: Set[A]): LazyStream[A] = LazyStream {
stack match {
case Nil => None
case head :: tail =>
if (visited.contains(head)) {
dfsStream(tail, visited).uncons
} else {
val newNeighbors = graph.neighbors(head).filterNot(visited.contains)
Some((head, dfsStream(newNeighbors ++ tail, visited + head)))
}
}
}
dfsStream(List(start), Set.empty)
}
}

// Create a sample graph
val graph = Graph(Map(
"A" -> List("B", "C"),
"B" -> List("D", "E"),
"C" -> List("F"),
"D" -> List("G"),
"E" -> List("G"),
"F" -> List("H"),
"G" -> List("I"),
"H" -> List("I"),
"I" -> List()
))

println("Lazy BFS traversal from A:")
val bfsResult = GraphTraversal.lazyBFS(graph, "A").toList
println(s"BFS order: ${bfsResult.mkString(" -> ")}")

println("Lazy DFS traversal from A:")
val dfsResult = GraphTraversal.lazyDFS(graph, "A").toList
println(s"DFS order: ${dfsResult.mkString(" -> ")}")

// 5. Functional Reactive Programming Framework
println("\n5. Functional Reactive Programming")

sealed trait Event[+A]
case class Value[A](value: A) extends Event[A]
case object NoEvent extends Event[Nothing]

class EventStream[A] private (private val events: () => LazyStream[Event[A]]) {
def getEvents: LazyStream[Event[A]] = events()

def map[B](f: A => B): EventStream[B] = new EventStream(() => {
getEvents.map {
case Value(a) => Value(f(a))
case NoEvent => NoEvent
}
})

def filter(p: A => Boolean): EventStream[A] = new EventStream(() => {
getEvents.filter {
case Value(a) => p(a)
case NoEvent => false
}
})

def merge(other: EventStream[A]): EventStream[A] = new EventStream(() => {
def mergeStreams(s1: LazyStream[Event[A]], s2: LazyStream[Event[A]]): LazyStream[Event[A]] = {
LazyStream {
(s1.uncons, s2.uncons) match {
case (Some((e1, t1)), Some((e2, t2))) =>
// Simplified merge - alternate between streams
Some((e1, mergeStreams(s2, t1)))
case (Some((e1, t1)), None) => Some((e1, t1))
case (None, Some((e2, t2))) => Some((e2, t2))
case (None, None) => None
}
}
}
mergeStreams(getEvents, other.getEvents)
})

def scan[B](initial: B)(f: (B, A) => B): EventStream[B] = new EventStream(() => {
def scanStream(stream: LazyStream[Event[A]], acc: B): LazyStream[Event[B]] = {
LazyStream {
stream.uncons.map { case (event, tail) =>
event match {
case Value(a) =>
val newAcc = f(acc, a)
(Value(newAcc), scanStream(tail, newAcc))
case NoEvent => (NoEvent, scanStream(tail, acc))
}
}
}
}
scanStream(getEvents, initial)
})

def take(n: Int): List[Event[A]] = getEvents.take(n).toList

def collectValues: List[A] = getEvents.take(100).toList.collect {
case Value(a) => a
}
}

object EventStream {
def fromValues[A](values: List[A]): EventStream[A] = new EventStream(() => {
LazyStream.fromList(values.map(Value(_)))
})

def periodic[A](value: A, period: Int): EventStream[A] = new EventStream(() => {
def periodicStream(counter: Int): LazyStream[Event[A]] = LazyStream {
val event = if (counter % period == 0) Value(value) else NoEvent
Some((event, periodicStream(counter + 1)))
}
periodicStream(0)
})

def mouse: EventStream[String] = new EventStream(() => {
val actions = List("click", "move", "drag", "scroll")
LazyStream.iterate(0)(_ + 1).map { i =>
if (i % 3 == 0) Value(actions(i % actions.length))
else NoEvent
}
})
}

// Demonstrate reactive programming
println("Creating reactive event streams:")

val numbers = EventStream.fromValues((1 to 10).toList)
val doubledNumbers = numbers.map(_ * 2)
val evenNumbers = doubledNumbers.filter(_ % 4 == 0)

println(s"Original numbers: ${numbers.collectValues}")
println(s"Doubled numbers: ${doubledNumbers.collectValues}")
println(s"Even doubled numbers: ${evenNumbers.collectValues}")

// Scan example (running sum)
val runningSum = numbers.scan(0)(_ + _)
println(s"Running sum: ${runningSum.collectValues}")

// Mouse event simulation
val mouseEvents = EventStream.mouse
val clickEvents = mouseEvents.filter(_ == "click")
println("First 10 mouse events:")
mouseEvents.take(10).zipWithIndex.foreach { case (event, index) =>
println(s"  Event $index: $event")
}

// 6. Lazy Data Processing Pipeline
println("\n6. Large-Scale Data Processing Pipeline")

case class DataRecord(id: Int, category: String, value: Double, timestamp: Long)

class LazyDataProcessor {
def generateLargeDataset(size: Int): LazyStream[DataRecord] = {
val categories = List("A", "B", "C", "D", "E")
val random = new scala.util.Random(42)

LazyStream.iterate(0)(_ + 1).take(size).map { i =>
DataRecord(
id = i,
category = categories(i % categories.length),
value = random.nextDouble() * 1000,
timestamp = System.currentTimeMillis() + i * 1000
)
}
}

def lazyAggregate[K](data: LazyStream[DataRecord], keyExtractor: DataRecord => K): Map[K, (Int, Double, Double)] = {
val aggregates = mutable.Map[K, (Int, Double, Double)]()

data.foreach { record =>
val key = keyExtractor(record)
val (count, sum, max) = aggregates.getOrElse(key, (0, 0.0, 0.0))
aggregates(key) = (count + 1, sum + record.value, math.max(max, record.value))
}

aggregates.toMap
}

def lazyWindow(data: LazyStream[DataRecord], windowSize: Int): LazyStream[List[DataRecord]] = {
data.sliding(windowSize)
}

def processInBatches[A](data: LazyStream[A], batchSize: Int)(processor: List[A] => Unit): Unit = {
var batch = List.empty[A]
var count = 0

data.foreach { item =>
batch = item :: batch
count += 1

if (count >= batchSize) {
processor(batch.reverse)
batch = List.empty[A]
count = 0
}
}

// Process remaining items
if (batch.nonEmpty) {
processor(batch.reverse)
}
}
}

val processor = new LazyDataProcessor()

println("Generating large dataset (1M records)...")
val largeDataset = processor.generateLargeDataset(1000000)

println("Processing first 1000 records...")
val sample = largeDataset.take(1000)

// Aggregate by category
val categoryAggregates = processor.lazyAggregate(sample, _.category)
println("Aggregates by category:")
categoryAggregates.foreach { case (category, (count, sum, max)) =>
println(f"  $category: count=$count%,d, sum=$sum%,.2f, max=$max%.2f, avg=${sum/count}%.2f")
}

// Window processing
println("\nProcessing in sliding windows of size 100:")
val windows = processor.lazyWindow(sample, 100)
val windowStats = windows.take(5).toList.zipWithIndex

windowStats.foreach { case (window, index) =>
val avgValue = window.map(_.value).sum / window.size
val categories = window.map(_.category).distinct.size
println(f"  Window $index: ${window.size} records, avg value=$avgValue%.2f, $categories categories")
}

// Batch processing simulation
println("\nBatch processing simulation:")
processor.processInBatches(sample, 250) { batch =>
val totalValue = batch.map(_.value).sum
println(f"  Processed batch of ${batch.size} records, total value: $totalValue%,.2f")
}

println("\n=== Advanced Lazy Evaluation Framework Complete! ===")
println("Advanced concepts demonstrated:")
println("- Custom memoized lazy data structures")
println("- Advanced streaming algorithms (sort, group, window)")
println("- Lazy dynamic programming optimization")
println("- Lazy graph traversal algorithms")
println("- Functional reactive programming with lazy streams")
println("- Large-scale lazy data processing pipelines")
println("- Memory-efficient processing of infinite data structures")
}
```

Comments

Be the first to comment on this lesson!

Lazy Evaluation and Streams: Efficient Data Processing

Introduction

Understanding Lazy Evaluation

Lazy Values

Lazy Function Parameters

Views: Lazy Collections

Creating and Using Views

LazyList (Streams)

Creating and Working with Streams

Practical Applications

Large Data Processing

Memory-Efficient Algorithms

Best Practices and Performance Tips

When to Use Lazy Evaluation

Summary

What's Next

🚀 Practice Exercise

Instructions

Interactive Playground

Comments

Sign In

Sign Up

Reset Password

Sign Out

Reset Password