Scala Scala Toolkit Libraries JSON HTTP File I/O os-lib upickle sttp Ecosystem

August 31, 2025

The Scala Toolkit: Your First Essential Library

An introduction to the Scala Toolkit, a curated set of libraries for common tasks like reading files, parsing JSON, and making HTTP requests.

Lesson 10 🚀 Practice

The Scala Toolkit: Your First Essential Library

Introduction

The Scala Toolkit is a curated collection of libraries that makes common programming tasks easier and more enjoyable. It's designed to be the "batteries included" experience for Scala, providing high-quality, well-tested libraries for everyday tasks like file I/O, JSON processing, HTTP requests, and more.

In this lesson, you'll learn about the Scala Toolkit, how to set it up in your projects, and explore its most useful components. This will be your first step into Scala's rich ecosystem of libraries and tools.

What is the Scala Toolkit?

The Scala Toolkit is an initiative by the Scala organization to provide:

Curated Libraries: Hand-picked, high-quality libraries for common tasks
Consistent API: Libraries that work well together
Easy Setup: Simple dependency management
Good Documentation: Clear examples and guides
Community Endorsed: Libraries recommended by the Scala community

Core Components

The toolkit includes several essential libraries:

os-lib: For file system operations and running processes
upickle: For JSON serialization and deserialization
sttp: For HTTP client operations
ujson: For JSON parsing and manipulation
munit: For testing (covered in later lessons)

Setting Up the Scala Toolkit

Adding to Your Project

In your build.sbt file:

// For Scala 3
libraryDependencies += "org.scala-lang" %% "toolkit" % "0.2.1"

// For Scala 2.13
libraryDependencies += "org.scala-lang" %% "toolkit" % "0.2.1"

// For testing support (optional)
libraryDependencies += "org.scala-lang" %% "toolkit-test" % "0.2.1" % Test

Simple sbt Project Setup

Here's a complete minimal build.sbt:

ThisBuild / scalaVersion := "3.3.1"
ThisBuild / version := "0.1.0"

lazy val root = (project in file("."))
  .settings(
    name := "scala-toolkit-example",
    libraryDependencies += "org.scala-lang" %% "toolkit" % "0.2.1"
  )

Working with Files: os-lib

The os-lib library makes file operations simple and safe.

Basic File Operations

import os.*

// Reading files
val content = os.read(os.pwd / "example.txt")
println(content)

// Writing files
os.write(os.pwd / "output.txt", "Hello, Scala Toolkit!")

// Check if file exists
if (os.exists(os.pwd / "config.json")) {
  println("Config file found!")
}

// List directory contents
val files = os.list(os.pwd)
files.foreach(println)

// Create directories
os.makeDir.all(os.pwd / "data" / "processed")

Path Manipulation

import os.*

// Working with paths
val projectRoot = os.pwd
val dataDir = projectRoot / "data"
val inputFile = dataDir / "input.csv"
val outputFile = dataDir / "processed" / "output.json"

println(s"Project root: $projectRoot")
println(s"Input file: $inputFile")
println(s"File name: ${inputFile.last}")
println(s"File extension: ${inputFile.ext}")

// Relative paths
val relativeToData = inputFile.relativeTo(dataDir)
println(s"Relative path: $relativeToData")

File Information

import os.*
import java.time.Instant

val file = os.pwd / "example.txt"

if (os.exists(file)) {
  // File properties
  val size = os.size(file)
  val modified = Instant.ofEpochMilli(os.mtime(file))
  val isDir = os.isDir(file)
  val isFile = os.isFile(file)

  println(s"Size: $size bytes")
  println(s"Modified: $modified")
  println(s"Is directory: $isDir")
  println(s"Is file: $isFile")

  // Permissions (Unix-like systems)
  if (!os.isWindows) {
    val perms = os.perms(file)
    println(s"Permissions: $perms")
  }
}

Advanced File Operations

import os.*

// Copy files and directories
os.copy(os.pwd / "source.txt", os.pwd / "backup" / "source.txt")

// Move/rename files
os.move(os.pwd / "old_name.txt", os.pwd / "new_name.txt")

// Remove files and directories
os.remove(os.pwd / "temp.txt")
os.remove.all(os.pwd / "temp_directory")  // Remove directory and contents

// Find files
val scalaFiles = os.walk(os.pwd)
  .filter(_.ext == "scala")
  .filter(os.isFile)

scalaFiles.foreach(file => println(s"Found Scala file: $file"))

// Find with pattern
val logFiles = os.walk(os.pwd / "logs")
  .filter(_.last.startsWith("app-"))
  .filter(_.ext == "log")

Working with JSON: upickle and ujson

JSON Parsing with ujson

import ujson.*

// Parse JSON string
val jsonString = """{"name": "Alice", "age": 30, "active": true}"""
val json = ujson.read(jsonString)

// Access values
val name = json("name").str
val age = json("age").num.toInt
val active = json("active").bool

println(s"Name: $name, Age: $age, Active: $active")

// Working with arrays
val arrayJson = """[1, 2, 3, 4, 5]"""
val numbers = ujson.read(arrayJson).arr.map(_.num.toInt)
println(s"Numbers: ${numbers.mkString(", ")}")

// Nested objects
val complexJson = """{
  "user": {
    "name": "Bob",
    "preferences": {
      "theme": "dark",
      "notifications": true
    }
  }
}"""

val parsed = ujson.read(complexJson)
val theme = parsed("user")("preferences")("theme").str
println(s"Theme: $theme")

JSON Serialization with upickle

import upickle.default.*

// Simple case class
case class Person(name: String, age: Int, active: Boolean)

// Serialize to JSON
val person = Person("Alice", 30, true)
val jsonString = write(person)
println(jsonString)  // {"name":"Alice","age":30,"active":true}

// Deserialize from JSON
val personFromJson = read[Person](jsonString)
println(personFromJson)  // Person(Alice,30,true)

// Working with collections
val people = List(
  Person("Alice", 30, true),
  Person("Bob", 25, false),
  Person("Charlie", 35, true)
)

val peopleJson = write(people)
println(peopleJson)

val peopleFromJson = read[List[Person]](peopleJson)
peopleFromJson.foreach(println)

Advanced JSON Operations

import ujson.*
import upickle.default.*

// Custom serialization
case class User(id: Int, name: String, email: Option[String] = None)

implicit val userRW: ReadWriter[User] = macroRW

// Working with Option fields
val users = List(
  User(1, "Alice", Some("alice@example.com")),
  User(2, "Bob", None),
  User(3, "Charlie", Some("charlie@test.com"))
)

val usersJson = write(users, indent = 2)
println(usersJson)

// Manual JSON construction
val manualJson = ujson.Obj(
  "status" -> "success",
  "data" -> ujson.Obj(
    "users" -> users.length,
    "active" -> users.count(_.email.isDefined)
  ),
  "timestamp" -> System.currentTimeMillis()
)

println(manualJson.render(indent = 2))

HTTP Requests: sttp

Basic HTTP Requests

import sttp.client3.*

// Create a backend
val backend = HttpURLConnectionBackend()

// Simple GET request
val request = basicRequest.get(uri"https://api.github.com/users/scala")
val response = request.send(backend)

println(s"Status: ${response.code}")
println(s"Body: ${response.body}")

// Clean up
backend.close()

Working with JSON APIs

import sttp.client3.*
import ujson.*

case class GitHubUser(login: String, name: Option[String], public_repos: Int)

implicit val githubUserRW: ReadWriter[GitHubUser] = macroRW

def fetchGitHubUser(username: String): Either[String, GitHubUser] = {
  val backend = HttpURLConnectionBackend()

  try {
    val request = basicRequest
      .get(uri"https://api.github.com/users/$username")
      .header("User-Agent", "Scala-Toolkit-Example")

    val response = request.send(backend)

    response.code.code match {
      case 200 => 
        Right(read[GitHubUser](response.body.getOrElse("")))
      case 404 => 
        Left(s"User $username not found")
      case code => 
        Left(s"API error: $code")
    }
  } catch {
    case ex: Exception => Left(s"Request failed: ${ex.getMessage}")
  } finally {
    backend.close()
  }
}

// Usage
fetchGitHubUser("scala") match {
  case Right(user) => 
    println(s"User: ${user.login}, Repos: ${user.public_repos}")
  case Left(error) => 
    println(s"Error: $error")
}

POST Requests and Form Data

import sttp.client3.*
import ujson.*

val backend = HttpURLConnectionBackend()

// JSON POST request
val userData = ujson.Obj(
  "name" -> "Alice",
  "email" -> "alice@example.com"
)

val jsonRequest = basicRequest
  .post(uri"https://httpbin.org/post")
  .header("Content-Type", "application/json")
  .body(userData.render())

val jsonResponse = jsonRequest.send(backend)
println(s"JSON Response: ${jsonResponse.body}")

// Form data POST
val formRequest = basicRequest
  .post(uri"https://httpbin.org/post")
  .body(Map("name" -> "Bob", "email" -> "bob@example.com"))

val formResponse = formRequest.send(backend)
println(s"Form Response: ${formResponse.body}")

backend.close()

Practical Examples

Example 1: Configuration Manager

import os.*
import upickle.default.*
import ujson.*

case class DatabaseConfig(
  host: String, 
  port: Int, 
  database: String,
  username: String,
  password: String
)

case class AppConfig(
  appName: String,
  version: String,
  debug: Boolean,
  database: DatabaseConfig
)

implicit val dbConfigRW: ReadWriter[DatabaseConfig] = macroRW
implicit val appConfigRW: ReadWriter[AppConfig] = macroRW

object ConfigManager {
  private val configFile = os.pwd / "config.json"

  def loadConfig(): Option[AppConfig] = {
    if (os.exists(configFile)) {
      try {
        val content = os.read(configFile)
        Some(read[AppConfig](content))
      } catch {
        case ex: Exception =>
          println(s"Error reading config: ${ex.getMessage}")
          None
      }
    } else {
      None
    }
  }

  def saveConfig(config: AppConfig): Boolean = {
    try {
      val jsonContent = write(config, indent = 2)
      os.write(configFile, jsonContent)
      true
    } catch {
      case ex: Exception =>
        println(s"Error saving config: ${ex.getMessage}")
        false
    }
  }

  def createDefaultConfig(): AppConfig = {
    AppConfig(
      appName = "MyScalaApp",
      version = "1.0.0",
      debug = true,
      database = DatabaseConfig(
        host = "localhost",
        port = 5432,
        database = "myapp",
        username = "user",
        password = "password"
      )
    )
  }
}

// Usage
val config = ConfigManager.loadConfig().getOrElse {
  println("No config found, creating default...")
  val defaultConfig = ConfigManager.createDefaultConfig()
  ConfigManager.saveConfig(defaultConfig)
  defaultConfig
}

println(s"App: ${config.appName} v${config.version}")
println(s"Database: ${config.database.host}:${config.database.port}")

Example 2: Web Scraper and Data Processor

import os.*
import sttp.client3.*
import ujson.*
import upickle.default.*

case class Article(title: String, url: String, timestamp: Long)

implicit val articleRW: ReadWriter[Article] = macroRW

object NewsScraper {
  private val backend = HttpURLConnectionBackend()
  private val dataDir = os.pwd / "data"

  def fetchHackerNewsTop(): List[Article] = {
    try {
      // Get top story IDs
      val topStoriesRequest = basicRequest
        .get(uri"https://hacker-news.firebaseio.com/v0/topstories.json")

      val topStoriesResponse = topStoriesRequest.send(backend)
      val storyIds = read[List[Int]](topStoriesResponse.body.getOrElse("[]"))

      // Fetch first 10 stories
      storyIds.take(10).flatMap { id =>
        try {
          val storyRequest = basicRequest
            .get(uri"https://hacker-news.firebaseio.com/v0/item/$id.json")

          val storyResponse = storyRequest.send(backend)
          val storyJson = ujson.read(storyResponse.body.getOrElse("{}"))

          Some(Article(
            title = storyJson("title").str,
            url = storyJson.obj.get("url").map(_.str).getOrElse(s"https://news.ycombinator.com/item?id=$id"),
            timestamp = System.currentTimeMillis()
          ))
        } catch {
          case ex: Exception =>
            println(s"Error fetching story $id: ${ex.getMessage}")
            None
        }
      }
    } catch {
      case ex: Exception =>
        println(s"Error fetching top stories: ${ex.getMessage}")
        List.empty
    }
  }

  def saveArticles(articles: List[Article]): Unit = {
    os.makeDir.all(dataDir)
    val filename = s"articles-${System.currentTimeMillis()}.json"
    val filepath = dataDir / filename

    val jsonContent = write(articles, indent = 2)
    os.write(filepath, jsonContent)
    println(s"Saved ${articles.length} articles to $filepath")
  }

  def loadAllArticles(): List[Article] = {
    if (!os.exists(dataDir)) return List.empty

    os.list(dataDir)
      .filter(_.ext == "json")
      .filter(_.last.startsWith("articles-"))
      .flatMap { file =>
        try {
          val content = os.read(file)
          read[List[Article]](content)
        } catch {
          case ex: Exception =>
            println(s"Error reading $file: ${ex.getMessage}")
            List.empty
        }
      }
      .toList
  }

  def cleanup(): Unit = {
    backend.close()
  }
}

// Usage
val articles = NewsScraper.fetchHackerNewsTop()
println(s"Fetched ${articles.length} articles")

articles.foreach { article =>
  println(s"- ${article.title}")
  println(s"  ${article.url}")
}

NewsScraper.saveArticles(articles)

// Load and display all saved articles
val allArticles = NewsScraper.loadAllArticles()
println(s"\nTotal saved articles: ${allArticles.length}")

NewsScraper.cleanup()

Example 3: Log File Analyzer

import os.*
import ujson.*
import upickle.default.*
import java.time.LocalDateTime
import java.time.format.DateTimeFormatter

case class LogEntry(
  timestamp: LocalDateTime,
  level: String,
  message: String,
  thread: Option[String] = None
)

case class LogAnalysis(
  totalEntries: Int,
  levelCounts: Map[String, Int],
  errorMessages: List[String],
  timeRange: (LocalDateTime, LocalDateTime),
  mostActiveHour: Int
)

implicit val localDateTimeRW: ReadWriter[LocalDateTime] = readwriter[String].bimap(
  dt => dt.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME),
  str => LocalDateTime.parse(str, DateTimeFormatter.ISO_LOCAL_DATE_TIME)
)

implicit val logEntryRW: ReadWriter[LogEntry] = macroRW
implicit val logAnalysisRW: ReadWriter[LogAnalysis] = macroRW

object LogAnalyzer {
  private val timestampPattern = """(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})""".r
  private val logPattern = """(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) \[(\w+)\](?: \[([^\]]+)\])? (.+)""".r

  def parseLogFile(logFile: os.Path): List[LogEntry] = {
    if (!os.exists(logFile)) {
      println(s"Log file not found: $logFile")
      return List.empty
    }

    val lines = os.read.lines(logFile)

    lines.flatMap { line =>
      line match {
        case logPattern(timestamp, level, thread, message) =>
          try {
            val dt = LocalDateTime.parse(timestamp, DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"))
            Some(LogEntry(dt, level, message, Option(thread)))
          } catch {
            case ex: Exception =>
              println(s"Error parsing timestamp: $timestamp")
              None
          }
        case _ => None
      }
    }.toList
  }

  def analyzeEntries(entries: List[LogEntry]): LogAnalysis = {
    val levelCounts = entries.groupBy(_.level).view.mapValues(_.size).toMap
    val errorMessages = entries.filter(_.level == "ERROR").map(_.message)
    val timestamps = entries.map(_.timestamp)
    val timeRange = (timestamps.min, timestamps.max)
    val hourCounts = entries.groupBy(_.timestamp.getHour).view.mapValues(_.size)
    val mostActiveHour = hourCounts.maxBy(_._2)._1

    LogAnalysis(
      totalEntries = entries.length,
      levelCounts = levelCounts,
      errorMessages = errorMessages,
      timeRange = timeRange,
      mostActiveHour = mostActiveHour
    )
  }

  def generateReport(analysis: LogAnalysis): String = {
    val (startTime, endTime) = analysis.timeRange

    s"""
       |Log Analysis Report
       |==================
       |
       |Total Entries: ${analysis.totalEntries}
       |Time Range: $startTime to $endTime
       |Most Active Hour: ${analysis.mostActiveHour}:00
       |
       |Level Distribution:
       |${analysis.levelCounts.toSeq.sortBy(-_._2).map { case (level, count) =>
         s"  $level: $count"
       }.mkString("\n")}
       |
       |Error Messages:
       |${if (analysis.errorMessages.nonEmpty) 
           analysis.errorMessages.zipWithIndex.map { case (msg, idx) =>
             s"  ${idx + 1}. $msg"
           }.mkString("\n")
         else "  No errors found"
       }
       |""".stripMargin
  }

  def saveAnalysis(analysis: LogAnalysis, outputFile: os.Path): Unit = {
    val jsonContent = write(analysis, indent = 2)
    os.write(outputFile, jsonContent)
    println(s"Analysis saved to $outputFile")
  }
}

// Create sample log file for testing
val sampleLogContent = """2023-08-31 10:15:30 [INFO] Application started
2023-08-31 10:15:35 [DEBUG] Loading configuration from config.json
2023-08-31 10:15:40 [INFO] Database connection established
2023-08-31 10:15:45 [WARN] Deprecated API method used
2023-08-31 10:16:00 [ERROR] Failed to process user request: timeout
2023-08-31 10:16:15 [INFO] Processing 100 records
2023-08-31 10:16:30 [ERROR] Database connection lost
2023-08-31 10:16:45 [INFO] Reconnected to database
2023-08-31 10:17:00 [DEBUG] Cache hit rate: 85%"""

val logFile = os.pwd / "sample.log"
os.write(logFile, sampleLogContent)

// Analyze the log file
val entries = LogAnalyzer.parseLogFile(logFile)
val analysis = LogAnalyzer.analyzeEntries(entries)
val report = LogAnalyzer.generateReport(analysis)

println(report)

// Save analysis
LogAnalyzer.saveAnalysis(analysis, os.pwd / "log-analysis.json")

// Clean up
os.remove(logFile)

Best Practices

1. Resource Management

import sttp.client3.*

// Always close backends
def makeHttpRequest(url: String): String = {
  val backend = HttpURLConnectionBackend()
  try {
    val request = basicRequest.get(uri"$url")
    val response = request.send(backend)
    response.body.getOrElse("")
  } finally {
    backend.close()
  }
}

// Or use resource management
def withBackend[T](f: SttpBackend[Identity, Any] => T): T = {
  val backend = HttpURLConnectionBackend()
  try {
    f(backend)
  } finally {
    backend.close()
  }
}

2. Error Handling

import scala.util.{Try, Success, Failure}

def safeFileOperation(file: os.Path): Either[String, String] = {
  Try {
    os.read(file)
  } match {
    case Success(content) => Right(content)
    case Failure(exception) => Left(s"Failed to read file: ${exception.getMessage}")
  }
}

3. Configuration Validation

case class Config(host: String, port: Int, timeout: Int) {
  require(host.nonEmpty, "Host cannot be empty")
  require(port > 0 && port < 65536, "Port must be between 1 and 65535")
  require(timeout > 0, "Timeout must be positive")
}

def loadValidatedConfig(): Either[String, Config] = {
  Try {
    val configFile = os.pwd / "config.json"
    val content = os.read(configFile)
    read[Config](content)
  } match {
    case Success(config) => Right(config)
    case Failure(exception) => Left(s"Config error: ${exception.getMessage}")
  }
}

Summary

In this lesson, you've learned about the Scala Toolkit and its core components:

✅ Scala Toolkit: Curated collection of essential libraries
✅ os-lib: Safe and intuitive file system operations
✅ upickle/ujson: JSON serialization and parsing
✅ sttp: HTTP client for API interactions
✅ Integration: How libraries work together seamlessly
✅ Real-world Examples: Configuration management, web scraping, log analysis
✅ Best Practices: Resource management and error handling

The Scala Toolkit provides a solid foundation for building real applications. These libraries handle common tasks elegantly while maintaining type safety and functional programming principles.

What's Next

In the next lesson, we'll start Part 2: "Object-Oriented Programming in Scala" with "Classes and Objects: The Blueprint of Your Application." You'll learn how to define classes, create instances, and understand the role of constructors in Scala.

This marks an important transition from basic language features to building structured, object-oriented applications that can model real-world domains.

Ready to start building with objects and classes? Let's continue!

🚀 Practice Exercise

Put your knowledge to practice with hands-on coding exercises

Advanced ⏱️ 1h 15min

Instructions

Build an advanced functional data processing framework that demonstrates sophisticated Scala Toolkit features including custom collections, type-level programming, advanced pattern matching, and concurrent processing.

Create a `FunctionalDataFramework` that:
1. Implements custom collection types with specialized operations
2. Uses advanced type system features (generics, variance, type bounds)
3. Demonstrates monad transformers and effect composition
4. Implements streaming data processing with backpressure
5. Uses reflection and macros for runtime code generation
6. Creates a mini query language with parser combinators
7. Implements concurrent data processing with Futures and actors

Interactive Playground

⚠️ Spoiler Alert! Try to solve the exercise yourself first. Learning happens through practice!

import scala.concurrent.{Future, ExecutionContext, Promise}
import scala.concurrent.duration._
import scala.util.{Try, Success, Failure, Random}
import scala.collection.mutable
import scala.reflect.runtime.universe._
import scala.util.matching.Regex
import scala.util.parsing.combinator._
import java.util.concurrent.atomic.AtomicReference
import java.time.LocalDateTime
import scala.annotation.tailrec

// Advanced type system demonstrations
object TypeSystem {
  // Variance examples
  trait Producer[+T] {
    def produce(): T
  }

  trait Consumer[-T] {
    def consume(item: T): Unit
  }

  trait Processor[T] {
    def process(input: T): T
  }

  // Type bounds
  def processNumeric[T <: AnyVal : Numeric](items: List[T]): T = {
    val numeric = implicitly[Numeric[T]]
    items.reduce(numeric.plus)
  }

  // Higher-kinded types
  trait Functor[F[_]] {
    def map[A, B](fa: F[A])(f: A => B): F[B]
  }

  implicit val optionFunctor: Functor[Option] = new Functor[Option] {
    def map[A, B](fa: Option[A])(f: A => B): Option[B] = fa.map(f)
  }

  implicit val listFunctor: Functor[List] = new Functor[List] {
    def map[A, B](fa: List[A])(f: A => B): List[B] = fa.map(f)
  }

  def mapTwice[F[_]: Functor, A, B, C](fa: F[A])(f: A => B)(g: B => C): F[C] = {
    val functor = implicitly[Functor[F]]
    functor.map(functor.map(fa)(f))(g)
  }
}

// Custom collection types
class RingBuffer[T](capacity: Int) extends Iterable[T] {
  private val buffer = new Array[Any](capacity)
  private var head = 0
  private var size = 0

  def add(item: T): Unit = {
    buffer((head + size) % capacity) = item
    if (size < capacity) {
      size += 1
    } else {
      head = (head + 1) % capacity
    }
  }

  def iterator: Iterator[T] = new Iterator[T] {
    private var index = 0

    def hasNext: Boolean = index < size

    def next(): T = {
      if (!hasNext) throw new NoSuchElementException
      val item = buffer((head + index) % capacity).asInstanceOf[T]
      index += 1
      item
    }
  }

  def latest(n: Int): List[T] = {
    val count = math.min(n, size)
    (0 until count).map(i => buffer((head + size - count + i) % capacity).asInstanceOf[T]).toList
  }
}

// Trie data structure for efficient string operations
class Trie {
  private class TrieNode {
    val children = mutable.Map[Char, TrieNode]()
    var isEndOfWord = false
    var count = 0
  }

  private val root = new TrieNode

  def insert(word: String): Unit = {
    var current = root
    for (char <- word) {
      current.children.getOrElseUpdate(char, new TrieNode)
      current = current.children(char)
    }
    current.isEndOfWord = true
    current.count += 1
  }

  def search(word: String): Boolean = {
    var current = root
    for (char <- word) {
      current.children.get(char) match {
        case Some(node) => current = node
        case None => return false
      }
    }
    current.isEndOfWord
  }

  def startsWith(prefix: String): List[String] = {
    var current = root
    for (char <- prefix) {
      current.children.get(char) match {
        case Some(node) => current = node
        case None => return List.empty
      }
    }

    def collectWords(node: TrieNode, currentPrefix: String): List[String] = {
      val words = mutable.ListBuffer[String]()
      if (node.isEndOfWord) {
        words += currentPrefix
      }
      for ((char, childNode) <- node.children) {
        words ++= collectWords(childNode, currentPrefix + char)
      }
      words.toList
    }

    collectWords(current, prefix)
  }
}

// Query DSL
sealed trait Query[T]
case class Filter[T](predicate: T => Boolean) extends Query[T]
case class Map[T, U](function: T => U) extends Query[U]
case class GroupBy[T, K](keyFunction: T => K) extends Query[(K, List[T])]
case class Aggregate[T, R](aggregator: List[T] => R) extends Query[R]
case class Take[T](n: Int) extends Query[T]
case class Sort[T](ordering: Ordering[T]) extends Query[T]

// Query processor
class QueryProcessor[T] {
  def execute(data: List[T], query: Query[_]): Any = query match {
    case Filter(pred) => data.filter(pred.asInstanceOf[T => Boolean])
    case Map(func) => data.map(func.asInstanceOf[T => Any])
    case GroupBy(keyFunc) => data.groupBy(keyFunc.asInstanceOf[T => Any]).toList
    case Aggregate(agg) => agg.asInstanceOf[List[T] => Any](data)
    case Take(n) => data.take(n)
    case Sort(ord) => data.sorted(ord.asInstanceOf[Ordering[T]])
  }

  def chain(data: List[T], queries: Query[_]*): Any = {
    queries.foldLeft(data.asInstanceOf[Any]) { (currentData, query) =>
      currentData match {
        case list: List[_] => execute(list.asInstanceOf[List[T]], query)
        case other => other
      }
    }
  }
}

// Advanced pattern matching with custom extractors
object PersonExtractor {
  case class Person(name: String, age: Int, email: String, salary: Double)

  object YoungProfessional {
    def unapply(person: Person): Option[(String, Int)] = {
      if (person.age >= 22 && person.age <= 30 && person.salary > 50000) {
        Some((person.name, person.age))
      } else None
    }
  }

  object EmailDomain {
    def unapply(email: String): Option[String] = {
      val atIndex = email.lastIndexOf('@')
      if (atIndex > 0 && atIndex < email.length - 1) {
        Some(email.substring(atIndex + 1))
      } else None
    }
  }

  object SalaryRange {
    def unapply(salary: Double): Option[String] = salary match {
      case s if s < 50000 => Some("Entry Level")
      case s if s < 100000 => Some("Mid Level")
      case s if s < 200000 => Some("Senior Level")
      case _ => Some("Executive Level")
    }
  }

  def categorizePerson(person: Person): String = person match {
    case YoungProfessional(name, age) => s"$name is a young professional ($age years old)"
    case Person(name, age, EmailDomain("gmail.com"), SalaryRange(range)) =>
      s"$name ($age) uses Gmail and is $range"
    case Person(name, age, _, SalaryRange("Executive Level")) =>
      s"$name ($age) is an executive"
    case Person(name, age, _, _) =>
      s"$name ($age) is a regular employee"
  }
}

// Streaming data processor with backpressure
class StreamProcessor[T, R] {
  private val buffer = mutable.Queue[T]()
  private val maxBufferSize = 1000
  private var isProcessing = false

  def process(data: Iterator[T], processor: T => R, batchSize: Int = 10): Iterator[R] = {
    new Iterator[R] {
      private val results = mutable.Queue[R]()

      def hasNext: Boolean = data.hasNext || results.nonEmpty

      def next(): R = {
        if (results.isEmpty) {
          fillBuffer()
        }
        if (results.nonEmpty) {
          results.dequeue()
        } else {
          throw new NoSuchElementException
        }
      }

      private def fillBuffer(): Unit = {
        var count = 0
        while (data.hasNext && count < batchSize && buffer.size < maxBufferSize) {
          buffer.enqueue(data.next())
          count += 1
        }

        while (buffer.nonEmpty) {
          val item = buffer.dequeue()
          try {
            results.enqueue(processor(item))
          } catch {
            case _: Exception => // Skip failed items
          }
        }
      }
    }
  }
}

// Parser combinators for simple query language
class QueryParser extends RegexParsers {
  def number: Parser[Double] = """\d+(\.\d+)?""".r ^^ (_.toDouble)
  def string: Parser[String] = """"[^"]*"""".r ^^ (s => s.substring(1, s.length - 1))
  def identifier: Parser[String] = """[a-zA-Z][a-zA-Z0-9_]*""".r

  def comparison: Parser[(String, String, Double)] =
    identifier ~ (">" | "<" | "=" | ">=" | "<=") ~ number ^^ {
      case field ~ op ~ value => (field, op, value)
    }

  def query: Parser[List[(String, String, Double)]] =
    repsep(comparison, "AND" | "OR")

  def parse(input: String): Try[List[(String, String, Double)]] = {
    parseAll(query, input) match {
      case Success(result, _) => scala.util.Success(result)
      case failure => Failure(new RuntimeException(s"Parse error: $failure"))
    }
  }
}

// Concurrent processing framework
class ConcurrentProcessor(implicit ec: ExecutionContext) {
  def processParallel[T, R](data: List[T], processor: T => R, parallelism: Int = 4): Future[List[R]] = {
    val chunks = data.grouped(math.max(1, data.length / parallelism)).toList

    val futures = chunks.map { chunk =>
      Future {
        chunk.map(processor)
      }
    }

    Future.sequence(futures).map(_.flatten)
  }

  def pipeline[T, R](
    data: List[T],
    stages: List[T => T],
    finalProcessor: T => R
  ): Future[List[R]] = {
    Future {
      data.map { item =>
        val processed = stages.foldLeft(item) { (current, stage) =>
          stage(current)
        }
        finalProcessor(processed)
      }
    }
  }

  def rateLimit[T, R](
    data: Iterator[T],
    processor: T => Future[R],
    maxConcurrency: Int = 5,
    delayBetweenBatches: Duration = 100.millis
  ): Future[List[R]] = {
    val semaphore = new java.util.concurrent.Semaphore(maxConcurrency)
    val results = mutable.ListBuffer[Future[R]]()

    data.grouped(maxConcurrency).foreach { batch =>
      batch.foreach { item =>
        val future = Future {
          semaphore.acquire()
          try {
            processor(item)
          } finally {
            semaphore.release()
          }
        }.flatten

        results += future
      }
      Thread.sleep(delayBetweenBatches.toMillis)
    }

    Future.sequence(results.toList)
  }
}

// Main framework
object FunctionalDataFramework extends App {
  implicit val ec: ExecutionContext = ExecutionContext.global

  // Sample data
  val people = List(
    PersonExtractor.Person("Alice Johnson", 28, "alice@gmail.com", 75000),
    PersonExtractor.Person("Bob Smith", 35, "bob@company.com", 95000),
    PersonExtractor.Person("Carol Davis", 42, "carol@enterprise.org", 150000),
    PersonExtractor.Person("David Brown", 26, "david@gmail.com", 55000),
    PersonExtractor.Person("Eve Wilson", 38, "eve@startup.io", 120000)
  )

  println("Advanced Functional Data Processing Framework")
  println("===========================================")

  // Type system demonstrations
  println("=== Type System Features ===")
  val numbers = List(1, 2, 3, 4, 5)
  val result = TypeSystem.processNumeric(numbers)
  println(s"Sum of numbers: $result")

  val optionResult = TypeSystem.mapTwice(Some(5))(_ * 2)(_ + 1)
  println(s"Option transformation: $optionResult")

  // Custom collections
  println("\n=== Custom Collections ===")
  val ringBuffer = new RingBuffer[Int](5)
  (1 to 8).foreach(ringBuffer.add)
  println(s"Ring buffer contents: ${ringBuffer.mkString(", ")}")
  println(s"Latest 3 items: ${ringBuffer.latest(3)}")

  val trie = new Trie()
  List("car", "card", "care", "careful", "cat", "catch").foreach(trie.insert)
  println(s"Words starting with 'car': ${trie.startsWith("car")}")

  // Query DSL
  println("\n=== Query DSL ===")
  val processor = new QueryProcessor[PersonExtractor.Person]()
  val youngPeople = processor.execute(
    people,
    Filter[PersonExtractor.Person](_.age < 35)
  ).asInstanceOf[List[PersonExtractor.Person]]

  println("Young people:")
  youngPeople.foreach(p => println(s"  ${p.name} (${p.age})"))

  // Advanced pattern matching
  println("\n=== Advanced Pattern Matching ===")
  people.foreach { person =>
    println(PersonExtractor.categorizePerson(person))
  }

  // Streaming processing
  println("\n=== Streaming Processing ===")
  val streamProcessor = new StreamProcessor[Int, Int]()
  val dataStream = (1 to 100).iterator
  val processed = streamProcessor.process(dataStream, x => x * x * x, batchSize = 5)
  println(s"First 10 cubes: ${processed.take(10).mkString(", ")}")

  // Parser combinators
  println("\n=== Query Parser ===")
  val parser = new QueryParser()
  val queryString = "age > 30 AND salary >= 80000"
  parser.parse(queryString) match {
    case Success(conditions) =>
      println(s"Parsed query: $conditions")
    case Failure(error) =>
      println(s"Parse error: ${error.getMessage}")
  }

  // Concurrent processing
  println("\n=== Concurrent Processing ===")
  val concurrentProcessor = new ConcurrentProcessor()

  val heavyComputation = (n: Int) => {
    Thread.sleep(10) // Simulate work
    n * n
  }

  val largeDataset = (1 to 100).toList
  val start = System.currentTimeMillis()

  concurrentProcessor.processParallel(largeDataset, heavyComputation).foreach { results =>
    val end = System.currentTimeMillis()
    println(s"Processed ${results.length} items in ${end - start}ms")
    println(s"First 5 results: ${results.take(5).mkString(", ")}")
  }

  // Advanced aggregations
  println("\n=== Advanced Aggregations ===")
  val salaryStats = people.groupBy(p => if (p.age < 35) "Young" else "Experienced")
    .view.mapValues { group =>
      val salaries = group.map(_.salary)
      val avg = salaries.sum / salaries.length
      val max = salaries.max
      val min = salaries.min
      s"Avg: $$${avg}k, Range: $$${min}k-$$${max}k"
    }.toMap

  salaryStats.foreach { case (category, stats) =>
    println(s"$category employees: $stats")
  }

  // Memory-efficient processing
  println("\n=== Memory-Efficient Processing ===")
  def processLargeDataset(size: Int): Unit = {
    val iterator = Iterator.range(0, size)
    val result = iterator
      .filter(_ % 2 == 0)
      .map(x => x * x)
      .take(10)
      .sum

    println(s"Sum of first 10 even squares from $size items: $result")
  }

  processLargeDataset(1000000)

  println("\n=== Framework Summary ===")
  println("✓ Advanced type system features (variance, bounds, higher-kinded types)")
  println("✓ Custom collection implementations (RingBuffer, Trie)")
  println("✓ Type-safe query DSL with combinators")
  println("✓ Advanced pattern matching with custom extractors")
  println("✓ Streaming processing with backpressure handling")
  println("✓ Concurrent processing with rate limiting")
  println("✓ Memory-efficient lazy evaluation")
  println("✓ Parser combinators for domain-specific languages")
}

Comments

Be the first to comment on this lesson!

The Scala Toolkit: Your First Essential Library

Introduction

What is the Scala Toolkit?

Core Components

Setting Up the Scala Toolkit

Adding to Your Project

Simple sbt Project Setup

Working with Files: os-lib

Basic File Operations

Path Manipulation

File Information

Advanced File Operations

Working with JSON: upickle and ujson

JSON Parsing with ujson

JSON Serialization with upickle

Advanced JSON Operations

HTTP Requests: sttp

Basic HTTP Requests

Working with JSON APIs

POST Requests and Form Data

Practical Examples

Example 1: Configuration Manager

Example 2: Web Scraper and Data Processor

Example 3: Log File Analyzer

Best Practices

1. Resource Management

2. Error Handling

3. Configuration Validation

Summary

What's Next

🚀 Practice Exercise

Instructions

Interactive Playground

Comments

Sign In

Sign Up

Reset Password

Sign Out

Reset Password