\nThe state of the art

Within the Scala ecosystem the Akka-HTTP library is a popular choice for implementing server side backends for HTTP APIs. Another quite popular option is the Play framework but using a full blown web framework to just provide a thin API is overkill in most cases. As most services need a database the Slick library is another popular choice which completes the picture.

However while all mentioned libraries are battle tested and proven they still have problems.

",{"_223":224,"_42":43,"_46":47,"_44":45,"_64":65,"_225":226,"_262":221},"previewContent","

\n Foreword\n
- \n \n
  - \n About this book\n
  - \n Copyleft Notice\n
  \n
- \n Thanks\n
\n
\n Our use case\n
- \n Service specification\n
  - \n Data model\n
  - \n Database\n
  - \n HTTP API\n
  \n
\n
\n The state of the art\n
- \n Problems\n
\n
\n Maybe there is another way\n
- \n Impure implementation\n
  - \n Models\n
  - \n Database layer\n
  - \n Akka-HTTP routes\n
  \n
- \n Pure implementation\n
  - \n Pure configuration handling\n
  - \n Models\n
  - \n Database layer\n
  - \n http4s routes\n
  \n
- \n What about tests?\n
  - \n Testing the impure service\n
  - \n Testing the pure service\n
  \n
- \n Adding benchmarks\n
  - \n Our environment\n
  - \n Comparison\n
  - \n Summary\n
  \n
- \n Documenting your API\n
  - \n The lay of the land\n
  - \n Using types to describe an API\n
  - \n A pure implementation using tapir.\n
  \n
- \n Moving to Scala 3…\n
  - \n Step 1: Updating to 2.13.x\n
  - \n Step 2: Migrating to Scala 3\n
  \n
\n
\n Epilogue\n
\n Notes\n

\nForeword

\n\n

Thank you for your interest in this book. I hope you’ll have a good time reading it and learn something from it.

\n\n

\nAbout this book

\n\n

This book is intended for the intermediate Scala programmer who is interested in functional programming and works mainly on the web service backend side. Ideally she has experience with libraries like Akka HTTP¹ and Slick² which are in heavy use in that area.

\n\n

However maybe you have wondered if we can’t do better even though aforementioned projects are battle tested and proven.

\n\n

The answer to this can be found in this book which is intended to be read from cover to cover in the given order. Within the book the following libraries will be used: Cats³, Cats Effect⁴, http4s⁵, Doobie⁶, Refined⁷, fs2⁸, tapir⁹, Monocle¹⁰ and probably others. ;-)

\n\n

This edition includes a chapter about migrating the project to Scala 3. Which includes all the nasty issues that we tend to run into if we touch code after a longer time.

\n\n

Code and book source can be found in the following repository: https://github.com/jan0sch/pfhais

\n\n

\nCopyleft Notice

\n\n

This book uses the Creative Commons Attribution ShareAlike 4.0 International (CC BY-SA 4.0) license. The code snippets in this book are licensed under CC0 which means you can use them without restriction. Excerpts from libraries maintain their license.

\n\n

\nThanks

\n\n

I would like to thank my beloved wife and family who bear with me and make all of this possible.
Also I send a big thank you to alle the nice people from the Scala community which I’ve had the pleasure to meet. Special thanks go to Adam Warski (tapir), Frank S. Thomas (refined), Julien Truffaut (Monocle) and Ross A. Baker (http4s) for their help, advise and patience.

\n\n\n\n

\nOur use case

\n\n

For better understanding we will implement a small use case in both impure and pure way. The following section will outline the specification.

\n\n

\nService specification

\n\n

First we need to specify the exact scope and API of our service. We’ll design a service with a minimal API to keep things simple. It shall fulfil the following requirements.

\n\n

The service shall provide HTTP API endpoints for:

\n\n

the creation of a product data type identified by a unique id
adding translations for a product name by language code and unique id
returning the existing translations for a product
returning a list of all existing products with their translations

\n\n

\nData model

\n\n

We will keep the model very simple to avoid going overboard with the implementation.

\n\n

A language code shall be defined by the ISO 639-1 (e.g. a two letter code).
A translation shall contain a language code and a product name (non-empty string).
A product shall contain a unique id (UUID version 4) and a list of translations.

\n\n

\nDatabase

\n\n

The data will be stored in a relational database (RDBMS). Therefore we need to define the tables and relations within the database.

\n\n

\nThe products table

\n\n

The table products must contain only the unique id which is also the primary key.

\n\n

\nThe names table

\n\n

The table names must contain a column for the product id, one for the language code and one for the name. Its primary key is the combination of the product id and the language code. All columns must not be null. The relation to the products is realised by a foreign key constraint to the products table via the product id.

\n\n

\nHTTP API

\n\n

The HTTP API shall provide the following endpoints on the given paths:

\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n

Path	HTTP method	Function
`/products`	POST	Create a product.
`/products`	GET	Get all products and translations.
`/product/{UUID}`	PUT	Add translations.
`/product/{UUID}`	GET	Get all translations for the product.

\n\n

The data shall be encoded in JSON using the following specification:

\n\n\n

JSON for a translation

1 {\n2   \"lang\": \"ISO-639-1 Code\",\n3   \"name\": \"A non empty string.\"\n4 }\n

\n\n\n\n\n\n

JSON for a product

1 {\n2   \"id\": \"The-UUID-of-the-product\",\n3   \"names\": [\n4     // A list of translations.\n5   ]\n6 }\n

\n\n\n\n\n

This should be enough to get us started.

\n\n

\nThe state of the art

\n\n

However while all mentioned libraries are battle tested and proven they still have problems.

\n\n

\nProblems

\n\n

In the domain of functional programming we want referential transparency which we will define in the following way:

\n\n\n\n

Building on that we need pure functions which are

\n\n

only dependent on their input
have no side effects

\n\n

This means in turn that our functions will be referential transparent.

\n\n

But, the mentioned libraries are built upon the Future from Scala which uses eager evaluation and breaks referential transparency. Let’s look at an example.

\n\n\n

Future example 1

1 import scala.concurrent.Future\n2 import scala.concurrent.ExecutionContext.Implicits.global\n3 \n4 for {\n5   _ <- Future { println(\"Hi there!\") }\n6   _ <- Future { println(\"Hi there!\") }\n7 } yield ()\n

\n\n\n\n\n

The code above will print the text Hi there! two times. But how about the following one?

\n\n\n

Future example 2

1 import scala.concurrent.Future\n2 import scala.concurrent.ExecutionContext.Implicits.global\n3 \n4 val printF = Future { println(\"Hi there!\") }\n5 \n6 for {\n7   _ <- printF\n8   _ <- printF\n9 } yield ()\n

\n\n\n\n\n

Instead of printing the text two times it will print it only once even when there is no usage of printF at all (try omitting the for comprehension). This means that Future breaks referential transparency!

\n\n\n\n

\nMaybe there is another way

\n\n

If we want referential transparency, we must push the side effects to the boundaries of our system (program) which can be done by using lazy evaluation. Let’s repeat the previous example in a different way.

\n\n\n

IO example 1

1 import cats.effect.IO\n2 import cats.implicits._\n3 \n4 val effect = for {\n5   _ <- IO(println(\"Hi there!\"))\n6   _ <- IO(println(\"Hi there!\"))\n7 } yield ()\n

\n\n\n\n\n

The above code will produce no output. Only if we evaluate the variable effect which is of type IO[Unit] will the output be generated (try effect.unsafeRunSync in the REPL). Also the second approach works like expected.

\n\n\n

IO example 2

1 import cats.effect.IO\n2 import cats.implicits._\n3 \n4 val printF = IO(println(\"Hi there!\"))\n5 \n6 val effect = for {\n7   _ <- printF\n8   _ <- printF\n9 } yield ()\n

\n\n\n\n\n\n\n

Suddenly we can much more easily reason about our code! And why is that? Well we don’t have unexpected side effects caused by code running even when it doesn’t need to. This is a sneak peak how pure code looks like. Now we only need to implement pure libraries for our use, or do we?

\n\n

Luckily for us meanwhile there are several pure options available in the Scala ecosystem. We will stick to the Cats family of libraries namely http4s and Doobie as replacements for Akka-HTTP and Slick. They build upon the Cats Effect library which is an implementation of an IO monad for Scala. Some other options exist but we’ll stick to the one from Cats.

\n\n

To be able to contrast both ways of implementing a service we will first implement it using Akka-HTTP and Slick and will then migrate to http4s and Doobie.

\n\n

\nImpure implementation

\n\n

We’ll be using the following libraries for the impure version of the service:

\n\n

Akka (including Akka-HTTP and Akka-Streams)
Slick (as database layer)
Flyway for database migrations (or evolutions)
Circe for JSON codecs and akka-http-json as wrapper
Refined for using refined types
the PostgreSQL JDBC driver

\n\n

I’ll spare you the sbt setup as you can look that up in the code repository (i.e. the impure folder in the book repo).

\n\n

\nModels

\n\n

First we’ll implement our models which are simple and straightforward. At first we need a class to store our translations or better a single translation.

\n\n\n

1 final case class Translation(lang: String, name: String)\n

\n\n\n\n\n\n\n

Technically it is okay but we have a bad feeling about it. Using Option[String] is of no use because both fields have to be set. But a String can always be null and contain a lot of unexpected stuff (literally anything).

\n\n\n\n

So let us define some refined types which we can use later on. At first we need a language code which obeys the restrictions of ISO-639-1 and we need a stronger definition for a product name. For the former we use a regular expression and for the latter we simply expect a string which is not empty.

\n\n\n

Refined types for models

1 type LanguageCode = String Refined MatchesRegex[W.`\"^[a-z]{2}$\"`.T]\n2 type ProductName = String Refined NonEmpty\n

\n\n\n\n\n

Now we can give our translation model another try.

\n\n\n

Translation model using refined types

1 final case class Translation(lang: LanguageCode, name: ProductName)\n

\n\n\n\n\n

Much better and while we’re at it we can also write the JSON codecs using the refined module of the Circe library. We put them into the companion object of the model.

\n\n\n

1 object Translation {\n2   implicit val decode: Decoder[Translation] =\n3     Decoder.forProduct2(\"lang\", \"name\")(Translation.apply)\n4 \n5   implicit val encode: Encoder[Translation] =\n6     Encoder.forProduct2(\"lang\", \"name\")(t => (t.lang, t.name))\n7 }\n

\n\n\n\n\n

Now onwards to the product model. Because we already know of refined types we can use them from start here.

\n\n\n

1 type ProductId = String Refined Uuid\n2 final case class Product(id: ProductId, names: List[Translation])\n

\n\n\n\n\n\n\n

If we look closely we realise that a List maybe empty. Which is valid for the list but not for our product because we need at least one entry. Luckily for us the Cats library has us covered with the NonEmptyList data type. Including the JSON codecs this leads us to our final implementation.
Last but not least we really should be using the existing UUID data type instead of rolling our own refined string version - even when it is cool. ;-)

\n\n\n

Product model using UUID type and NeL

 1 type ProductId = java.util.UUID\n 2 final case class Product(id: ProductId, names: NonEmptyList[Translation])\n 3 \n 4 object Product {\n 5   implicit val decode: Decoder[Product] =\n 6     Decoder.forProduct2(\"id\", \"names\")(Product.apply)\n 7 \n 8   implicit val encode: Encoder[Product] =\n 9     Encoder.forProduct2(\"id\", \"names\")(p => (p.id, p.names))\n10 }\n

\n\n\n\n\n

We kept the type name ProductId by using a type alias. This is convenient but remember that a type alias does not add extra type safety (e.g.Â type Foo = String will be a String).

\n\n\n\n

Well, maybe because a list may contain duplicate entries but the database will surely not because of unique constraints! So, let’s switch to a NonEmptySet which is also provided by Cats.

\n\n\n

Product model using UUID and NeS

1 type ProductId = java.util.UUID\n2 final case class Product(id: ProductId, names: NonEmptySet[Translation])\n

\n\n\n\n\n

Now we have the models covered and can move on to the database layer.

\n\n

\nDatabase layer

\n\n

The database layer should provide a programmatic access to the database but also should it manage changes in the database. The latter one is called migrations or evolutions. From the available options we chose Flyway as the tool to manage our database schema.

\n\n

\nMigrations

\n\n

Flyway uses raw SQL scripts which have to be put into a certain location being /db/migration (under the resources folder) in our case. Also the files have to be named like VXX__some_name.sql (XX being a number) starting with V1. Please note that there are two underscores between the version prefix and the rest of the name! Because our database schema is very simply we’re done quickly:

\n\n\n

Flyway migration for creating the database

 1 CREATE TABLE \"products\" (\n 2   \"id\" UUID NOT NULL,\n 3   CONSTRAINT \"products_pk\" PRIMARY KEY (\"id\")\n 4 );\n 5 \n 6 CREATE TABLE \"names\" (\n 7   \"product_id\" UUID       NOT NULL,\n 8   \"lang_code\"  VARCHAR(2) NOT NULL,\n 9   \"name\"       TEXT       NOT NULL,\n10   CONSTRAINT \"names_pk\" \n11     PRIMARY KEY (\"product_id\", \"lang_code\"),\n12   CONSTRAINT \"names_product_id_fk\" \n13     FOREIGN KEY (\"product_id\") \n14     REFERENCES \"products\" (\"id\") \n15     ON DELETE CASCADE ON UPDATE CASCADE\n16 );\n

\n\n\n\n\n

In the code you’ll see that we additionally set comments which are omitted from the code snippet above. This might be overkill here but it is a very handy feature to have and I advice you to use it for more complicated database schemas. Because the right comment (read information) in the right place might save a lot of time when trying to understand things.

\n\n

Next we move on to the programmatic part which at first needs a configuration of our database connection. With Slick you have a multitude of options but we’ll use the “Typesafe Config”¹ approach.

\n\n\n

Database configuration in application.conf

 1 database {\n 2   profile = \"slick.jdbc.PostgresProfile$\"\n 3   db {\n 4     connectionPool = \"HikariCP\"\n 5     dataSourceClass = \"org.postgresql.ds.PGSimpleDataSource\"\n 6     properties {\n 7       serverName = \"localhost\"\n 8       portNumber = \"5432\"\n 9       databaseName = \"impure\"\n10       user = \"impure\"\n11       password = \"secret\"\n12     }\n13     numThreads = 10\n14   }\n15 }\n

\n\n\n\n\n

After we have this in place we can run the migrations via the API of Flyway. For this we have to load the configuration (we do it by creating an actor system), extract the needed information and create a JDBC url and use that with username and password to obtain a Flyway instance. On that one we simply call the method migrate() which will do the right thing. Basically it will check if the schema exists and decide to either create it, apply pending migrations or simply do nothing. The method will return the number of applied migrations.

\n\n\n

Apply database migrations via Flyway

 1 implicit val system: ActorSystem    = ActorSystem()\n 2 implicit val mat: ActorMaterializer = ActorMaterializer()\n 3 implicit val ec: ExecutionContext   = system.dispatcher\n 4 \n 5 val url = \"jdbc:postgresql://\" +\n 6 system.settings.config.getString(\"database.db.properties.serverName\") +\n 7 \":\" + system.settings.config.getString(\"database.db.properties.portNumber\") +\n 8 \"/\" + system.settings.config.getString(\"database.db.properties.databaseName\")\n 9 val user = system.settings.config.getString(\"database.db.properties.user\")\n10 val pass = system.settings.config.getString(\"database.db.properties.password\")\n11 val flyway = Flyway.configure().dataSource(url, user, pass).load()\n12 val _ = flyway.migrate()\n

\n\n\n\n\n

Let us continue to dive into the Slick table definitions.

\n\n

\nSlick tables

\n\n

Slick offers several options for approaching the database. For our example we will be using the lifted embedding but if needed Slick also provides the ability to perform plain SQL queries.
For the lifted embedding we have to define out tables in a way Slick can understand. While this can be tricky under certain circumstances our simple model is straightforward to implement.

\n\n\n

Slick product table definition

1 final class Products(tag: Tag) extends Table[(UUID)](tag, \"products\") {\n2   def id = column[UUID](\"id\", O.PrimaryKey)\n3 \n4   def * = (id)\n5 }\n6 val productsTable = TableQuery[Products]\n

\n\n\n\n\n

As you can see above we’re using simple data types (not the refined ones) to have a more easy Slick implementation. However we can also use refined types for the price of using either the slick-refined library or writing custom column mappers.
Next we’ll implement the table for the translations which will also need some constraints.

\n\n\n

Slick translations table definition

 1 final class Names(tag: Tag) extends Table[(UUID, String, String)](tag, \"names\") {\n 2   def productId = column[UUID](\"product_id\")\n 3   def langCode  = column[String](\"lang_code\")\n 4   def name      = column[String](\"name\")\n 5 \n 6   def pk = primaryKey(\"names_pk\", (productId, langCode))\n 7   def productFk =\n 8     foreignKey(\"names_product_id_fk\", productId, productsTable)(\n 9       _.id,\n10       onDelete = ForeignKeyAction.Cascade,\n11       onUpdate = ForeignKeyAction.Cascade\n12     )\n13 \n14   def * = (productId, langCode, name)\n15 }\n16 val namesTable = TableQuery[Names]\n

\n\n\n\n\n

As you can see the definition of constraints is also pretty simple. Now our repository needs some functions for a more convenient access to the data.

\n\n\n

Slick repository functions

 1 def loadProduct(id: ProductId): Future[Seq[(UUID, String, String)]] = {\n 2   val program = for {\n 3     (p, ns) <- productsTable\n 4       .filter(_.id === id)\n 5       .join(namesTable)\n 6       .on(_.id === _.productId)\n 7   } yield (p.id, ns.langCode, ns.name)\n 8   dbConfig.db.run(program.result)\n 9 }\n10 \n11 def loadProducts(): DatabasePublisher[(UUID, String, String)] = {\n12   val program = for {\n13     (p, ns) <- productsTable.join(namesTable)\n14                  .on(_.id === _.productId).sortBy(_._1.id)\n15   } yield (p.id, ns.langCode, ns.name)\n16   dbConfig.db.stream(program.result)\n17 }\n18 \n19 def saveProduct(p: Product): Future[List[Int]] = {\n20   val cp      = productsTable += (p.id)\n21   val program = DBIO.sequence(\n22     cp :: saveTranslations(p).toList\n23   ).transactionally\n24   dbConfig.db.run(program)\n25 }\n26 \n27 def updateProduct(p: Product): Future[List[Int]] = {\n28   val program = namesTable\n29     .filter(_.productId === p.id)\n30     .delete\n31     .andThen(DBIO.sequence(saveTranslations(p).toList))\n32     .transactionally\n33   dbConfig.db.run(program)\n34 }\n35 \n36 protected def saveTranslations(p: Product): NonEmptyList[DBIO[Int]] = {\n37   val save = saveTranslation(p.id)(_)\n38   p.names.toNonEmptyList.map(t => save(t))\n39 }\n40 \n41 /**\n42   * Create a query to insert or update a given translation in the database.\n43   *\n44   * @param id The unique ID of the product.\n45   * @param t  The translation to be saved.\n46   * @return A composable sql query for Slick.\n47   */\n48 protected def saveTranslation(id: ProductId)(t: Translation): DBIO[Int] =\n49   namesTable.insertOrUpdate((id, t.lang, t.name))\n

\n\n\n\n\n

The last two functions are helpers to enable us to create a load queries which we can compose. They are used in the saveProduct and updateProduct functions to create a list of queries that are executed as bulk while the call to transactionally ensures that they will run within a transaction. When updating a product we first delete all existing translations to allow the removal of existing translations via an update. To be able to do so we use the andThen helper from Slick.
The loadProduct function simply returns a list of database rows from the needed join. Therefore we need a function which builds a Product type out of that.

\n\n\n

Helper function to create a Product

 1 def fromDatabase(rows: Seq[(UUID, String, String)]): Option[Product] = {\n 2   val po = for {\n 3     (id, c, n) <- rows.headOption\n 4     t          <- Translation.fromUnsafe(c)(n)\n 5     p          <- Product(\n 6                     id = id,\n 7                     names = NonEmptySet.one[Translation](t)\n 8                   ).some\n 9   } yield p\n10   po.map(\n11     p =>\n12       rows.drop(1).foldLeft(p) { (a, cols) =>\n13         val (id, c, n) = cols\n14         Translation.fromUnsafe(c)(n).fold(a)(t =>\n15           a.copy(names = a.names.add(t))\n16         )\n17     }\n18   )\n19 }\n

\n\n\n\n\n

But oh no! The compiler refuses to build it:

\n\n\n

Missing cats.Order

1 [error] .../impure/models/Product.scala:45:74:\n2   could not find implicit value for parameter A: \n3     cats.kernel.Order[com.wegtam.books.pfhais.impure.models.Translation]\n4 [error] p <- Product(id = id, names = NonEmptySet.one[Translation](t)).some\n5 [error]                                                            ^\n

\n\n\n\n\n

It seems we have to provide an instance of Order for our Translation model to make Cats happy. So we have think of an ordering for our model. A simple approach would be to simply order by the language code. Let’s try this:

\n\n\n

Providing Order for LanguageCode

1 import cats._\n2 import cats.syntax.order._\n3 \n4 implicit val orderLanguageCode: Order[LanguageCode] = \n5   new Order[LanguageCode] {\n6     def compare(x: LanguageCode, y: LanguageCode): Int =\n7       x.value.compare(y.value)\n8   }\n

\n\n\n\n\n\n\n

You might have noticed the explicit call to .value to get the underlying string instance of our refined type. This is needed because the other option (using x.compare(y)) will compile but bless you with stack overflow errors. The reason is probably that the latter is compiled into code calling OrderOps#compare which is recursive.

\n\n\n

Providing Order for Translation

1 import cats._\n2 import cats.syntax.order._\n3 \n4 implicit val order: Order[Translation] =\n5   new Order[Translation] {\n6     def compare(x: Translation, y: Translation): Int =\n7       x.lang.compare(y.lang)\n8   }\n

\n\n\n\n\n

So far we should have everything in place to make use of our database. Now we need to wire it all together.

\n\n

\nAkka-HTTP routes

\n\n

Defining the routes is pretty simple if you’re used to the Akka-HTTP routing DSL syntax.

\n\n\n

Basic routes with Akka-HTTP

 1 val route = path(\"product\" / JavaUUID) { id: ProductId =>\n 2   get {\n 3     ???\n 4   } ~ put {\n 5     ???\n 6   }\n 7 } ~ path(\"products\") {\n 8   get {\n 9     ???\n10   } ~\n11   post {\n12     ???\n13   }\n14 }\n

\n\n\n\n\n

We will fill in the details later on. But now for starting the actual server to make use of our routes.

\n\n\n

Starting an Akka-HTTP server

1 val host       = system.settings.config.getString(\"api.host\")\n2 val port       = system.settings.config.getInt(\"api.port\")\n3 val srv        = Http().bindAndHandle(route, host, port)\n4 val pressEnter = StdIn.readLine()\n5 srv.flatMap(_.unbind()).onComplete(_ => system.terminate())\n

\n\n\n\n\n

The code will fire up a server using the defined routes and hostname and port from the configuration to start a server. It will run until you press enter and then terminate. Let us now visit the code for each routing endpoint. We will start with the one for returning a single product.

\n\n\n

Returning a single product

 1 path(\"product\" / JavaUUID) { id: ProductId =>\n 2   get {\n 3     complete {\n 4       for {\n 5         rows <- repo.loadProduct(id)\n 6         prod <- Future { Product.fromDatabase(rows) }\n 7       } yield prod\n 8     }\n 9   }\n10 }\n

\n\n\n\n\n

We load the raw product data from the repository and convert it into a proper product model. But to make the types align we have to wrap the second call in a Future otherwise we would get a compiler error. We don’t need to marshal the response because we are using the akka-http-json library which provides for example an ErrorAccumulatingCirceSupport import that handles this. Unless of course you do not have circe codecs defined for your types.

\n\n\n

Updating a single product

1 val route = path(\"product\" / JavaUUID) { id: ProductId =>\n2   put {\n3     entity(as[Product]) { p =>\n4       complete {\n5         repo.updateProduct(p)\n6       }\n7     }\n8   }\n9 }\n

\n\n\n\n\n

The route for updating a product is also very simple. We’re extracting the product entity via the entity(as[T]) directive from the request body and simply give it to the appropriate repository function. Now onwards to creating a new product.

\n\n\n

Creating a product

1 path(\"products\") {\n2   post {\n3     entity(as[Product]) { p =>\n4       complete {\n5         repo.saveProduct(p)\n6       }\n7     }\n8   }\n9 }\n

\n\n\n\n\n

As you can see the function is basically the same except that we’re calling a different function from the repository. Last but not least let us take a look at the return all products endpoint.

\n\n\n

Return all products

 1 path(\"products\") {\n 2   get {\n 3     complete {\n 4       val products = for {\n 5         rows <- repo.loadProducts()\n 6         ps <- Future {\n 7           rows.toList.groupBy(_._1).map {\n 8             case (_, cols) => Product.fromDatabase(cols)\n 9           }\n10         }\n11       } yield ps\n12       products.map(_.toList.flatten)\n13     }\n14   }\n15 }\n

\n\n\n\n\n

This looks more complicated that the other endpoints. So what exactly are we doing here?
Well first we load the raw product data from the repository. Afterwards we convert it into the proper data model or to be more exact into a list of product entities.

\n\n\n\n

The first thing that comes to mind is that we’re performing operations in memory. This is not different from the last time when we converted the data for a single product. Now however we’re talking about all products which may be a lot of data. Another obvious point is that we get a list of Option[Product] which we explicitly flatten at the end.

\n\n\n\n

Maybe we should consider streaming the results. But we still have to group and combine the rows which belong to a single product into a product entity. Can we achieve that with streaming? Well, let’s look at our data flow.
We receive a list of 3 columns from the database in the following format: product id, language code, name. The tricky part being that multiple rows (list entries) can belong to the same product recognizable by the same value for the first column product id. At first we should simplify our problem by ensuring that the list will be sorted by the product id. This is done by adjusting the function loadProducts in the repository.

\n\n\n

Sort the returned list of entries.

1 def loadProducts(): DatabasePublisher[(UUID, String, String)] = {\n2   val program = for {\n3     (p, ns) <- productsTable.join(namesTable).on(_.id === _.productId)\n4                .sortBy(_._1.id)\n5   } yield (p.id, ns.langCode, ns.name)\n6   dbConfig.db.stream(program.result)\n7 }\n

\n\n\n\n\n

Now we can rely on the fact that we have seen all entries for one product if the product id in our list changes. Let’s adjust our code in the endpoint to make use of streaming now. Because Akka-HTTP is based on Akka-Streams we can simply use that.

\n\n\n

Return all products as stream

 1 path(\"products\") {\n 2   get {\n 3     implicit val jsonStreamingSupport: JsonEntityStreamingSupport =\n 4       EntityStreamingSupport.json()\n 5 \n 6     val src = Source.fromPublisher(repo.loadProducts())\n 7     val products: Source[Product, NotUsed] = src\n 8       .collect(\n 9         cs =>\n10           Product.fromDatabase(Seq(cs)) match {\n11             case Some(p) => p\n12         }\n13       )\n14       .groupBy(Int.MaxValue, _.id)\n15       .fold(Option.empty[Product])(\n16         (op, x) => op.fold(x.some)(p =>\n17           p.copy(names = p.names ::: x.names).some\n18         )\n19       )\n20       .mergeSubstreams\n21       .collect(\n22         op =>\n23           op match {\n24             case Some(p) => p\n25         }\n26       )\n27     complete(products)\n28   }\n29 }\n

\n\n\n\n\n

Wow, this may look scary but let’s break it apart piece by piece. At first we need an implicit value which provides streaming support for JSON. Next we create a Source from the database stream. Now we implement the processing logic via the high level streams API. We collect every defined output of our helper function fromDatabase which leads to a stream of Product entities. But we have created way too many (Each product will be created as often as it has translations.). So we group our stream by the product id which creates a new stream for each product id holding only the entities for the specific product. We fold over each of these streams by merging together the list of translations (names). Afterwards we merge the streams back together and run another collect function to simply get a result stream of Product and not of Option[Product]. Last but not least the stream is passed to the complete function which will do the right thing.

\n\n

\nProblems with the solution

\n\n

The solution has two problems:

\n\n

The number of individual streams (and thus products) is limited to Int.MaxValue.
The groupBy operator holds the references to these streams in memory opening a possible out of memory issue here.

\n\n

As the first problem is simply related to the usage of groupBy we may say that we only have one problem: The usage of groupBy. ;-)
For a limited amount of data the proposed solution is perfectly fine so we will leave it as is for now.

\n\n

Regarding the state of our service we have a working solution, so congratulations and let’s move on to the pure implementation.

\n\n

\nPure implementation

\n\n

Like in the previous section I will spare you the details of the sbt setup. We will be using the following set of libraries:

\n\n

http4s
Doobie (as database layer)
Flyway for database migrations (or evolutions)
Circe for JSON codecs
Refined for using refined types
the PostgreSQL JDBC driver
pureconfig (for proper configuration loading)

\n\n

\nPure configuration handling

\n\n

Last time we simply loaded our configuration via the typesafe config library but can’t we do a bit better here? The answer is yes by using the pureconfig¹ library. First we start by implementing the necessary parts of our configuration as data types.

\n\n\n

Configuration data types

 1 final case class ApiConfig(host: NonEmptyString, port: PortNumber)\n 2 \n 3 object ApiConfig {\n 4   implicit val configReader: ConfigReader[ApiConfig] =\n 5     deriveReader[ApiConfig]\n 6 }\n 7 \n 8 final case class DatabaseConfig(driver: NonEmptyString,\n 9                                 url: DatabaseUrl,\n10                                 user: DatabaseLogin,\n11                                 pass: DatabasePassword)\n12 \n13 object DatabaseConfig {\n14   implicit val configReader: ConfigReader[DatabaseConfig] =\n15     deriveReader[DatabaseConfig]\n16 }\n

\n\n\n\n\n

As we can see the code is pretty simple. The implicits in the companion objects are needed for pureconfig to actually map from a configuration to your data types. As you can see we are using a function deriveReader which will derive (like in mathematics) the codec (Yes, it is similar to a JSON codec thus the name.) for us.

\n\n\n\n

Below is an example of deriving a Order instance using the kittens ²library. It uses shapeless under the hood and provides automatic and semi automatic derivation for a lot of type class instances from Cats like Eq, Order, Show, Functor and so on.

\n\n\n

Deriving Order via kittens

1 import cats._\n2 import cats.derived\n3 \n4 implicit val order: Order[Translation] = {\n5   import derived.auto.order._\n6   derived.semi.order[Translation]\n7 }\n

\n\n\n\n\n

\nModels

\n\n

Because we have already written our models we just re-use them here. The only thing we change is the semi automatic derivation of the JSON codecs. We just need to import the appropriate circe package and call the derive functions.

\n\n\n

Derive JSON codecs

1 import io.circe._\n2 import io.circe.generic.semiauto._\n3 \n4 implicit val decode: Decoder[Product] = deriveDecoder[Product]\n5 implicit val encode: Encoder[Product] = deriveEncoder[Product]\n6 implicit val decode: Decoder[Translation] = deriveDecoder[Translation]\n7 implicit val encode: Encoder[Translation] = deriveEncoder[Translation]\n

\n\n\n\n\n\n\n

\nDatabase layer

\n\n

In general the same applies to the database layer as we have already read in the “impure” section.

\n\n

\nMigrations

\n\n

For the sake of simplicity we will stick to Flyway for our database migrations. However we will wrap the migration code in a different way (read Encapsulate it properly within an IO to defer side effects.). While we’re at it we may just as well write our migration code using the interpreter pattern (it became famous under the name “tagless final” in Scala).

\n\n\n

Database migrator base

1 trait DatabaseMigrator[F[_]] {\n2   def migrate(url: DatabaseUrl,\n3               user: DatabaseLogin,\n4               pass: DatabasePassword): F[Int]\n5 }\n

\n\n\n\n\n

We define a trait which describes the functionality desired by our interpreter and use a higher kinded type parameter to be able to abstract over the type. But now let’s continue with our Flyway interpreter.

\n\n\n

Flyway migrator interpreter

 1 final class FlywayDatabaseMigrator extends DatabaseMigrator[IO] {\n 2   override def migrate(url: DatabaseUrl,\n 3                        user: DatabaseLogin,\n 4                        pass: DatabasePassword): IO[Int] =\n 5     IO {\n 6       val flyway: Flyway = Flyway.configure()\n 7         .dataSource(url, user, pass)\n 8         .load()\n 9       flyway.migrate()\n10     }\n11 }\n

\n\n\n\n\n

As we can see, the implementation is pretty simple and we just wrap our code into an IO monad to constrain the effect. Having the migration code settled we can move on to the repository.

\n\n\n\n

If we take a closer look at the method definition of Flyway.migrate, we see this:

\n\n\n

Method definition of Flyway.migrate

1 public int migrate() throws FlywayException\n

\n\n\n\n\n

While IO will gladly defer side effects for us it won’t stop enclosed code from throwing exceptions. This is not that great. So what can we do about it?
Having an instance of MonadError in scope we could just use the .attempt function provided by it. But is this enough or better does this provide a sensible solution for us? Let’s play a bit on the REPL.

\n\n\n

MonadError on the REPL

 1 @ import cats._, cats.effect._, cats.implicits._\n 2 @ val program = for {\n 3            _ <- IO(println(\"one\"))\n 4            _ <- IO(println(\"two\"))\n 5            x <- IO.pure(42)\n 6            } yield x\n 7 @ program.attempt.unsafeRunSync match {\n 8            case Left(e) =>\n 9              println(e.getMessage)\n10              -1\n11            case Right(r) => r\n12            }\n13 one\n14 two\n15 res3: Int = 42\n16 @ val program = for {\n17            _ <- IO(println(\"one\"))\n18            _ <- IO(throw new Error(\"BOOM!\"))\n19            x <- IO.pure(42)\n20            } yield x\n21 @ program.attempt.unsafeRunSync match {\n22            case Left(e) =>\n23              println(e.getMessage)\n24              -1\n25            case Right(r) => r\n26            }\n27 one\n28 BOOM!\n29 res5: Int = -1\n

\n\n\n\n\n

This looks like we just have to use MonadError then. Hurray, we don’t need to change our code in the migrator. As model citizens of the functional programming camp we just defer the responsibility upwards to the calling site.

\n\n

\nDoobie

\n\n

As we already started with using a tagless final approach we might as well continue with it and define a base for our repository.

\n\n\n

Base trait for the repository

1 trait Repository[F[_]] {\n2   def loadProduct(id: ProductId): F[Seq[(ProductId, LanguageCode, ProductName)]]\n3 \n4   def loadProducts(): Stream[F, (ProductId, LanguageCode, ProductName)]\n5 \n6   def saveProduct(p: Product): F[Int]\n7 \n8   def updateProduct(p: Product): F[Int]\n9 }\n

\n\n\n\n\n

There is nothing exciting here except that we feel brave now and try to use proper refined types in our database functions. This is possible due to the usage of the doobie-refined module. To be able to map the UUID data type (and others) we also need to include the doobie-postgresql module. For convenience we are still using ProductId instead of UUID in our definition. In addition we wire the return type of loadProducts to be a fs2.Stream because we want to achieve pure functional streaming here. :-)
So let’s see what a repository using doobie looks like.

\n\n\n

The doobie repository.

 1 final class DoobieRepository[F[_]: Sync](tx: Transactor[F])\n 2   extends Repository[F] {\n 3 \n 4   override def loadProduct(id: ProductId) = ???\n 5 \n 6   override def loadProducts() = ???\n 7 \n 8   override def saveProduct(p: Product) = ???\n 9 \n10   override def updateProduct(p: Product) = ???\n11 }\n

\n\n\n\n\n

We keep our higher kinded type as abstract as we can but we want it to be able to suspend our side effects. Therefore we require an implicit Sync.³
If we look at the detailed function definitions further below, the first big difference is that with doobie you write plain SQL queries. You can do this with Slick too⁴ but with doobie it is the only way. If you’re used to object relational mapping (ORM) or other forms of query compilers then this may seem strange at first. But: “In data processing it seems, all roads eventually lead back to SQL!”⁵ ;-)
We won’t discuss the benefits or drawbacks here but in general I also lean towards the approach of using the de facto lingua franca for database access because it was made for this and so far no query compiler was able to beat hand crafted SQL in terms of performance. Another benefit is that if you ask a database guru for help, she will be much more able to help you with plain SQL queries than with some meta query which is compiled into something that you have no idea of.

\n\n\n

Loading a product.

1 override def loadProduct(id: ProductId) = \n2   sql\"\"\"SELECT products.id, names.lang_code, names.name \n3         FROM products\n4         JOIN names ON products.id = names.product_id\n5         WHERE products.id = $id\"\"\"\n6     .query[(ProductId, LanguageCode, ProductName)]\n7     .to[Seq]\n8     .transact(tx)\n

\n\n\n\n\n

The loadProduct function simply returns all rows for a single product from the database like its Slick counterpart in the impure variant. The parameter will be correctly interpolated by Doobie therefore we don’t need to worry about SQL injections here. We specify the type of the query, instruct Doobie to transform it into a sequence and give it to the transactor.

\n\n\n\n\n

Load all products

1 override def loadProducts() =\n2   sql\"\"\"SELECT products.id, names.lang_code, names.name\n3       FROM products\n4       JOIN names ON products.id = names.product_id\n5       ORDER BY products.id\"\"\"\n6     .query[(ProductId, LanguageCode, ProductName)]\n7     .stream\n8     .transact(tx)\n

\n\n\n\n\n

Our loadProducts function is equivalent to the first one but it returns the data for all products sorted by product and as a stream using the fs2 library which provides pure functional streaming.

\n\n\n

Save a product

 1 override def saveProduct(p: Product): F[Int] = {\n 2   val namesSql = \n 3     \"INSERT INTO names (product_id, lang_code, name) VALUES (?, ?, ?)\"\n 4   val namesValues = p.names.map(t => (p.id, t.lang, t.name))\n 5   val program = for {\n 6     pi <- sql\"INSERT INTO products (id) VALUES(${p.id})\".update.run\n 7     ni <- Update[(ProductId, LanguageCode, ProductName)](namesSql)\n 8             .updateMany(namesValues)\n 9   } yield pi + ni\n10   program.transact(tx)\n11 }\n

\n\n\n\n\n

When saving a product we use monadic notation for our program to have it short circuit in the case of failure. Doobie will also put all commands into a database transaction. The function itself will try to create the “master” entry into the products table and save all translations afterwards.

\n\n\n

Update a product

 1 override def updateProduct(p: Product): F[Int] = {\n 2   val namesSql =\n 3     \"INSERT INTO names (product_id, lang_code, name) VALUES (?, ?, ?)\"\n 4   val namesValues = p.names.map(t => (p.id, t.lang, t.name))\n 5   val program = for {\n 6     dl <- sql\"DELETE FROM names WHERE product_id = ${p.id}\".update.run\n 7     ts <- Update[(ProductId, LanguageCode, ProductName)](namesSql)\n 8             .updateMany(namesValues)\n 9   } yield dl + ts\n10   program.transact(tx)\n11 }\n

\n\n\n\n\n

The updateProduct function uses also monadic notation like the saveProduct function we talked about before. The difference is that it first deletes all known translations before saving the given ones.

\n\n

\nhttp4s routes

\n\n

The routing DSL of http4s differs from the one of Akka-HTTP. Although I like the latter one more it poses no problem to model out a base for our routes.

\n\n\n

Base for http4s routes

 1 val productRoutes: HttpRoutes[IO] = HttpRoutes.of[IO] {\n 2   case GET -> Root / \"product\" / id =>\n 3     ???\n 4   case PUT -> Root / \"product\" / id =>\n 5     ???\n 6 }\n 7 val productsRoutes: HttpRoutes[IO] = HttpRoutes.of[IO] {\n 8   case GET -> Root / \"products\" =>\n 9     ???\n10   case POST -> Root / \"products\" =>\n11     ???\n12 }\n

\n\n\n\n\n

As we can see the DSL is closer to Scala syntax and quite easy to read. But before we move on to the details of each route let’s think about how we can model this a bit more abstract. While it is fine to have our routes bound to IO it would be better to have more flexibility here. We have several options here but for starters we just extract our routes into their own classes like in the following schema.

\n\n\n

Routing classes

 1 final class ProductRoutes[F[_]: Sync](repo: Repository[F])\n 2   extends Http4sDsl[F] {\n 3 \n 4   val routes: HttpRoutes[F] = HttpRoutes.of[F] {\n 5     case GET -> Root / \"product\" / UUIDVar(id) =>\n 6       ???\n 7     case req @ PUT -> Root / \"product\" / UUIDVar(id) =>\n 8       ???\n 9   }\n10 }\n11 \n12 final class ProductsRoutes[F[_]: Sync](repo: Repository[F]) \n13   extends Http4sDsl[F] {\n14 \n15   val routes: HttpRoutes[F] = HttpRoutes.of[F] {\n16     case GET -> Root / \"products\" =>\n17       ???\n18     case req @ POST -> Root / \"products\" =>\n19       ???\n20   }\n21 }\n

\n\n\n\n\n

So far they only need the repository to access and manipulate data. Now let’s take on the single route implementations.

\n\n\n

Product routes

 1 final class ProductRoutes[F[_]: Sync](repo: Repository[F])\n 2   extends Http4sDsl[F] {\n 3   implicit def decodeProduct = jsonOf\n 4   implicit def encodeProduct = jsonEncoderOf\n 5 \n 6   val routes: HttpRoutes[F] = HttpRoutes.of[F] {\n 7     case GET -> Root / \"product\" / UUIDVar(id) =>\n 8       for {\n 9         rows <- repo.loadProduct(id)\n10         resp <- Ok(Product.fromDatabase(rows))\n11       } yield resp\n12     case req @ PUT -> Root / \"product\" / UUIDVar(id) =>\n13       for {\n14         p <- req.as[Product]\n15         _ <- repo.updateProduct(p)\n16         r <- NoContent()\n17       } yield r\n18   }\n19 }\n

\n\n\n\n\n

First we need to bring JSON codecs in scope for http4s thus the implicit definitions on top of the file. In the route for loading a single product we simply load the database rows which we pipe through our helper function to construct a proper Product and return that.
The update route (via PUT) transforms the request body into a Product and gives that to the update function of the repository. Finally a NoContent response is returned.

\n\n\n

Products routes (1st try)

 1 final class ProductsRoutes[F[_]: Sync](repo: Repository[F])\n 2   extends Http4sDsl[F] {\n 3   implicit def decodeProduct = jsonOf\n 4   implicit def encodeProduct = jsonEncoderOf\n 5 \n 6   val routes: HttpRoutes[F] = HttpRoutes.of[F] {\n 7     case GET -> Root / \"products\" =>\n 8       val ps: Stream[F, Product] = repo.loadProducts\n 9         .map(cs => Product.fromDatabase(List(cs)))\n10         .collect {\n11           case Some(p) => p\n12         }\n13       Ok(ps)\n14     case req @ POST -> Root / \"products\" =>\n15       for {\n16         p <- req.as[Product]\n17         _ <- repo.saveProduct(p)\n18         r <- NoContent()\n19       } yield r\n20   }\n21 }\n

\n\n\n\n\n

Our first take on the routes for products looks pretty complete already. Again we need implicit definitions for our JSON codecs to be able to serialize and de-serialize our entities. The POST route for creating a product is basically the same as the update route from the previous part. We create a Product from the request body, pass it to the save function of the repository and return a 205 NoContent response.
The GET route for returning all products calls the appropriate repository function which returns a stream which we map over using our helper function. Afterwards we use collect to convert our stream from Option[Product] to a stream of Product which we pass to the Ok function of http4s.

\n\n\n\n

To solve this we need to dive into the fs2 API and leverage it’s power to merge our products back together. So let’s see how we do.

\n\n

\nStreaming - Take 1

\n\n

Because we believe ourselves to be clever we pick the simple sledge hammer approach and just run some accumulator on the stream. So what do we need? A helper function and some code changes on the stream (e.g. in the route).

\n\n\n

Merging products

1 def merge(ps: List[Product])(p: Product): List[Product] =\n2   ps.headOption.fold(List(p)) { h =>\n3     if (h.id === p.id)\n4       h.copy(names = h.names ::: p.names) :: ps.drop(1)\n5     else\n6       p :: ps\n7   }\n

\n\n\n\n\n

So this function will take a list (that may be empty) and a product and will merge the top most element (the head) of the list with the given one. It will return an updated list that either contains an updated head element or a new head. Leaving aside the question of who guarantees that the relevant list element will always be the head, we may use it.

\n\n\n

Adapted streaming route

1 case GET -> Root / \"products\" =>\n2   val ps: Stream[F, Product] = repo.loadProducts\n3     .map(cs => Product.fromDatabase(List(cs)))\n4     .collect {\n5       case Some(p) => p\n6     }\n7     .fold(List.empty[Product])((acc, p) => Product.merge(acc)(p))\n8   Ok(ps)\n

\n\n\n\n\n

Looks so simple, does it? Just a simple fold which uses our accumulator and we should be settled. But life is not that simple…

\n\n\n

Compiler error

1 found   : fs2.Stream[F,List[com.wegtam.books.pfhais.pure.models.Product]]\n2 required: fs2.Stream[F,com.wegtam.books.pfhais.pure.models.Product]\n3          .fold(List.empty[Product])((acc, p) => Product.merge(acc)(p))\n4               ^\n

\n\n\n\n\n

The compiler complains that we have changed the type of the stream and rightly so. So let’s fix that compiler error.

\n\n\n\n

Let’s take a look again and think about what it means to change a stream of products into a stream of a list of products. It means that we will be building the whole thing in memory! Well if we wanted that we could have skipped streaming at all. So back to the drawing board.

\n\n

\nStreaming - Take 2

\n\n

We need to process our stream of database columns (or products if we use the converter like before) in such a way that all related entities will be grouped into one product and emitted as such. After browsing the documentation of fs2 we stumble upon a function called groupAdjacentBy so we try that one.

\n\n\n

Proper streaming of products

 1 case GET -> Root / \"products\" =>\n 2   val ps = repo.loadProducts\n 3     .groupAdjacentBy(_._1)\n 4     .map {\n 5       case (id, rows) => Product.fromDatabase(rows.toList)\n 6     }\n 7     .collect {\n 8       case Some(p) => p\n 9     }\n10   Ok(ps)\n

\n\n\n\n\n

Okay, this does not look complicated and it even compiles - Hooray! :-)
So let’s break it apart piece by piece. The group function of fs2 will partition the input depending on the given function into chunks. A Chunk is used internally by fs2 for all kinds of stuff. You may compare it to a sub-stream of Akka-Streams. However the documentation labels it as: Strict, finite sequence of values that allows index-based random access of elements.
Having our chunks we can map over each one converting it into a list which is then passed to our helper function fromDatabase to create proper products. Last but not least we need to collect our entities to get from an Option[Product] to a stream of Product.

\n\n

\nJSON trouble

\n\n

Now that we have a proper streaming solution we try it out but what do we get when we expect a list of products?

\n\n\n\n\n

Broken JSON

 1 % http :53248/products\n 2 HTTP/1.1 200 OK\n 3 Content-Type: application/json\n 4 Transfer-Encoding: chunked\n 5 \n 6 {\n 7   \"id\":\"8773899b-fcfa-401f-af3e-b188ebb0c00c\",\n 8   \"names\":[\n 9     {\"lang\":\"de\",\"name\":\"Erdbeere\"},\n10     {\"lang\":\"en\",\"name\":\"Strawberry\"}\n11   ]\n12 }\n13 {\n14   \"id\":\"983aaf86-abe4-44af-9896-d8f2d2c5f82c\",\n15   \"names\":[\n16     {\"lang\":\"de\",\"name\":\"Gurke\"},\n17     {\"lang\":\"en\",\"name\":\"Cucumber\"}\n18   ]\n19 }\n

\n\n\n\n\n

Well, whatever this is, it is not JSON! It might look like it, but it isn’t. However quite often you can see such things in the wild (read in production).

\n\n\n\n

If we think about it then this sounds like a bug in http4s and indeed we find an issue⁷ for it. Because the underlying problem is not as trivial as it first sounds maybe we should try to work around the issue.
The fs2 API offers concatenation of streams and the nifty intersperse function to insert elements between emitted ones. So let’s give it a try.

\n\n\n

Fix JSON encoding issues

 1 case GET -> Root / \"products\" =>\n 2   val prefix = Stream.eval(\"[\".pure[F])\n 3   val suffix = Stream.eval(\"]\".pure[F])\n 4   val ps = repo.loadProducts\n 5     .groupAdjacentBy(_._1)\n 6     .map {\n 7       case (id, rows) => Product.fromDatabase(rows.toList)\n 8     }\n 9     .collect {\n10       case Some(p) => p\n11     }\n12     .map(_.asJson.noSpaces)\n13     .intersperse(\",\")\n14   @SuppressWarnings(Array(\"org.wartremover.warts.Any\"))\n15   val result: Stream[F, String] = prefix ++ ps ++ suffix\n16   Ok(result)\n

\n\n\n\n\n

First we create streams for the first and last JSON that we need to emit. Please note that we cannot simply use a String here but have to lift it into our HKT F. The usage of pure is okay because we simply lift a fixed value. Then we extend our original stream processing by explicitly converting our products to JSON and inserting the delimiter (a comma) manually using the intersperse function. In the end we simply concatenate our streams and return the result.
Our solution is quite simple, having the downside that we need to suppress a warning from the wartremover⁸ tool. This is somewhat annoying but can happen. If we remove the annotation, we’ll get a compiler error:

\n\n\n

Wartremover error

1 [error] ... [wartremover:Any] Inferred type containing Any\n2 [error] val result: Stream[F, String] = prefix ++ ps ++ suffix\n3 [error]                                        ^\n4 [error] ... [wartremover:Any] Inferred type containing Any\n5 [error] val result: Stream[F, String] = prefix ++ ps ++ suffix\n6 [error]                                              ^\n7 [error] two errors found\n

\n\n\n\n\n

So let’s check if we have succeeded:

\n\n\n

Checking our JSON

 1 % http :53248/products\n 2 HTTP/1.1 200 OK\n 3 Content-Type: text/plain; charset=UTF-8\n 4 Transfer-Encoding: chunked\n 5 \n 6 [\n 7     {\n 8         \"id\": \"8773899b-fcfa-401f-af3e-b188ebb0c00c\", \n 9         \"names\": [\n10             {\n11                 \"lang\": \"de\",\n12                 \"name\": \"Erdbeere\"\n13             },\n14             {\n15                 \"lang\": \"en\",\n16                 \"name\": \"Strawberry\"\n17             }\n18         ]\n19     },\n20     {\n21         \"id\": \"983aaf86-abe4-44af-9896-d8f2d2c5f82c\", \n22         \"names\": [\n23             {\n24                 \"lang\": \"de\",\n25                 \"name\": \"Gurke\"\n26             },\n27             {\n28                 \"lang\": \"en\",\n29                 \"name\": \"Cucumber\"\n30             }\n31         ]\n32     }\n33 ]\n

\n\n\n\n\n

This looks good, so we congratulations: We are done with our routes!

\n\n

\nStarting the application

\n\n

Within our main entry point we simply initialise all needed components and wire them together. We’ll step through each part in this section. The first thing you’ll notice is that we use the IOApp provided by the Cats effect library⁹.

\n\n\n

Main application

1 object Pure extends IOApp {\n2   @SuppressWarnings(Array(\"org.wartremover.warts.Any\"))\n3   def run(args: List[String]): IO[ExitCode] = ???\n4 }\n

\n\n\n\n\n

Yet again we need to suppress a warning from wartremover here. But let’s continue to initialising the database connection.

\n\n\n

Database initialisation

 1 val migrator: DatabaseMigrator[IO] = new FlywayDatabaseMigrator\n 2 \n 3 val program = for {\n 4   (apiConfig, dbConfig) <- IO {\n 5     val cfg = ConfigFactory.load\n 6     (loadConfigOrThrow[ApiConfig](cfg, \"api\"),\n 7      loadConfigOrThrow[DatabaseConfig](cfg, \"database\"))\n 8   }\n 9   ms <- migrator.migrate(dbConfig.url, dbConfig.user, dbConfig.pass)\n10   tx = Transactor\n11     .fromDriverManager[IO](dbConfig.driver,\n12       dbConfig.url,\n13       dbConfig.user,\n14       dbConfig.pass)\n15   repo = new DoobieRepository(tx)\n

\n\n\n\n\n

We create our database migrator explicitly wired to the IO data type. Now we start with a for comprehension in which we load our configuration via pureconfig yet again within an IO. After successful loading of the configuration we continue with migrating the database. Finally we create the transactor needed by Doobie and the database repository.

\n\n\n

Routes and http4s server

 1 val program = for {\n 2   // ...\n 3   productRoutes  = new ProductRoutes(repo)\n 4   productsRoutes = new ProductsRoutes(repo)\n 5   routes         = productRoutes.routes <+> productsRoutes.routes\n 6   httpApp        = Router(\"/\" -> routes).orNotFound\n 7   server         = BlazeServerBuilder[IO].bindHttp(apiConfig.port,\n 8                      apiConfig.host).withHttpApp(httpApp)\n 9   fiber          = server.resource.use(_ => IO(StdIn.readLine())).as(ExitCode.Succes\\\n10 s)\n11 } yield fiber\n

\n\n\n\n\n

Here we create our routes via the classes, combine them (via <+> operator) and create the http4s app explicitly using an IO thus wiring our abstract routes to IO. The service will - like the impure one - run until you press enter. But it won’t run yet. ;-)

\n\n\n

Running the service

 1 program.attempt.unsafeRunSync match {\n 2   case Left(e) =>\n 3     IO {\n 4       println(\"*** An error occured! ***\")\n 5       if (e != null) {\n 6         println(e.getMessage)\n 7       }\n 8       ExitCode.Error\n 9     }\n10   case Right(r) => r\n11 }\n

\n\n\n\n\n

If you remember playing around with MonadError then you’ll recognize the attempt here. We attempt to run our program and execute possible side effects via the unsafeRunSync method from Cats effect. But to provide a proper return type for the IOApp we need to evaluate the return value which is either an error or a proper exit code. In case of an error we print it out on the console (no fancy logging here) and explicitly set an error code as the return value.

\n\n\n\n

As it seems we are done with our pure service! Or are we? Let’s see what we need add to both services if we want to test them.

\n\n

\nWhat about tests?

\n\n

In the domain of strong static typing (not necessarily functional) you might hear phrases like “It compiles therefore it must be correct thus we don’t need tests!”. While there is a point that certain kinds of tests can be omitted in favour of strong static typing such stances overlook that even a correctly typed program may produce the wrong output. The other extreme (coming from dynamic typed land) is to substitute typing with testing - which is even worse. Remember that testing is usually a probabilistic approach and cannot guarantee the absence of bugs. If you have ever refactored a large code base in both paradigms then you’ll very likely come to esteem a good type system.

\n\n

However, we need tests, so let’s write some. But before let us think a bit about what kinds of tests we need. :-)
Our service must read and create data in the JSON format. This format should be fixed and changes to it should raise some red flag because: Hey, we just broke our API! Furthermore we want to unleash the power of ScalaCheck¹ to benefit from property based testing. But even when we’re not using that we can still use it to generate test data for us.
Besides the regular unit tests there should be integration tests if a service is written. We can test a lot of things on the unit test side but in the end the integration of all our moving parts is what matters and often (usually on the not so pure side of things) you would have to trick (read mock) a lot to test things in isolation.

\n\n

\nTesting the impure service

\n\n

We will start with writing some data generators using the ScalaCheck library.

\n\n

\nGenerators

\n\n

ScalaCheck already provides several generators for primitives but for our data models we have to do some more plumbing. Let’s start with generating a language code.

\n\n\n

ScalaCheck generate LanguageCode

1 val genLanguageCode: Gen[LanguageCode] = Gen.oneOf(LanguageCodes.all)\n

\n\n\n\n\n

Using the Gen.oneOf helper from the library the code becomes dead simple. Generating a UUID is nothing special either.

\n\n\n

ScalaCheck generate UUID

1 val genUuid: Gen[UUID] = Gen.delay(UUID.randomUUID)\n2 )\n

\n\n\n\n\n

You might be tempted to use Gen.const here but please don’t because that one will be memorized and thus never change. Another option is using a list of randomly generated UUID values from which we then chose one. That would be sufficient for generators which only generate a single product but if we want to generate lists of them we would have duplicate ids sooner than later.

\n\n\n

ScalaCheck generate ProductName

1 val DefaultProductName: ProductName = \"I am a product name!\"\n2 val genProductName: Gen[ProductName] = for {\n3   cs <- Gen.nonEmptyListOf(Gen.alphaNumChar)\n4   name = RefType.applyRef[ProductName](cs.mkString)\n5            .getOrElse(DefaultProductName)\n6 } yield name\n

\n\n\n\n\n

So what do we have here? We want to generate a non empty string (because that is a requirement for our ProductName) but we also want to return a properly typed entity. First we let ScalaCheck generate a non empty list of random characters which we give to a utility function of refined. However we need a fallback value in case the validation done by refined fails. Therefore we defined a general default product name beforehand.

\n\n\n\n\n

ScalaCheck using Refined unsafeApply

1 val genProductName: Gen[ProductName] =\n2   Gen.nonEmptyListOf(Gen.alphaNumChar).map(cs => \n3     Refined.unsafeApply(cs.mkString)\n4   )\n

\n\n\n\n\n

Now that we have generators for language codes and product names we can write a generator for our Translation type.

\n\n\n

ScalaCheck generate Translation

 1 val genTranslation: Gen[Translation] = for {\n 2   c <- genLanguageCode\n 3   n <- genProductName\n 4 } yield\n 5   Translation(\n 6     lang = c,\n 7     name = n\n 8   )\n 9 \n10 implicit val arbitraryTranslation: Arbitrary[Translation] =\n11   Arbitrary(genTranslation)\n

\n\n\n\n\n

As we can see the code is also quite simple. Additionally we create an implicit arbitrary value which will be used automatically by the forAll test helper if it is in scope. To be able to generate a Product we will need to provide a non empty list of translations.

\n\n\n

ScalaCheck generate lists of translations

1 val genTranslationList: Gen[List[Translation]] = for {\n2   ts <- Gen.nonEmptyListOf(genTranslation)\n3 } yield ts\n4 \n5 val genNonEmptyTranslationList: Gen[NonEmptyList[Translation]] = for {\n6   t  <- genTranslation\n7   ts <- genTranslationList\n8   ns = NonEmptyList.fromList(ts)\n9 } yield ns.getOrElse(NonEmptyList.of(t))\n

\n\n\n\n\n

The first generator will create a non empty list of translations but will be typed as a simple List. Therefore we create a second generator which uses the fromList helper of the non empty list from Cats. Because that helper returns an Option (read is a safe function) we need to fallback to using a simple of function at the end.
With all these in place we can finally create our Product instances.

\n\n\n

ScalaCheck generate Product

 1 val genProduct: Gen[Product] = for {\n 2   id <- genProductId\n 3   ts <- genNonEmptyTranslationList\n 4 } yield\n 5   Product(\n 6     id = id,\n 7     names = ts\n 8   )\n 9 \n10 implicit val arbitraryProduct: Arbitrary[Product] =\n11   Arbitrary(genProduct)\n

\n\n\n\n\n

The code is basically the same as for Translation - the arbitrary implicit included.

\n\n

\nUnit Tests

\n\n

To avoid repeating the construction of our unit test classes we will implement a base class for tests which is quite simple.

\n\n\n

Base class for unit tests

1 abstract class BaseSpec extends WordSpec \n2   with MustMatchers with ScalaCheckPropertyChecks {}\n

\n\n\n\n\n

Feel free to use other test styles - after all ScalaTest offers a lot² of them. I tend to lean towards the more verbose ones like WordSpec. Maybe that is because I spent a lot of time with RSpec³ in the Ruby world. ;-)

\n\n\n

Testing Product#fromDatabase

1 import com.wegtam.books.pfhais.impure.models.TypeGenerators._\n2 \n3 forAll(\"input\") { p: Product =>\n4   val rows = p.names.map(t => (p.id, t.lang.value, t.name.value)).toList\n5   Product.fromDatabase(rows) must contain(p)\n6 }\n

\n\n\n\n\n

The code above is a very simple test of our helper function fromDatabase which works in the following way:

\n\n

The forAll will generate a lot of Product entities using the generator.
From each entity a list of “rows” is constructed like they would appear in the database.
These constructed rows are given to the fromDatabase function.
The returned Option must then contain the generated value.

\n\n

Because we construct the input for the function from a valid generated instance the function must always return a valid output.

\n\n

Now let’s continue with testing our JSON codec for Product.

\n\n

\nTesting a JSON codec

\n\n

We need our JSON codec to provide several guarantees:

\n\n

It must fail to decode invalid JSON input format (read garbage).
It must fail to decode valid JSON input format with invalid data (read wrong semantics).
It must succeed to decode completely valid input.
It must encode JSON which contains all fields included in the model.
It must be able to decode JSON that itself encoded.

\n\n

The first one is pretty simple and to be honest: You don’t have to write a test for this because that should be guaranteed by the Circe library. Things look a bit different for very simple JSON representations though (read when encoding to numbers or strings).
I’ve seen people arguing about point 5 and there may be applications for it but implementing encoders and decoders in a non-reversible way will make your life way more complicated.

\n\n\n

Testing decoding garbage input

1 forAll(\"input\") { s: String =>\n2   decode[Product](s).isLeft must be(true)\n3 }\n

\n\n\n\n\n

There is not much to say about the test above: It will generate a lot of random strings which will be passed to the decoder which must fail.

\n\n\n

Testing invalid input values

1 forAll(\"id\", \"names\") { (id: String, ns: List[String]) =>\n2   val json = \"\"\"{\n3     |\"id\":\"\"\" + id.asJson.noSpaces + \"\"\",\n4     |\"names\":\"\"\" + ns.asJson.noSpaces + \"\"\"\n5     |}\"\"\".stripMargin\n6   decode[Product](json).isLeft must be(true)\n7 }\n

\n\n\n\n\n

This test will generate random instances for id which are all wrong because it must be a UUID and not a string. Also the instances for names will mostly (but maybe not always) be wrong because there might be empty strings or an even empty list. So the decoder is given a valid JSON format but invalid values, therefore it must fail.

\n\n\n

Testing valid input

 1 forAll(\"input\") { i: Product =>\n 2   val json = s\"\"\"{\n 3     |\"id\": ${i.id.asJson.noSpaces},\n 4     |\"names\": ${i.names.asJson.noSpaces}\n 5     |}\"\"\".stripMargin\n 6   withClue(s\"Unable to decode JSON: $json\") {\n 7     decode[Product](json) match {\n 8       case Left(e)  => fail(e.getMessage)\n 9       case Right(v) => v must be(i)\n10     }\n11   }\n12 }\n

\n\n\n\n\n

In this case we manually construct a valid JSON input using values from a generated valid Product entity. This is passed to the decoder and the decoder must not only succeed but return an instance equal to the generated one.

\n\n\n

Test included fields

1 forAll(\"input\") { i: Product =>\n2   val json = i.asJson.noSpaces\n3   json must include(s\"\"\"\"id\":${i.id.asJson.noSpaces}\"\"\")\n4   json must include(s\"\"\"\"names\":${i.names.asJson.noSpaces}\"\"\")\n5 }\n

\n\n\n\n\n

The test will generate again a lot of entities and we construct a JSON string from each. We then expect the string to include several field names and their correctly encoded values. You might ask why we do not check for more things like: Are these fields the only ones within the JSON string? Well, this would be more cumbersome to test and a JSON containing more fields than we specify won’t matter for the decoder because it will just ignore them.

\n\n\n\n\n

Decoding encoded JSON

1 forAll(\"input\") { p: Product =>\n2   decode[Product](p.asJson.noSpaces) match {\n3     case Left(_)  => fail(\"Must be able to decode encoded JSON!\")\n4     case Right(d) => withClue(\"Must decode the same product!\")(d must be(p))\n5   }\n6 }\n

\n\n\n\n\n

Here we encode a generated entity and pass it to the encoder which must return the same entity.

\n\n

\nMore tests

\n\n

So this is basically what we do for models. Because we have more than one we will have to write tests for each of the others. I will spare you the JSON tests for Translation but that one also has a helper function called fromUnsafe so let’s take a look at the function.

\n\n\n

Translation#fromUnsafe

1 def fromUnsafe(lang: String)(name: String): Option[Translation] =\n2   for {\n3     l <- RefType.applyRef[LanguageCode](lang).toOption\n4     n <- RefType.applyRef[ProductName](name).toOption\n5   } yield Translation(lang = l, name = n)\n

\n\n\n\n\n

This function simply tries to create a valid Translation entity from unsafe input values using the helpers provided by refined. As we can see it is a total function (read is safe to use). To cover all corner cases we must test it with safe and unsafe input.

\n\n\n

Testing Translation#fromUnsafe (1)

 1 forAll(\"lang\", \"name\") { (l: String, n: String) =>\n 2   whenever(\n 3     RefType\n 4       .applyRef[LanguageCode](l)\n 5       .toOption\n 6       .isEmpty || RefType.applyRef[ProductName](n).toOption.isEmpty\n 7   ) {\n 8     Translation.fromUnsafe(l)(n) must be(empty)\n 9   }\n10 }\n

\n\n\n\n\n

Here we generate two random strings which we explicitly check to be invalid using the whenever helper. Finally the function must return an empty Option e.g. None for such values.

\n\n\n

Testing Translation#fromUnsafe (2)

1 forAll(\"input\") { t: Translation =>\n2   Translation.fromUnsafe(t.lang.value)(t.name.value) must contain(t)\n3 }\n

\n\n\n\n\n

The test for valid input is very simple because we simply use the values from our automatically generated valid instances. :-)

\n\n\n\n

So far we have no tests for our Repository class which handles all the database work. Neither have we tests for our routes. We have several options for testing here but before can test either of them we have do to some refactoring. For starters we should move our routes out of our main application into separate classes to be able to test them more easily.

\n\n\n\n

Yes, we should. There are of course limits and pros and cons to that but in general this makes sense. Also this has nothing to do with being “impure” or “pure” but with clean structure.

\n\n

\nSome refactoring

\n\n

Moving the routes into separate classes poses no big problem we simply create a ProductRoutes and a ProductsRoutes class which will hold the appropriate routes. As a result our somewhat messy main application code becomes more readable.

\n\n\n

New Impure main application

 1 def main(args: Array[String]): Unit = {\n 2   implicit val system: ActorSystem    = ActorSystem()\n 3   implicit val mat: ActorMaterializer = ActorMaterializer()\n 4   implicit val ec: ExecutionContext   = system.dispatcher\n 5 \n 6   val url = ???\n 7   val user           = ???\n 8   val pass           = ???\n 9   val flyway: Flyway = ???\n10   val _              = flyway.migrate()\n11 \n12   val dbConfig: DatabaseConfig[JdbcProfile] =\n13     DatabaseConfig.forConfig(\"database\", system.settings.config)\n14   val repo = new Repository(dbConfig)\n15 \n16   val productRoutes  = new ProductRoutes(repo)\n17   val productsRoutes = new ProductsRoutes(repo)\n18   val routes         = productRoutes.routes ~ productsRoutes.routes\n19 \n20   val host       = system.settings.config.getString(\"api.host\")\n21   val port       = system.settings.config.getInt(\"api.port\")\n22   val srv        = Http().bindAndHandle(routes, host, port)\n23   val pressEnter = StdIn.readLine()\n24   srv.flatMap(_.unbind()).onComplete(_ => system.terminate())\n25 }\n

\n\n\n\n\n

We simply create our instances from our routing classes and construct our global routes directly from them. This is good but if we want to test the routes in isolation we still have the problem that they are hard-wired to our Repository class which is implemented via Slick. Several options exist to handle this:

\n\n

Use an in-memory test database with according configuration.
Abstract further and use a trait instead of the concrete repository implementation.
Write integration tests which will require a working database.

\n\n

Using option 1 is tempting but think about it some more. While the benefit is that we can use our actual implementation and just have to fire up an in-memory database (for example h2), there are also some drawbacks:

\n\n

You have to handle evolutions for the in-memory database.
Your evolutions have to be completely portable SQL (read ANSI SQL). Otherwise you’ll have to write each of your evolutions scripts two times (one for production, one for testing).
Your code has to be database agnostic. This sounds easier than it is. Even the tools you’re using may use database specific features under the hood.
Several features are simply not implemented in some databases. Think of things like cascading deletion via foreign keys.

\n\n

Taking option 2 is a valid choice but it will result in more code. Also you must pay close attention to the “test repository” implementation to avoid introducing bugs there. Going for the most simple approach is usually feasible. Think of a simple test repository implementation that will just return hard coded values or values passed to it via constructor.

\n\n

However we will go with option 3 in this case. It has the drawback that you’ll have to provide a real database environment (and maybe more) for testing. But it is as close to production as you can get. Also you will need these either way to test your actual repository implementation, so let’s get going.

\n\n

\nIntegration Tests

\n\n

First we need to configure our test database because we do not want to accidentally wipe a production database. For our case we leave everything as is and just change the database name.

\n\n\n

Configuration file for integration tests

 1 api {\n 2   host = \"localhost\"\n 3   port = 49152\n 4 }\n 5 \n 6 database {\n 7   profile = \"slick.jdbc.PostgresProfile$\"\n 8   db {\n 9     connectionPool = \"HikariCP\"\n10     dataSourceClass = \"org.postgresql.ds.PGSimpleDataSource\"\n11     properties {\n12       serverName = \"localhost\"\n13       portNumber = \"5432\"\n14       databaseName = \"impure_test\"\n15       user = \"impure\"\n16       password = \"secret\"\n17     }\n18     numThreads = 10\n19   }\n20 }\n

\n\n\n\n\n

Another thing we should do is provide a test configuration for our logging framework. We use the logback library and Slick will produce a lot of logging output on the DEBUG level so we should fix that. It is nice to have logging if you need it but it also clutters up your log files. We create a file logback-test.xml in the directory src/it/resources which should look like this:

\n\n\n

Logging configuration for integration tests

 1 <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n 2 <configuration debug=\"false\">\n 3   <appender name=\"console\" class=\"ch.qos.logback.core.ConsoleAppender\">\n 4     <filter class=\"ch.qos.logback.classic.filter.ThresholdFilter\">\n 5       <level>WARN</level>\n 6     </filter>\n 7     <encoder>\n 8       <pattern>%date %highlight(%-5level) %cyan(%logger{0}) - %msg%n</pattern>\n 9     </encoder>\n10   </appender>\n11 \n12   <appender name=\"async-console\" class=\"ch.qos.logback.classic.AsyncAppender\">\n13     <appender-ref ref=\"console\"/>\n14     <queueSize>5000</queueSize>\n15     <discardingThreshold>0</discardingThreshold>\n16   </appender>\n17 \n18   <logger name=\"com.wegtam.books.pfhais.impure\" level=\"INFO\" additivity=\"false\">\n19     <appender-ref ref=\"console\"/>\n20   </logger>\n21 \n22   <root>\n23     <appender-ref ref=\"console\"/>\n24   </root>\n25 </configuration>\n

\n\n\n\n\n

Due to the nature of integration tests we want to use production or “production like” settings and environment. But really starting our application or service for each test will be quite cumbersome so we should provide a base class for our tests. In this class we will start our service, migrate our database and provide an opportunity to shut it down properly after testing.

\n\n

Because we do not want to get into trouble when running an Akka-HTTP on the same port, we first create a helper function which will determine a free port number.

\n\n\n

Finding a free port number for tests

1 import java.net.ServerSocket\n2 \n3 def findAvailablePort(): Int = {\n4   val serverSocket = new ServerSocket(0)\n5   val freePort     = serverSocket.getLocalPort\n6   serverSocket.setReuseAddress(true)\n7   serverSocket.close()\n8   freePort\n9 }\n

\n\n\n\n\n

The code is quite simple and very useful for such cases. Please note that it is important to use setReuseAddress because otherwise the found socket will be blocked for a certain amount of time. But now let us continue with our base test class.

\n\n\n

BaseSpec for integration tests

 1 abstract class BaseSpec\n 2     extends TestKit(\n 3       ActorSystem(\n 4         \"it-test\",\n 5         ConfigFactory\n 6           .parseString(s\"api.port=${BaseSpec.findAvailablePort()}\")\n 7           .withFallback(ConfigFactory.load())\n 8       )\n 9     )\n10     with AsyncWordSpecLike\n11     with MustMatchers\n12     with ScalaCheckPropertyChecks\n13     with BeforeAndAfterAll\n14     with BeforeAndAfterEach {\n15 \n16   implicit val materializer: ActorMaterializer = ActorMaterializer()\n17 \n18   private val url  = ???\n19   private val user = ???\n20   private val pass = ???\n21   protected val flyway: Flyway = \n22     Flyway.configure().dataSource(url, user, pass).load()\n23 \n24   override protected def afterAll(): Unit =\n25     TestKit.shutdownActorSystem(system, FiniteDuration(5, SECONDS))\n26 \n27   override protected def beforeAll(): Unit = {\n28     val _ = flyway.migrate()\n29   }\n30 }\n

\n\n\n\n\n

As you can see we are using the Akka-Testkit to initialise an actor system. This is useful because there are several helpers available which you might need. We configure the actor system with our free port using the loaded configuration as fallback. Next we globally create an actor materializer which is needed by Akka-HTTP and Akka-Streams. Also we create a globally available Flyway instance to make cleaning and migrating the database easier.
The base class also implements the beforeAll and afterAll methods which will be run before and after all tests. They are used to initially migrate the database and to shut down the actor system properly in the end.

\n\n

\nTesting the repository

\n\n

Now that we our parts in place we can write an integration test for our repository implementation.

\n\n\n\n

First we need to do some things globally for the test scope.

\n\n\n

Repository test: global stuff

 1 private val dbConfig: DatabaseConfig[JdbcProfile] =\n 2   DatabaseConfig.forConfig(\"database\", system.settings.config)\n 3 private val repo = new Repository(dbConfig)\n 4 \n 5 override protected def beforeEach(): Unit = {\n 6   flyway.clean()\n 7   val _ = flyway.migrate()\n 8   super.beforeEach()\n 9 }\n10 \n11 override protected def afterEach(): Unit = {\n12   flyway.clean()\n13   super.afterEach()\n14 }\n15 \n16 override protected def afterAll(): Unit = {\n17   repo.close()\n18   super.afterAll()\n19 }\n

\n\n\n\n\n

We create one Repository instance for all our tests here. The downside is that if one test crashes it then the other will be affected too. On the other hand we avoid running into database connection limits and severe code limbo to ensure closing a repository connection after each test no matter the result.
Also we clean and migrate before each test and clean also after each test. This ensures having a clean environment.

\n\n\n\n

Onwards to the test for loading a single product.

\n\n\n

Repository test: loadProduct

 1 \"#loadProduct\" when {\n 2   \"the ID does not exist\" must {\n 3     \"return an empty list of rows\" in {\n 4       val id = UUID.randomUUID\n 5       for {\n 6         rows <- repo.loadProduct(id)\n 7       } yield {\n 8         rows must be(empty)\n 9       }\n10     }\n11   }\n12 \n13   \"the ID exists\" must {\n14     \"return a list with all product rows\" in {\n15       genProduct.sample match {\n16         case None => fail(\"Could not generate data sample!\")\n17         case Some(p) =>\n18           for {\n19             _    <- repo.saveProduct(p)\n20             rows <- repo.loadProduct(p.id)\n21           } yield {\n22             Product.fromDatabase(rows) match {\n23               case None => fail(\"No product created from database rows!\")\n24               case Some(c) =>\n25                 c.id must be(p.id)\n26                 c mustEqual p\n27             }\n28           }\n29       }\n30     }\n31   }\n32 }\n

\n\n\n\n\n

Loading a non existing product must not produce any result and is simple to test if our database is empty. Testing the loading of a real product is not that much more complicated. We use the ScalaCheck generators to create one, save it and load it again. The loaded product must of course be equal to the saved one.

\n\n\n\n\n

Repository test: loadProducts

 1 \"#loadProducts\" when {\n 2   \"no products exist\" must {\n 3     \"return an empty stream\" in {\n 4       val src = Source.fromPublisher(repo.loadProducts())\n 5       for {\n 6         ps <- src.runWith(Sink.seq)\n 7       } yield {\n 8         ps must be(empty)\n 9       }\n10     }\n11   }\n12 \n13   \"some products exist\" must {\n14     \"return a stream with all product rows\" in {\n15       genProducts.sample match {\n16         case None => fail(\"Could not generate data sample!\")\n17         case Some(ps) =>\n18           val expected = ps.flatMap(\n19           p => p.names.toNonEmptyList.toList.map(\n20             n => (p.id, n.lang, n.name)\n21             )\n22           )\n23           for {\n24             _ <- Future.sequence(ps.map(p => repo.saveProduct(p)))\n25             src = Source\n26               .fromPublisher(repo.loadProducts())\n27               // more code omitted here\n28             rows <- src.runWith(Sink.seq)\n29           } yield {\n30             rows must not be (empty)\n31             rows.size mustEqual ps.size\n32             rows.toList.sorted mustEqual ps.sorted\n33           }\n34       }\n35     }\n36   }\n37 }\n

\n\n\n\n\n

Testing the loading of all products if none exit is trivial like the one for a non existing single product. For the case of multiple products we generate a list of them which we save. Afterwards we load them and use the same transformation logic like in the routes to be able to construct proper Product instances. One thing you might notice is the explicit sorting which is due to the fact that we want to ensure that our product lists are both sorted before comparing them.

\n\n\n

Repository test: saveProduct

 1 \"#saveProduct\" when {\n 2   \"the product does not already exist\" must {\n 3     \"save the product to the database\" in {\n 4       genProduct.sample match {\n 5         case None => fail(\"Could not generate data sample!\")\n 6         case Some(p) =>\n 7           for {\n 8             cnts <- repo.saveProduct(p)\n 9             rows <- repo.loadProduct(p.id)\n10           } yield {\n11             withClue(\"Data missing from database!\")(\n12               cnts.fold(0)(_ + _) must be(p.names.toNonEmptyList.size + 1))\n13             Product.fromDatabase(rows) match {\n14               case None => fail(\"No product created from database rows!\")\n15               case Some(c) =>\n16                 c.id must be(p.id)\n17                 c mustEqual p\n18             }\n19           }\n20       }\n21     }\n22   }\n23 \n24   \"the product does already exist\" must {\n25     \"return an error and not change the database\" in {\n26       (genProduct.sample, genProduct.sample) match {\n27         case (Some(a), Some(b)) =>\n28           val p = b.copy(id = a.id)\n29           for {\n30             cnts <- repo.saveProduct(a)\n31             nosv <- repo.saveProduct(p).recover {\n32               case _ => 0\n33             }\n34             rows <- repo.loadProduct(a.id)\n35           } yield {\n36             withClue(\"Saving a duplicate product must fail!\")(nosv must be(0))\n37             Product.fromDatabase(rows) match {\n38               case None => fail(\"No product created from database rows!\")\n39               case Some(c) =>\n40                 c.id must be(a.id)\n41                 c mustEqual a\n42             }\n43           }\n44         case _ => fail(\"Could not create data sample!\")\n45       }\n46     }\n47   }\n48 }\n

\n\n\n\n\n

Here we test the saving which in the first case should simply write the appropriate data into the database. If the product already exists however this should not happen. Our database constraints will ensure that this does not happen (or so we hope ;-)). Slick will throw an exception which we catch in the test code using the recover method from Future to return a zero indicating no affected database rows. In the end we test for this zero and also check if the originally saved product has not been changed.

\n\n\n

Repository test: updateProduct

 1 \"#updateProduct\" when {\n 2   \"the product does exist\" must {\n 3     \"update the database\" in {\n 4       (genProduct.sample, genProduct.sample) match {\n 5         case (Some(a), Some(b)) =>\n 6           val p = b.copy(id = a.id)\n 7           for {\n 8             cnts <- repo.saveProduct(a)\n 9             upds <- repo.updateProduct(p)\n10             rows <- repo.loadProduct(a.id)\n11           } yield {\n12             withClue(\"Already existing product was not created!\")(\n13               cnts.fold(0)(_ + _) must be(a.names.toNonEmptyList.size + 1)\n14             )\n15             Product.fromDatabase(rows) match {\n16               case None => fail(\"No product created from database rows!\")\n17               case Some(c) =>\n18                 c.id must be(a.id)\n19                 c mustEqual p\n20             }\n21           }\n22         case _ => fail(\"Could not create data sample!\")\n23       }\n24     }\n25   }\n26 \n27   \"the product does not exist\" must {\n28     \"return an error and not change the database\" in {\n29       genProduct.sample match {\n30         case None => fail(\"Could not generate data sample!\")\n31         case Some(p) =>\n32           for {\n33             nosv <- repo.updateProduct(p).recover {\n34               case _ => 0\n35             }\n36             rows <- repo.loadProduct(p.id)\n37           } yield {\n38             withClue(\"Updating a not existing product must fail!\")\n39               (nosv must be(0))\n40             withClue(\"Product must not exist in database!\")\n41               (rows must be(empty))\n42           }\n43       }\n44     }\n45   }\n46 }\n

\n\n\n\n\n

For testing an update we generate two samples, save one to the database, change the id of the other to the one from the first and execute an update. This update should proceed without problems and the data in the database must have been changed correctly.
If the product does not exist then we use the same recover technique like in the saveProduct test.

\n\n

Congratulations, we have made a check mark on our first integration test using a real database using randomly generated data!

\n\n

\nTesting the routes

\n\n

Regarding the route testing there several options as always. For this one we will define some use cases and then develop some more helper code which will allow us to fire up our routes and do real HTTP requests and then check the database and the responses. We build upon our BaseSpec class and call it BaseUseCaseSpec. In it we will do some more things like define a global base URL which can be used from the tests to make correct requests. Additionally we will write a small actor which simply starts an Akka-HTTP server.

\n\n\n

An actor for starting our routes

 1 final class BaseUseCaseActor(repo: Repository, mat: ActorMaterializer)\n 2   extends Actor with ActorLogging {\n 3   import context.dispatcher\n 4 \n 5   implicit val system: ActorSystem             = context.system\n 6   implicit val materializer: ActorMaterializer = mat\n 7 \n 8   override def receive: Receive = {\n 9     case BaseUseCaseActorCmds.Start =>\n10       val productRoutes  = new ProductRoutes(repo)\n11       val productsRoutes = new ProductsRoutes(repo)\n12       val routes         = productRoutes.routes ~ productsRoutes.routes\n13       val host           = context.system.settings.config.getString(\"api.host\")\n14       val port           = context.system.settings.config.getInt(\"api.port\")\n15       val _              = Http().bindAndHandle(routes, host, port)\n16     case BaseUseCaseActorCmds.Stop =>\n17       context.stop(self)\n18   }\n19 }\n20 \n21 object BaseUseCaseActor {\n22   def props(repo: Repository, mat: ActorMaterializer): Props =\n23     Props(new BaseUseCaseActor(repo, mat))\n24 \n25   sealed trait BaseUseCaseActorCmds\n26 \n27   object BaseUseCaseActorCmds {\n28     case object Start extends BaseUseCaseActorCmds\n29     case object Stop extends BaseUseCaseActorCmds\n30   }\n31 }\n

\n\n\n\n\n

As you can see, the actor is quite simple. Upon receiving the Start command it will initialise the routes and start an Akka-HTTP server. To be able to do this it needs the database repository and an actor materializer which are passed via the constructor. Regarding the BaseUseCaseSpec we will concentrate on the code that differs from the base class.

\n\n\n

Base class for use case testing

 1 abstract class BaseUseCaseSpec\n 2   extends TestKit(\n 3     ActorSystem(\n 4       \"it-test\",\n 5       ConfigFactory\n 6         .parseString(s\"api.port=${BaseUseCaseSpec.findAvailablePort()}\")\n 7         .withFallback(ConfigFactory.load())\n 8     )\n 9   )\n10   with AsyncWordSpecLike\n11   with MustMatchers\n12   with ScalaCheckPropertyChecks\n13   with BeforeAndAfterAll\n14   with BeforeAndAfterEach {\n15   // ...\n16   final val baseUrl: String = s\"\"\"http://${system.settings.config\n17     .getString(\"api.host\")}:${system.settings.config\n18     .getInt(\"api.port\")}\"\"\"\n19 \n20   protected val dbConfig: DatabaseConfig[JdbcProfile] =\n21     DatabaseConfig.forConfig(\"database\", system.settings.config)\n22   protected val repo = new Repository(dbConfig)\n23 \n24   override protected def beforeAll(): Unit = {\n25     val _ = flyway.migrate()\n26     val a = system.actorOf(BaseUseCaseActor.props(repo, materializer))\n27     a ! BaseUseCaseActorCmds.Start\n28   }\n29   // ...\n30 }\n

\n\n\n\n\n

Here we create our base URL and our database repository while we use the beforeAll function to initialise our actor before any test is run. Please note that this has the same drawback like sharing the repository across tests: If a test crashes your service then the others will be affected too.
But let’s write a test for our first use case: loading a product!

\n\n

Again we will use the beforeEach and afterEach helpers to clean up our database. Now let’s take a look at a test for loading a product that does not exist.

\n\n\n

Use case: load a non existing product

 1 \"Loading a Product by ID\" when {\n 2   \"the ID does not exist\" must {\n 3     val expectedStatus = StatusCodes.NotFound\n 4 \n 5     s\"return $expectedStatus\" in {\n 6       val id = UUID.randomUUID\n 7 \n 8       for {\n 9         resp <- http.singleRequest(\n10                   HttpRequest(\n11                     method = HttpMethods.GET,\n12                     uri = s\"$baseUrl/product/$id\",\n13                     headers = Seq(),\n14                     entity = HttpEntity(\n15                       contentType = ContentTypes.`application/json`,\n16                       data = ByteString(\"\")\n17                     )\n18                   )\n19                 )\n20       } yield {\n21         resp.status must be(expectedStatus)\n22       }\n23     }\n24   }\n25 }\n

\n\n\n\n\n

Maybe a bit verbose but we do a real HTTP request here and check the response status code. So how does it look if we run it?

\n\n\n

1 [info] LoadProduct:\n2 [info] Loading a Product by ID\n3 [info]   when the ID does not exist\n4 [info]   - must return 404 Not Found *** FAILED ***\n5 [info]     200 OK was not equal to 404 Not Found (LoadProduct.scala:88)\n

\n\n\n\n\n

That is not cool. What happened? Let’s take a look at the code.

\n\n\n

1 get {\n2   complete {\n3     for {\n4       rows <- repo.loadProduct(id)\n5       prod <- Future { Product.fromDatabase(rows) }\n6     } yield prod\n7   }\n8 }\n

\n\n\n\n\n

Well, no wonder - we are simply returning the output of our fromDatabase helper function which may be empty. This will result in an HTTP status code 200 with an empty body. If you don’t believe me just fire up the service and do a request by hand via curl or httpie.

\n\n\n\n

Luckily for us Akka-HTTP has us covered with the rejectEmptyResponse directive which we can use.

\n\n\n

 1 get {\n 2   rejectEmptyResponse {\n 3     complete {\n 4       for {\n 5         rows <- repo.loadProduct(id)\n 6         prod <- Future { Product.fromDatabase(rows) }\n 7       } yield prod\n 8     }\n 9   }\n10 }\n

\n\n\n\n\n

Cool, it seems we’re set with this one. So onward to testing to load an existing product via the API.

\n\n\n

Use case: load a product

 1 \"Loading a Product by ID\" when {\n 2   \"the ID does exist\" must {\n 3     val expectedStatus = StatusCodes.OK\n 4 \n 5     s\"return $expectedStatus and the Product\" in {\n 6       genProduct.sample match {\n 7         case None => fail(\"Could not generate data sample!\")\n 8         case Some(p) =>\n 9           for {\n10             _    <- repo.saveProduct(p)\n11             rows <- repo.loadProduct(p.id)\n12             resp <- http.singleRequest(\n13                       HttpRequest(\n14                         method = HttpMethods.GET,\n15                         uri = s\"$baseUrl/product/${p.id}\",\n16                         headers = Seq(),\n17                         entity = HttpEntity(\n18                           contentType = ContentTypes.`application/json`,\n19                           data = ByteString(\"\")\n20                         )\n21                       )\n22                     )\n23             body <- resp.entity.dataBytes.runFold(ByteString(\"\"))(_ ++ _)\n24           } yield {\n25             withClue(\"Seeding product data failed!\")(rows must not be(empty))\n26             resp.status must be(expectedStatus)\n27             decode[Product](body.utf8String) match {\n28               case Left(e)  => fail(s\"Could not decode response: $e\")\n29               case Right(d) => d mustEqual p\n30             }\n31           }\n32       }\n33     }\n34   }\n35 }\n

\n\n\n\n\n

Here we simple save our generated product into the database before executing our request. We also check if the product has actually been written to be on the safe side. Additionally we also check if the decoded response body matches the product we expect. In contrast to our first test this one works instantly so not all hope is lost for our coding skills. ;-)
We’ll continue with the use case of saving (or creating) a product via the API. This time we will make the code snippets shorter.

\n\n\n

Use case: save a product (1)

 1 val expectedStatus = StatusCodes.BadRequest\n 2 \n 3 s\"return $expectedStatus\" in {\n 4   for {\n 5     resp <- http.singleRequest(\n 6       HttpRequest(\n 7         method = HttpMethods.POST,\n 8         uri = s\"$baseUrl/products\",\n 9         headers = Seq(),\n10         entity = HttpEntity(\n11           contentType = ContentTypes.`application/json`,\n12           data = ByteString(\n13             scala.util.Random.alphanumeric.take(256).mkString\n14           )\n15         )\n16       )\n17     )\n18   } yield {\n19     resp.status must be(expectedStatus)\n20   }\n21 }\n

\n\n\n\n\n

Here we test posting garbage instead of valid JSON to the endpoint which must result in a “400 Bad Request” returned to us.

\n\n\n

Use case: save a product (2)

 1 val expectedStatus = StatusCodes.InternalServerError\n 2 \n 3 s\"return $expectedStatus and not save the Product\" in {\n 4   (genProduct.sample, genProduct.sample) match {\n 5     case (Some(a), Some(b)) =>\n 6       val p = b.copy(id = a.id)\n 7       for {\n 8         _    <- repo.saveProduct(a)\n 9         rows <- repo.loadProduct(a.id)\n10         resp <- http.singleRequest(\n11           HttpRequest(\n12             method = HttpMethods.POST,\n13             uri = s\"$baseUrl/products\",\n14             headers = Seq(),\n15             entity = HttpEntity(\n16               contentType = ContentTypes.`application/json`,\n17               data = ByteString(p.asJson.noSpaces)\n18             )\n19           )\n20         )\n21         rows2 <- repo.loadProduct(a.id)\n22       } yield {\n23         withClue(\"Seeding product data failed!\")(rows must not be(empty))\n24         resp.status must be(expectedStatus)\n25         Product.fromDatabase(rows2) match {\n26           case None    =>\n27             fail(\"Seeding product was not saved to database!\")\n28           case Some(s) =>\n29             withClue(\"Existing product must not be changed!\")(s mustEqual a)\n30         }\n31       }\n32     case _ => fail(\"Could not generate data sample!\")\n33   }\n34 }\n

\n\n\n\n\n

This one executes it bit more but basically we try to save (or better create) an already existing product. Therefore the constraints of our database should product an error which in turn must return a “500 Internal Server Error” to use. Additionally we verify that the existing product in the database was not changed.

\n\n\n

Use case: save a product (3)

 1 val expectedStatus = StatusCodes.OK\n 2 \n 3 s\"return $expectedStatus and save the Product\" in {\n 4   genProduct.sample match {\n 5     case None => fail(\"Could not generate data sample!\")\n 6     case Some(p) =>\n 7       for {\n 8         resp <- http.singleRequest(\n 9           HttpRequest(\n10             method = HttpMethods.POST,\n11             uri = s\"$baseUrl/products\",\n12             headers = Seq(),\n13             entity = HttpEntity(\n14               contentType = ContentTypes.`application/json`,\n15               data = ByteString(p.asJson.noSpaces)\n16             )\n17           )\n18         )\n19         rows <- repo.loadProduct(p.id)\n20       } yield {\n21         resp.status must be(expectedStatus)\n22         Product.fromDatabase(rows) match {\n23           case None    => fail(\"Product was not saved to database!\")\n24           case Some(s) => s mustEqual p\n25         }\n26       }\n27   }\n28 }\n

\n\n\n\n\n

Last but not least we are testing to save a not already existing valid product into the database. Again we check for the expected status code of “200 OK” and verify that the product saved into the database is the one we sent to the API. Let’s move on to testing the loading of all products now.

\n\n\n

Use case: load all products (1)

 1 val expectedStatus = StatusCodes.OK\n 2 \n 3 s\"return $expectedStatus and an empty list\" in {\n 4   for {\n 5     resp <- http.singleRequest(\n 6               HttpRequest(\n 7                 method = HttpMethods.GET,\n 8                 uri = s\"$baseUrl/products\",\n 9                 headers = Seq(),\n10                 entity = HttpEntity(\n11                   contentType = ContentTypes.`application/json`,\n12                   data = ByteString(\"\")\n13                 )\n14               )\n15             )\n16     body <- resp.entity.dataBytes.runFold(ByteString(\"\"))(_ ++ _)\n17   } yield {\n18     resp.status must be(expectedStatus)\n19     decode[List[Product]](body.utf8String) match {\n20       case Left(e)  => fail(s\"Could not decode response: $e\")\n21       case Right(d) => d must be(empty)\n22     }\n23   }\n24 }\n

\n\n\n\n\n

Our first test case is loading all products if no product exists so we expect an empty list here and the appropriate status code.

\n\n\n

Use case: load all products (2)

 1 val expectedStatus = StatusCodes.OK\n 2 \n 3 s\"return $expectedStatus and a list with all products\" in {\n 4   genProducts.sample match {\n 5     case None => fail(\"Could not generate data sample!\")\n 6     case Some(ps) =>\n 7       for {\n 8         _    <- Future.sequence(ps.map(p => repo.saveProduct(p)))\n 9         resp <- http.singleRequest(\n10                   HttpRequest(\n11                     method = HttpMethods.GET,\n12                     uri = s\"$baseUrl/products\",\n13                     headers = Seq(),\n14                     entity = HttpEntity(\n15                       contentType = ContentTypes.`application/json`,\n16                       data = ByteString(\"\")\n17                     )\n18                   )\n19                 )\n20         body <- resp.entity.dataBytes.runFold(ByteString(\"\"))(_ ++ _)\n21       } yield {\n22         resp.status must be(expectedStatus)\n23         decode[List[Product]](body.utf8String) match {\n24           case Left(e)  => fail(s\"Could not decode response: $e\")\n25           case Right(d) => d.sorted mustEqual ps.sorted\n26         }\n27       }\n28   }\n29 }\n

\n\n\n\n\n

This one is also straight forward: We save our generated list of products to the database and query the API which must return a “200 OK” status code and a correct list of products in JSON format. Looks like we have one more use case to tackle: Updating a product via the API.

\n\n\n

Use case: update a product (1)

 1 val expectedStatus = StatusCodes.BadRequest\n 2 \n 3 s\"return $expectedStatus\" in {\n 4   genProduct.sample match {\n 5     case None => fail(\"Could not generate data sample!\")\n 6     case Some(p) =>\n 7       for {\n 8         _    <- repo.saveProduct(p)\n 9         rows <- repo.loadProduct(p.id)\n10         resp <- http.singleRequest(\n11           HttpRequest(\n12             method = HttpMethods.PUT,\n13             uri = s\"$baseUrl/product/${p.id}\",\n14             headers = Seq(),\n15             entity = HttpEntity(\n16               contentType = ContentTypes.`application/json`,\n17               data = ByteString(scala.util.Random.alphanumeric.take(256).mkString)\n18             )\n19           )\n20         )\n21         rows2 <- repo.loadProduct(p.id)\n22       } yield {\n23         withClue(\"Seeding product data failed!\")(rows must not be(empty))\n24         resp.status must be(expectedStatus)\n25         Product.fromDatabase(rows2) match {\n26           case None    =>\n27             fail(\"Seeding product was not saved to database!\")\n28           case Some(s) =>\n29             withClue(\"Existing product must not be changed!\")(s mustEqual p)\n30         }\n31       }\n32   }\n

\n\n\n\n\n

First we test with garbage JSON in the request. Before doing the request we actually create a product to avoid getting an error caused by a possibly missing product. Afterwards we check for the expected “400 Bad Request” status code and verify that our product has not been updated.

\n\n\n

Use case: update product (2)

 1 val expectedStatus = StatusCodes.OK\n 2 \n 3 s\"return $expectedStatus and update the Product\" in {\n 4   (genProduct.sample, genProduct.sample) match {\n 5     case (Some(a), Some(b)) =>\n 6       val p = b.copy(id = a.id)\n 7       for {\n 8         _    <- repo.saveProduct(a)\n 9         rows <- repo.loadProduct(a.id)\n10         resp <- http.singleRequest(\n11           HttpRequest(\n12             method = HttpMethods.PUT,\n13             uri = s\"$baseUrl/product/${p.id}\",\n14             headers = Seq(),\n15             entity = HttpEntity(\n16               contentType = ContentTypes.`application/json`,\n17               data = ByteString(p.asJson.noSpaces)\n18             )\n19           )\n20         )\n21         rows2 <- repo.loadProduct(p.id)\n22       } yield {\n23         withClue(\"Seeding product data failed!\")(rows must not be(empty))\n24         resp.status must be(expectedStatus)\n25         Product.fromDatabase(rows2) match {\n26           case None    => fail(\"Seeding product was not saved to database!\")\n27           case Some(s) => s mustEqual p\n28         }\n29       }\n30     case _ => fail(\"Could not generate data sample!\")\n31   }\n32 }\n

\n\n\n\n\n

Next we test updating an existing product using valid JSON. We check the status code and if the product has been correctly updated within the database.

\n\n\n

Use case: update product (3)

 1 val expectedStatus = StatusCodes.InternalServerError\n 2 \n 3 s\"return $expectedStatus\" in {\n 4   genProduct.sample match {\n 5     case None => fail(\"Could not generate data sample!\")\n 6     case Some(p) =>\n 7       for {\n 8         resp <- http.singleRequest(\n 9           HttpRequest(\n10             method = HttpMethods.PUT,\n11             uri = s\"$baseUrl/product/${p.id}\",\n12             headers = Seq(),\n13             entity = HttpEntity(\n14               contentType = ContentTypes.`application/json`,\n15               data = ByteString(p.asJson.noSpaces)\n16             )\n17           )\n18         )\n19         rows <- repo.loadProduct(p.id)\n20       } yield {\n21         resp.status must be(expectedStatus)\n22         rows must be(empty)\n23       }\n24   }\n25 }\n

\n\n\n\n\n

Finally we test updating a non existing product which should produce the expected status code and not save into the database. Wow, it seems we done for good with our impure implementation. Well except for some benchmarking but let’s save that for later.
If we look onto our test code coverage (which is a metric that you should use) then things look pretty good. We are missing some parts but in general we should have things covered.

\n\n\n\n

\nTesting the pure service

\n\n

We will skip the explanation of the ScalaCheck generators because they only differ slightly from the ones used in the impure part.

\n\n

\nUnit Tests

\n\n

The model tests are omitted here because they are basically the same as in the impure section. If you are interested in them just look at the source code. In contrast to the impure part we will now write unit tests for our routes. Meaning we will be able to test our routing logic without spinning up a database.

\n\n

\nTesting our routes

\n\n

To be able to test our routes we will have to implement a TestRepository first which we will use instead of the concrete implementation which is wired to a database.

\n\n\n

TestRepository implementation - take 1

 1 class TestRepository[F[_]: Effect](data: Seq[Product]) extends Repository[F] {\n 2   override def loadProduct(id: ProductId) = {\n 3     data.find(_.id === id) match {\n 4       case None => Seq.empty.pure[F]\n 5       case Some(p) =>\n 6         val ns = p.names.toNonEmptyList.toList.to[Seq]\n 7         ns.map(n => (p.id, n.lang, n.name)).pure[F]\n 8     }\n 9   }\n10 \n11   override def loadProducts() = {\n12     Stream.empty\n13   }\n14 \n15   override def saveProduct(p: Product): F[Int] =\n16     data.find(_.id === p.id).fold(0.pure[F])(_ => 1.pure[F])\n17 \n18   override def updateProduct(p: Product): F[Int] =\n19     data.find(_.id === p.id).fold(0.pure[F])(_ => 1.pure[F])\n20 \n21 }\n

\n\n\n\n\n

As you can see we try to stay abstract (using our HKT F[_] here) and we have basically left out the implementation of loadProducts because it will just return an empty stream. We will get back to it later on. Aside from that the class can be initialised with a (potentially empty) list of Product entities which will be used as a “database”. The save and update functions won’t change any data, they will just return a 0 or a 1 depending on the product being present in the seed data list.

\n\n\n

Unit test: Product routes (1)

 1 val emptyRepository: Repository[IO] = new TestRepository[IO](Seq.empty)\n 2 val expectedStatusCode = Status.NotFound\n 3 \n 4 s\"return $expectedStatusCode\" in {\n 5   forAll(\"id\") { id: ProductId =>\n 6     Uri.fromString(\"/product/\" + id.toString) match {\n 7       case Left(_) => fail(\"Could not generate valid URI!\")\n 8       case Right(u) =>\n 9         def service: HttpRoutes[IO] =\n10           Router(\"/\" -> new ProductRoutes(emptyRepository).routes)\n11         val response: IO[Response[IO]] = service.orNotFound.run(\n12           Request(method = Method.GET, uri = u)\n13         )\n14         val result = response.unsafeRunSync\n15         result.status must be(expectedStatusCode)\n16         result.body.compile.toVector.unsafeRunSync must be(empty)\n17     }\n18   }\n19 }\n

\n\n\n\n\n

Above you can see the test for querying a non existing product which must return an empty response using a “404 Not Found” status code. First we try to create a valid URI from our generated ProductId. If that succeeds we create a small service wrapper for our routes in which we inject our empty TestRepository. Finally we create a response using this service and a request that we construct. Because we are in the land of IO we have to actually execute it (via unsafeRunSync) to get any results back. Finally we validate the status code and the response body.

\n\n\n\n\n

1 [info]   when GET /product/ID\n2 [info]     when product does not exist\n3 [info]     - must return 404 Not Found *** FAILED ***\n4 [info]       TestFailedException was thrown during property evaluation.\n5 [info]         Message: 200 OK was not equal to 404 Not Found\n6 [info]         Location: (ProductRoutesTest.scala:46)\n7 [info]         Occurred when passed generated values (\n8 [info]           id = 3f298ae4-4f7b-415b-9888-c2e9b34a4883\n9 [info]         )\n

\n\n\n\n\n

It seems that we just pass an empty response if we do not find the product. This is not nice, so let’s fix this. The culprit is the following line in our ProductRoutes file:

\n\n\n

1 for {\n2   // ...\n3   resp <- Ok(Product.fromDatabase(rows))\n4 } yield resp\n

\n\n\n\n\n

Okay, this looks easy. Let’s try this one:

\n\n\n

1 for {\n2   // ...\n3   resp <- Product.fromDatabase(rows).fold(NotFound())(p => Ok(p))\n4 } yield resp\n

\n\n\n\n\n

Oh no, the compiler complains:

\n\n\n

1 Cannot convert from Product to an Entity, because no \n2   EntityEncoder[F, com.wegtam.books.pfhais.pure.models.Product] \n3   instance could be found.\n

\n\n\n\n\n

Right, before we had an Option[Product] here for which we created an implicit JSON encoder. So if we create one for the Product itself then we should be fine.

\n\n\n

1 implicit def encodeProduct[A[_]: Applicative]: EntityEncoder[A, Product] = \n2   jsonEncoderOf\n

\n\n\n\n\n

Now back to our test:

\n\n\n

1 [info]   when GET /product/ID\n2 [info]     when product does not exist\n3 [info]     - must return 404 Not Found\n

\n\n\n\n\n

Great! Seems like we are doing fine, so let’s continue. Next in line is testing a query for an existing product.

\n\n\n

Unit test: Product routes (2)

 1 implicit def decodeProduct: EntityDecoder[IO, Product] = jsonOf\n 2 val expectedStatusCode = Status.Ok\n 3 \n 4 s\"return $expectedStatusCode and the product\" in {\n 5   forAll(\"product\") { p: Product =>\n 6     Uri.fromString(\"/product/\" + p.id.toString) match {\n 7       case Left(_) => fail(\"Could not generate valid URI!\")\n 8       case Right(u) =>\n 9         val repo: Repository[IO] = new TestRepository[IO](Seq(p))\n10         def service: HttpRoutes[IO] =\n11           Router(\"/\" -> new ProductRoutes(repo).routes)\n12         val response: IO[Response[IO]] = service.orNotFound.run(\n13           Request(method = Method.GET, uri = u)\n14         )\n15         val result = response.unsafeRunSync\n16         result.status must be(expectedStatusCode)\n17         result.as[Product].unsafeRunSync must be(p)\n18     }\n19   }\n20 }\n

\n\n\n\n\n

This time we generate a whole product, again try to create a valid URI and continue as before. But this time we inject our TestRepository containing a list with our generated product. In the end we test our expected status code and the response body must contain our product. For the last part to work we must have an implicit EntityDecoder in scope.

\n\n\n

Unit test: Product routes (3)

 1 val expectedStatusCode = Status.BadRequest\n 2 \n 3 s\"return $expectedStatusCode\" in {\n 4   forAll(\"id\") { id: ProductId =>\n 5     Uri.fromString(\"/product/\" + id.toString) match {\n 6       case Left(_) => fail(\"Could not generate valid URI!\")\n 7       case Right(u) =>\n 8         def service: HttpRoutes[IO] =\n 9           Router(\"/\" -> new ProductRoutes(emptyRepository).routes)\n10         val payload = scala.util.Random.alphanumeric.take(256).mkString\n11         val response: IO[Response[IO]] = service.orNotFound.run(\n12           Request(method = Method.PUT, uri = u)\n13             .withEntity(payload.asJson.noSpaces)\n14         )\n15         val result = response.unsafeRunSync\n16         result.status must be(expectedStatusCode)\n17         result.body.compile.toVector.unsafeRunSync must be(empty)\n18     }\n19   }\n20 }\n

\n\n\n\n\n

Here we are trying to update a product which doesn’t have to exist because we send totally garbage JSON with the request which should result in a “400 Bad Request” status code. However if we run our test we get an exception instead:

\n\n\n

1 [info]   when PUT /product/ID\n2 [info]     when request body is invalid\n3 [info]     - must return 400 Bad Request *** FAILED ***\n4 [info]       InvalidMessageBodyFailure was thrown during property\n5 [info]         Occurred when passed generated values (\n6 [info]           id = 932682c9-1f9a-463a-84db-a0992d466aa3\n7 [info]         )\n

\n\n\n\n\n

So let’s take a deep breath and look at our code:

\n\n\n

1 case req @ PUT -> Root / \"product\" / UUIDVar(id) =>\n2   for {\n3     p <- req.as[Product]\n4     _ <- repo.updateProduct(p)\n5     r <- NoContent()\n6   } yield r\n

\n\n\n\n\n

As we can see we do no error handling at all. So maybe we can rewrite this a little bit.

\n\n\n

1 case req @ PUT -> Root / \"product\" / UUIDVar(id) =>\n2   req\n3     .as[Product]\n4     .flatMap { p =>\n5       repo.updateProduct(p) *> NoContent()\n6     }\n7     .handleErrorWith {\n8       case InvalidMessageBodyFailure(_, _) => BadRequest()\n9     }\n

\n\n\n\n\n

Now we explicitly handle any error which occurs when decoding the request entity. But in fact: I am lying to you. We only handle the invalid message body failure here. On the other hand it is enough to make our test happy. :-)

\n\n\n\n

Onwards to our next test cases in which we use a valid JSON payload for our request.

\n\n\n

Unit test: Product routes (4)

 1 val expectedStatusCode = Status.NotFound\n 2 \n 3 s\"return $expectedStatusCode\" in {\n 4   forAll(\"product\") { p: Product =>\n 5     Uri.fromString(\"/product/\" + p.id.toString) match {\n 6       case Left(_) => fail(\"Could not generate valid URI!\")\n 7       case Right(u) =>\n 8         def service: HttpRoutes[IO] =\n 9           Router(\"/\" -> new ProductRoutes(emptyRepository).routes)\n10         val response: IO[Response[IO]] = service.orNotFound.run(\n11           Request(method = Method.PUT, uri = u)\n12             .withEntity(p)\n13         )\n14         val result = response.unsafeRunSync\n15         result.status must be(expectedStatusCode)\n16         result.body.compile.toVector.unsafeRunSync must be(empty)\n17     }\n18   }\n19 }\n

\n\n\n\n\n

We expect a “404 Not Found” if we try to update a valid product which does not exist. But what do we get in the tests?

\n\n\n

1 Message: 204 No Content was not equal to 404 Not Found\n

\n\n\n\n\n

Well not exactly what we planned for but it is our own fault. We used the *> operator which ignores the value from the previous operation. So we need to fix that.

\n\n\n

 1 case req @ PUT -> Root / \"product\" / UUIDVar(id) =>\n 2   req\n 3     .as[Product]\n 4     .flatMap { p =>\n 5       for {\n 6         cnt <- repo.updateProduct(p)\n 7         res <- cnt match {\n 8           case 0 => NotFound()\n 9           case _ => NoContent()\n10         }\n11       } yield res\n12     }\n13     .handleErrorWith {\n14       case InvalidMessageBodyFailure(_, _) => BadRequest()\n15     }\n

\n\n\n\n\n

We rely on the return value of our update function which contains the number of affected database rows. If it is zero then nothing has been done implying that the product was not found. Otherwise we return our “204 No Content” response as before. Still we miss one last test for our product routes.

\n\n\n

Unit test: Product routes (5)

 1 val expectedStatusCode = Status.NoContent\n 2 \n 3 s\"return $expectedStatusCode\" in {\n 4   forAll(\"product\") { p: Product =>\n 5     Uri.fromString(\"/product/\" + p.id.toString) match {\n 6       case Left(_) => fail(\"Could not generate valid URI!\")\n 7       case Right(u) =>\n 8         val repo: Repository[IO] = new TestRepository[IO](Seq(p))\n 9         def service: HttpRoutes[IO] =\n10           Router(\"/\" -> new ProductRoutes(repo).routes)\n11         val response: IO[Response[IO]] = service.orNotFound.run(\n12           Request(method = Method.PUT, uri = u)\n13             .withEntity(p)\n14         )\n15         val result = response.unsafeRunSync\n16         result.status must be(expectedStatusCode)\n17         result.body.compile.toVector.unsafeRunSync must be(empty)\n18     }\n19   }\n20 }\n

\n\n\n\n\n

Basically this is the same test as before with the exception that we now give our routes a properly seeded test repository. Great, we have tested our ProductRoutes and without having to spin up a database! But we still have work to do, so let’s move on to testing the ProductsRoutes implementation. Before we do that we adapt the code for creating a product using our gained knowledge from our update test.

\n\n\n

ProductsRoutes: adapted create endpoint

 1 case req @ POST -> Root / \"products\" =>\n 2   req\n 3     .as[Product]\n 4     .flatMap { p =>\n 5       for {\n 6         cnt <- repo.saveProduct(p)\n 7         res <- cnt match {\n 8           case 0 => NotFound()\n 9           case _ => InternalServerError()\n10         }\n11       } yield res\n12     }\n13     .handleErrorWith {\n14       case InvalidMessageBodyFailure(_, _) => BadRequest()\n15     }\n

\n\n\n\n\n

Now to our tests, we will start with sending garbage JSON via the POST request.

\n\n\n

Unit test: Products routes (1)

 1 val expectedStatusCode = Status.BadRequest\n 2 \n 3 s\"return $expectedStatusCode\" in {\n 4   def service: HttpRoutes[IO] =\n 5     Router(\"/\" -> new ProductsRoutes(emptyRepository).routes)\n 6   val payload = scala.util.Random.alphanumeric.take(256).mkString\n 7   val response: IO[Response[IO]] = service.orNotFound.run(\n 8     Request(method = Method.POST, uri = Uri.uri(\"/products\"))\n 9       .withEntity(payload.asJson.noSpaces)\n10   )\n11   val result = response.unsafeRunSync\n12   result.status must be(expectedStatusCode)\n13   result.body.compile.toVector.unsafeRunSync must be(empty)\n14 }\n

\n\n\n\n\n

There is nothing special here, the test is same as for the ProductRoutes except for the changed URI and HTTP method. Also the code is a bit simpler because we do not need to generate a dynamic request URI like before.

\n\n\n

Unit test: Products routes (2)

 1 val expectedStatusCode = Status.NoContent\n 2 \n 3 s\"return $expectedStatusCode\" in {\n 4   forAll(\"product\") { p: Product =>\n 5     val repo: Repository[IO] = new TestRepository[IO](Seq(p))\n 6     def service: HttpRoutes[IO] =\n 7       Router(\"/\" -> new ProductsRoutes(repo).routes)\n 8     val response: IO[Response[IO]] = service.orNotFound.run(\n 9       Request(method = Method.POST, uri = Uri.uri(\"/products\"))\n10         .withEntity(p)\n11     )\n12     val result = response.unsafeRunSync\n13     result.status must be(expectedStatusCode)\n14     result.body.compile.toVector.unsafeRunSync must be(empty)\n15   }\n16 }\n

\n\n\n\n\n

Saving a product using valid JSON payload should succeed and in fact it does because of the code we have in our TestRepository instance. If you remember we use the following code for saveProduct:

\n\n\n

1 override def saveProduct(p: Product): F[Int] =\n2   data.find(_.id === p.id).fold(0.pure[F])(_ => 1.pure[F])\n

\n\n\n\n\n

This code will return a 0 if the product we try to save does not exist in the seed data set and only a 1 if it can be found within aforementioned set. This code clearly doesn’t make any sense except for our testing. This way we can ensure the behaviour of the save function without having to create a new Repository instance with hard coded behaviour. :-)

\n\n\n

Unit test: Products routes (3)

 1 val expectedStatusCode = Status.InternalServerError\n 2 \n 3 s\"return $expectedStatusCode\" in {\n 4   forAll(\"product\") { p: Product =>\n 5     def service: HttpRoutes[IO] =\n 6       Router(\"/\" -> new ProductsRoutes(emptyRepository).routes)\n 7     val response: IO[Response[IO]] = service.orNotFound.run(\n 8       Request(method = Method.POST, uri = Uri.uri(\"/products\"))\n 9         .withEntity(p)\n10     )\n11     val result = response.unsafeRunSync\n12     result.status must be(expectedStatusCode)\n13     result.body.compile.toVector.unsafeRunSync must be(empty)\n14   }\n15 }\n

\n\n\n\n\n

We use the empty repository this time to ensure that the saveProduct function will return a zero, triggering the desired logic in our endpoint. Almost done, so let’s check the endpoint for returning all products.

\n\n\n

Unit test: Products routes (4)

 1 val expectedStatusCode = Status.Ok\n 2 \n 3 s\"return $expectedStatusCode and an empty list\" in {\n 4   def service: HttpRoutes[IO] =\n 5     Router(\"/\" -> new ProductsRoutes(emptyRepository).routes)\n 6   val response: IO[Response[IO]] = service.orNotFound.run(\n 7     Request(method = Method.GET, uri = Uri.uri(\"/products\"))\n 8   )\n 9   val result = response.unsafeRunSync\n10   result.status must be(expectedStatusCode)\n11   result.as[List[Product]].unsafeRunSync mustEqual List.empty[Product]\n12 }\n

\n\n\n\n\n

We simply expect an empty list if no products exist. It is as simple as that and works right out of the box. Last but not least we need to test the return of existing products. But before we do this let’s take a look at our TestRepository implementation.

\n\n\n

TestRepository: stubbed loadProducts

1 override def loadProducts() = Stream.empty\n

\n\n\n\n\n

Uh, oh, that does not bode well! So we will need to fix that first. Because we were wise to chose fs2 as our streaming library of choice the solution is as simple as this.

\n\n\n

TestRepository: fixed loadProducts

1 override def loadProducts() = {\n2   val rows = data.flatMap { p =>\n3     val ns = p.names.toNonEmptyList.toList.to[Seq]\n4     ns.map(n => (p.id, n.lang, n.name))\n5   }\n6   Stream.emits(rows)\n7 }\n

\n\n\n\n\n

Now we can write our last test.

\n\n\n

Unit test: Products routes (5)

 1 implicit def decodeProducts: EntityDecoder[IO, List[Product]] = jsonOf\n 2 val expectedStatusCode = Status.Ok\n 3 \n 4 s\"return $expectedStatusCode and a list of products\" in {\n 5   forAll(\"products\") { ps: List[Product] =>\n 6     val repo: Repository[IO] = new TestRepository[IO](ps)\n 7     def service: HttpRoutes[IO] =\n 8       Router(\"/\" -> new ProductsRoutes(repo).routes)\n 9     val response: IO[Response[IO]] = service.orNotFound.run(\n10       Request(method = Method.GET, uri = Uri.uri(\"/products\"))\n11     )\n12     val result = response.unsafeRunSync\n13     result.status must be(expectedStatusCode)\n14     result.as[List[Product]].unsafeRunSync mustEqual ps\n15   }\n16 }\n

\n\n\n\n\n

To decode the response correctly an implicit EntityDecoder of the appropriate type is needed in scope. But the rest of the test should look pretty familiar to you by now.

\n\n\n\n

It seems the only parts left to test are the FlywayDatabaseMigrator and our DoobieRepository classes. Testing them will require a running database so we are leaving the cosy world of unit tests behind and venture forth into integration test land. But fear not, we already have some - albeit impure - experience here.

\n\n

\nIntegration Tests

\n\n

As usual we start up by implementing a base class that we can use to provide common settings and functions across our tests.

\n\n\n

BaseSpec for pure integration tests

 1 abstract class BaseSpec extends WordSpec \n 2     with MustMatchers\n 3     with ScalaCheckPropertyChecks\n 4     with BeforeAndAfterAll\n 5     with BeforeAndAfterEach {\n 6 \n 7   protected val config = ConfigFactory.load()\n 8   protected val dbConfig = loadConfig[DatabaseConfig](config, \"database\")\n 9 \n10   override def beforeAll(): Unit = {\n11     val _ = withClue(\"Database configuration could not be loaded!\") {\n12       dbConfig.isRight must be(true)\n13     }\n14   }\n15 }\n

\n\n\n\n\n

You can see that we keep it simple here and only load the database configuration and ensure that is has been indeed loaded correctly in the beforeAll function.

\n\n

\nTesting the FlywayDatabaseMigrator

\n\n

To ensure the basic behaviour of our FlywayDatabaseMigrator we write a simple test.

\n\n\n

Integration test: database migrator (1)

 1 \"the database is not available\" must {\n 2   \"throw an exception\" in {\n 3     val cfg = DatabaseConfig(\n 4       driver = \"This is no driver name!\",\n 5       url = \"jdbc://some.host/whatever\",\n 6       user = \"no-user\",\n 7       pass = \"no-password\"\n 8     )\n 9     val migrator: DatabaseMigrator[IO] = new FlywayDatabaseMigrator\n10     val program = migrator.migrate(cfg.url, cfg.user, cfg.pass)\n11     an[FlywayException] must be thrownBy program.unsafeRunSync\n12   }\n13 }\n

\n\n\n\n\n

Within this test we construct an invalid database configuration and expect that the call to migrate throws an exception. If you remember, we had this issue already and chose not to handle any exceptions but let the calling site do this - for example via a MonadError instance.

\n\n\n

Integration test: database migrator (2)

 1 dbConfig.map { cfg =>\n 2   val migrator: DatabaseMigrator[IO] = new FlywayDatabaseMigrator\n 3   val program = migrator.migrate(cfg.url, cfg.user, cfg.pass)\n 4   program.unsafeRunSync must be > 0\n 5 }\n 6 // ---\n 7 dbConfig.map { cfg =>\n 8   val migrator: DatabaseMigrator[IO] = new FlywayDatabaseMigrator\n 9   val program = migrator.migrate(cfg.url, cfg.user, cfg.pass)\n10   val _ = program.unsafeRunSync\n11   program.unsafeRunSync must be(0)\n12 }\n

\n\n\n\n\n

The other two tests are also quite simple, we just expect it to return either zero or the number of applied migrations depending on the state of the database. It goes without saying that we of course use the beforeEach and afterEach helpers within the test to prepare and clean our database properly.

\n\n

Last but not least we take a look at testing our actual repository implementation which uses Doobie. To avoid trouble we need to define a globally available ContextShift in our test which is as simple as this:

\n\n\n

1 implicit val cs = IO.contextShift(ExecutionContexts.synchronous)\n

\n\n\n\n\n

Now we can start writing our tests.

\n\n\n

Integration test: repository (1)

 1 val tx = Transactor\n 2   .fromDriverManager[IO](c.driver, c.url, c.user, c.pass)\n 3 val repo = new DoobieRepository(tx)\n 4 forAll(\"ID\") { id: ProductId =>\n 5   for {\n 6     rows <- repo.loadProduct(id)\n 7   } yield {\n 8     rows must be(empty)\n 9   }\n10 }\n

\n\n\n\n\n

Here we simply test that the loadProduct function returns an empty list if the requested product does not exist in the database.

\n\n\n

Integration test: repository (2)

1 forAll(\"product\") { p: Product =>\n2   for {\n3     _    <- repo.saveProduct(p)\n4     rows <- repo.loadProduct(p.id)\n5   } yield {\n6     rows must not be(empty)\n7     Product.fromDatabase(rows) must contain(p)\n8   }\n9 }\n

\n\n\n\n\n

From now on we’ll omit the transactor and repository creation from the code examples. As you can see a generated product is saved to the database and loaded again and verified in the end.

\n\n\n

Integration test: repository (3)

1 val rows = repo.loadProducts().compile.toList\n2 rows.unsafeRunSync must be(empty)\n

\n\n\n\n\n

Testing that loadProducts returns an empty stream if no products exist is as simple as the code above. :-)

\n\n\n

Integration test: repository (4)

 1 forAll(\"products\") { ps: List[Product] =>\n 2   for {\n 3     _    <- ps.traverse(repo.saveProduct)\n 4     rows = repo.loadProducts()\n 5       .groupAdjacentBy(_._1)\n 6       .map {\n 7         case (id, rows) => Product.fromDatabase(rows.toList)\n 8       }\n 9       .collect {\n10         case Some(p) => p\n11       }\n12       .compile\n13       .toList\n14   } yield {\n15     val products = rows.unsafeRunSync\n16     products must not be(empty)\n17     products mustEqual ps\n18   }\n19 }\n

\n\n\n\n\n

In contrast the test code for checking the return of existing products is a bit more involving. But let’s step through it together. First we save the list of generated products to the database which we do using the traverse function provided by Cats. In impure land we used Future.sequence here if you remember - but now we want to stay pure. ;-)
Next we call our loadProducts function and apply a part of the logic from our ProductsRoutes to it, namely we construct a proper stream of products which we turn into a list via compile and list in the end. Finally we check that the list is not empty and equal to our generated list.

\n\n\n

Integration test: repository (5)

 1 forAll(\"product\") { p: Product =>\n 2   for {\n 3     cnt  <- repo.saveProduct(p)\n 4     rows <- repo.loadProduct(p.id)\n 5   } yield {\n 6     cnt must be > 0\n 7     rows must not be(empty)\n 8     Product.fromDatabase(rows) must contain(p)\n 9   }\n10 }\n

\n\n\n\n\n

The code for testing saveProduct is nearly identical to the loadProduct test as you can see. We simply check additionally that the function returns the number of affected database rows.

\n\n\n

Integration test: repository (6)

1 forAll(\"product\") { p: Product =>\n2   for {\n3     cnt  <- repo.updateProduct(p)\n4     rows <- repo.loadProduct(p.id)\n5   } yield {\n6     cnt must be(0)\n7     rows must be(empty)\n8   }\n9 }\n

\n\n\n\n\n

Updating a non existing product must return a zero and save nothing to the database, which is what we test above.

\n\n\n

Integration test: repository (7)

 1 forAll(\"productA\", \"productB\") { (a: Product, b: Product) =>\n 2   val p = b.copy(id = a.id)\n 3   for {\n 4     _    <- repo.saveProduct(a)\n 5     cnt  <- repo.updateProduct(p)\n 6     rows <- repo.loadProduct(p.id)\n 7   } yield {\n 8     cnt must be > 0\n 9     rows must not be(empty)\n10     Product.fromDatabase(rows) must contain(p)\n11   }\n12 }\n

\n\n\n\n\n

Finally we test updating a concrete product by generating two of them, saving the first into the database and running an update using the second with the id from the first.

\n\n

Wow, it seems we are finished! Congratulations, we can now check mark the point “write a pure http service in Scala” on our list. :-)

\n\n\n\n

\nAdding benchmarks

\n\n

Now that we have our implementations in place we can start comparing them. We will start with implementing some benchmarks to test the performance of both implementations.

\n\n\n\n

There are several applications available to perform load tests and benchmarks. Regarding the latter the Apache JMeter¹ project is a good starting point. It is quite easy to get something running. Like the documentation says: For the real stuff you should only use the command line application and use the GUI to create and test your benchmark.

\n\n

We’ll skip a long introduction and tutorial for JMeter because you can find a lot within the documentation and there are lots of tutorials online.

\n\n

\nOur environment

\n\n

Within the book repository you’ll find a folder named jmeter it contains several things:

\n\n

Several files ending with .jmx like Pure-Create-Products.jmx and so on.
A CSV file containing 100.000 valid product IDs named product-ids.csv.
A file named benchmarks.md.

\n\n

The .jmx files are the configuration files for JMeter which can be used to run the benchmarks. They are hopefully named understandable and are expected to be run in the following order:

\n\n

Create products
Load products
Update products
Load all products

\n\n\n\n

The file product-ids.csv is expected in the /tmp folder, so you’ll have to copy it there or adjust the benchmark configurations. Finally the file benchmarks.md holds detailed information about the benchmark runs (each one was done three times in a row).

\n\n

\nSystem environment

\n\n

Service and testing software (Apache JMeter) were run on different workstations connected via 100 MBit/s network connection.

\n\n

\nService workstation

\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n

CPU	Core i5-9600K, 6 Cores, 3,7 GHz
RAM	32 GB
HDD	2x Samsung SSD 860 PRO 512GB, SATA
OS	FreeBSD 12 (HT disabled)
JDK	11.0.4+11-2
DB	PostgreSQL 11.3

\n\n

\nClient workstation

\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n

CPU	AMD Ryzen Threadripper 2950X
RAM	32 GB
HDD	2x Samsung SSD 970 PRO 512GB, M.2
OS	FreeBSD 12 (HT disabled)
JDK	11.0.4+11-2

\n\n

Apache JMeter version 5.1.1 was used to run the benchmark and if not noted otherwise 10 threads were used with a 10 seconds ramp up time for each benchmark.

\n\n

\nComparison

\n\n

So let’s start with comparing the results. As mentioned more details can be found in the file benchmarks.md. We’ll stick to using the average of the metrics across all three benchmark runs. The following abbreviations will be used in the tables and legends.

\n\n

AVG: The average response time in milli seconds.
MED: The median response time in milli seconds.
90%: 90 percent of all requests were handled within the response time in milli seconds or less.
95%: 95 percent of all requests were handled within the response time in milli seconds or less.
99%: 99 percent of all requests were handled within the response time in milli seconds or less.
MIN: The minium response time in milli seconds.
MAX: The maximum response time in milli seconds.
ERR: The error rate in percent.
R/S: The number of requests per second that could be handled.
MEM: The maximum amount of memory used by the service during the benchmark in MB.
LD: The average system load on service machine during the benchmark.

\n\n

\nCreate 100.000 products

\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n

Metric	Impure	Pure
AVG	98	12
MED	95	11
90%	129	15
95%	143	18
99%	172	30
MIN	53	5
MAX	1288	675
ERR	0%	0%
R/S	100.56	765.33
MEM	1158	1308
LD	16	9

\n\n

Wow, I honestly have to say that I didn’t expect that. Usually the world has come to believe that the impure approach might be dirty but is definitely always faster. Well it seems we’re about to correct that. I don’t know about you but I’m totally fine with that. ;-)
But now let’s break it apart piece by piece. The first thing that catches the eye is that the pure service seems to be about seven times faster than the impure one! The average 100 requests per second on the impure side stand against an average 765 requests per second on the pure side. Also the metrics regarding the response times support that. Regarding the memory usage we can see that the pure service needed about 13% more memory than the impure one. Living in times in which memory is cheap I consider this a small price to pay for a significant performance boost.
Last but not least I found it very interesting that the average system load was much higher (nearly twice as high) in the impure implementation. While it is okay to make use of your resources a lower utilisation allows more “breathing room” for other tasks (operating system, database, etc.).

\n\n

\nLoad 100.000 products

Metric	Impure	Pure
AVG	7	7
MED	8	7
90%	10	9
95%	11	10
99%	14	19
MIN	4	2
MAX	347	118
ERR	0%	0%
R/S	1162.20	1248.63
MEM	1449	1538
LD	13	8

\n\n

The loading benchmark provides a more balanced picture. While the pure service is still slightly ahead (about 7% faster), it uses about 6% more memory than the impure one. Overall both implementations deliver nearly the same results. But again the pure one causes significant lower system load like in the first benchmark.

\n\n

\nUpdate 100.000 products

Metric	Impure	Pure
AVG	78	12
MED	75	11
90%	104	16
95%	115	20
99%	140	34
MIN	42	5
MAX	798	707
ERR	0%	0%
R/S	125.66	765.26
MEM	1176	1279
LD	16	8

\n\n

Updating existing products results in nearly the same picture as the “create products” benchmark. Interestingly the impure service performs about 20% better on an update than on a create. I have no idea why but it caught my eye. The other metrics are as said nearly identical to the first benchmark. The pure service uses a bit more memory (around 8%) but is around 6 times faster than the impure one causing only half of the system load.

\n\n\n\n

\nBulk load all 100.000 products

\n\n

For our last benchmark we load all existing products via the GET /products route. Because this causes a lot of load we reduce the number of threads in our JMeter configuration from 10 to 2 and only use 50 iterations. But enough talk here are the numbers.

Metric	Impure	Pure
AVG	19061	14496
MED	19007	14468
90%	19524	14689
95%	19875	14775
99%	20360	14992
MIN	17848	14008
MAX	21315	16115
ERR	0%	0%
R/M	6.30	8.30
MEM	7889	1190
LD	5	4

\n\n\n\n

As you can see the difference in system load is way smaller this time. While it is still 25% a load of 4 versus 5 on a machine like the test machine makes almost no difference. However the pure service is again faster (about 25%). Looking at the memory footprint we can see that the impure one uses nearly seven times as much memory as the pure one.
But before we burst into cheers about that let’s remember what we did in the impure implementation! Yes, we used the groupBy operator of Akka which keeps a lot of stuff in memory so the fault for this is ours. ;-)
Because I’m not in the mood to mess with Akka until we’re on the memory safe side here, we’ll just ignore the memory footprint for this benchmark. Summarising that the pure service is again faster than the impure one.

\n\n

\nSummary

\n\n\n\n

If you look around in the internet (and also in the literature) you’ll find a lot of sources stating that “functional programming is slow” or “functional programming does not perform” and so on. Well I would argue that we have proven that this is not the case! Although we cannot generalise our findings because we only took a look at a specific niche within the corner of a specific environment, I think this is pretty exciting!

\n\n

Not only do you benefit from having code that can more easily reasoned about but you gain better testing possibilities and in the end your application also performs better! :-)

\n\n

While we pay some price (increased memory footprint) for it because there is no free lunch. It seems that it is worth it to work in a clean and pure fashion. So next time someone argues in favour of some dirty impure monstrosity because it is faster, just remember and tell ‘em that this might not be true!

\n\n

\nDocumenting your API

\n\n

Before we celebrate ourselves we have to tackle one missing point: We have to document our API.

\n\n\n\n

No, it is not! Leaving the issues of proper documented code aside here, we will concentrate on documenting the API. The de facto standard in our days seems to be using Swagger¹ for this. To keep things simple we will stick to it. Besides that it won’t hurt to have some documentation in text form (a small file could be enough) which should explain the quirks of our API. The bigger the project the earlier you may encounter flaws in the logic which might not be changeable because whatever reasons there are. ;-)

\n\n

\nThe lay of the land

\n\n

Swagger provides a bit of tooling and there are likely lots of projects trying to bring it to your favourite web framework or tool kit. In many cases it might be a good idea to bundle the Swagger UI² with your service and make it available on a specific path. Depending on your needs and environment you’ll want to protect that path via authentication and make it configurable for turning it off in production.

\n\n

Having the UI you must decide which path you want to go down:

\n\n

Write a swagger.yml file which describes your API, create JSON from it and deliver that as an asset.
Create the JSON dynamically via some library at runtime.

\n\n

The first point has the benefit that you description will be more or less set in stone and you avoid possible performance impacts and other quirks at runtime. However you must think of a way to test that your service actually fulfils the description (read specification) that you deliver. The most common tool for writing your API description is probably the Swagger Editor³.

\n\n

Taking the second path will result in a description which reflects your actual code. However none of the tools available I’ve seen so far has fulfilled that promise to 100 percent. You’ll very likely have to make extensive use of annotations to make your API description usable, resulting also in a decoupling of code and description. Also you might encounter “funny” deduced data types like Future1 and so on - which are annoying and confusing for the user. For those favouring the impure approach there is the swagger-akka-http library⁴.

\n\n\n\n

The answer is yes! We can describe our API using static typing and have the compiler check it and can deduce server and client code from it.

\n\n\n\n

\nUsing types to describe an API

\n\n

Describing your API using types is not bleeding edge academic research stuff like you might have guessed. There are several libraries existing for it! :-)

\n\n

Personally I stumbled upon such things first some years ago when seeing the talk “Using object algebras to design embedded DSLs” (Curry On 2016)⁵. The related project is the library endpoints⁶. However there are other projects too including the rho library⁷ included in the http4s project. Another one is tapir⁸ which we will be using in our example.
In the Haskell camp there is the beautiful Servant library⁹

\n\n\n\n

First I wanted to use something which allows us to generate a http4s server. This already narrowed down the options a bit. Also it should be able to generate API documentation (which nowadays means Swagger/OpenAPI support). Furthermore it should support not only http4s but more options. So after playing a bit around I decided to use the tapir library.

\n\n\n\n

\nA pure implementation using tapir.

\n\n

We basically cloned our pure folder into the tapir folder and start to apply our changes to the already pure implementation. But first some theory.

\n\n

\nBasics

\n\n

The tapir library assumes that you describe your API using the Endpoint type which is more concrete defined as follows: Endpoint[I, E, O, S]

\n\n

The type I defines the input given into the endpoint.
Type type E defines the error (or errors) which may be returned by the endpoint.
The type O defines the possible output of the endpoint.
The type S specifies the type of streams which are used for in- and output.

\n\n

An endpoint can have attributes like name or description which will be used in the generated documentation. You can also map input and output parameters into case classes.

\n\n\n\n

Regarding the encoding and decoding of data we will need our Circe codecs but additionally some schema definitions required by tapir. In concrete we will need to define implicit SchemaFor[T] instances for each of our models. We will start with the Translation model.

\n\n\n

Tapir schema for Translation

1 implicit val schemaFor: SchemaFor[Translation] = SchemaFor(\n2   Schema.SProduct(\n3     Schema.SObjectInfo(\"Translation\"),\n4     List((\"lang\", Schema.SString), (\"name\", Schema.SString)),\n5     List(\"lang\", \"name\")\n6   )\n7 )\n

\n\n\n\n\n

As you can see this is quite straightforward and not very complicated. In the future we might be able to derive such things but for now we have to define them. The schema is defined as a product type which is further described by the “object info” type containing a name, the field names and their schemas and at least a list of field names which are required to construct the type.

\n\n\n

Tapir schema for Product (1)

 1 implicit val schemaFor: SchemaFor[Product] = SchemaFor(\n 2   Schema.SProduct(\n 3     Schema.SObjectInfo(\"Product\"),\n 4     List(\n 5       (\"id\", Schema.SString),\n 6       (\"names\", Schema.SArray(Translation.schemaFor.schema))\n 7     ),\n 8     List(\"id\", \"names\")\n 9   )\n10 )\n

\n\n\n\n\n

This is basically the same thing as before except that we rely on the existing schema definition of Translation and use that here. You might notice that we explicitly define our NonEmptySet as an SArray here. This is because we want a list like representation on the JSON side.

\n\n\n\n

Yes, we can! :-) To gain more flexibility we can provide a generic schema for our NonEmptySet type.

\n\n\n

Tapir schema for NonEmptySet from Cats

1 implicit def schemaForNeS[T](implicit a: SchemaFor[T]): \n2   SchemaFor[NonEmptySet[T]] =\n3     SchemaFor(Schema.SArray(a.schema))\n

\n\n\n\n\n

Here we define that a NonEmptySet will be an array using the schema of whatever type it contains. Now we rewrite our previous schema definition as follows.

\n\n\n

Tapir schema for Product (2)

 1 implicit val schemaFor: SchemaFor[Product] = SchemaFor(\n 2   Schema.SProduct(\n 3     Schema.SObjectInfo(\"Product\"),\n 4     List(\n 5       (\"id\", Schema.SString),\n 6       (\"names\", schemaForNeS[Translation].schema)\n 7     ),\n 8     List(\"id\", \"names\")\n 9   )\n10 )\n

\n\n\n\n\n

It is not necessarily shorter but we have gained some more flexibility and can reuse our schema for the non empty set in several places.

\n\n

\nProduct routes

\n\n

Having the basics settled we can try to write our first endpoint. Let’s refactor our product routes. We will define our endpoints in the companion object of the class.

\n\n\n

Tapir endpoint for loading a product

1 // Our type is Endpoint[ProductId, StatusCode, Product, Nothing]\n2 val getProduct = endpoint.get\n3   .in(\"product\" / path[ProductId](\"id\"))\n4   .errorOut(statusCode)\n5   .out(jsonBody[Product])\n

\n\n\n\n\n

So what do we have here? First we specify the HTTP method by using the get function of the endpoint. Now we need to define our path and inputs. We do this using the in helper which accepts path fragments separated by slashes and also a path[T] helper which allows us to extract a type directly from a path fragment. This way we define our entry point product/id in which id must match our ProductId type.
To be able to be more flexible about our returned status codes we use the errorOut function which in our case just receives a status code (indicated by passing statusCode to it).
Finally we define that the endpoint will return the JSON representation of a product by using the out and jsonBody helpers. This all is reflected in the actual type signature of our endpoint which reads Endpoint[ProductId, StatusCode, Product, Nothing]. If we remember the basics then we know that this amounts to an endpoint which takes a ProductId as input, produces a StatusCode as possible error and returns a Product upon success.

\n\n\n\n

Our endpoint alone won’t do us any good so we need an actual server side implementation of it. While we could have used the serverLogic function to directly attach our logic onto the endpoint definition this would have nailed us down to a concrete server implementation.
So we’re going to implement it in the ProductRoutes class.

\n\n\n

http4s implementation of the load product endpoint

1 val getRoute: HttpRoutes[F] = ProductRoutes.getProduct.toRoutes { id =>\n2   for {\n3     rows <- repo.loadProduct(id)\n4     resp = Product\n5       .fromDatabase(rows)\n6       .fold(StatusCodes.NotFound.asLeft[Product])(_.asRight[StatusCode])\n7   } yield resp\n8 }\n

\n\n\n\n\n

We use the toRoutes helper of tapir which expects a function with the actual logic. As you can see the implementation is straightforward and only differs slightly from our original one. Currently there is no other way to handle our “not found” case than using the fold at the end. But if you remember, we did the same thing in the original code.

\n\n

That was not that difficult, for something which some people like to talk about as “academic fantasies from fairy tale land”. ;-)

\n\n

Onward to our next route: updating an existing product. First we need to define our endpoint.

\n\n\n

Tapir endpoint for updating a product

 1 // Our type is Endpoint[(ProductId, Product), StatusCode, Unit, Nothing]\n 2 val updateProduct =\n 3   endpoint.put\n 4     .in(\"product\" / path[ProductId](\"id\"))\n 5     .in(\n 6       jsonBody[Product]\n 7         .description(\"The updated product data which should be saved.\")\n 8     )\n 9     .errorOut(statusCode)\n10     .out(statusCode(StatusCodes.NoContent))\n

\n\n\n\n\n

This is only slightly more code than our first endpoint. We use the put method this time and the same logic as before to extract our product id from the path. But we also need our product input which we expect as JSON in the request body. The jsonBody function used is also extended with the description helper here which will provide data for a possibly generated documentation. We’ll come to generated API docs later on.
We also restrict our errors to status codes via the errorOut(statusCode) directive. Last but not least we have to define our output. Per default a status code of “200 OK” will be used which is why we override it in the out function with the “204 No Content” preferred by us.

\n\n\n

http4s implementation of the update product endpoint

 1 private val updateRoute: HttpRoutes[F] = \n 2   ProductRoutes.updateProduct.toRoutes {\n 3     case (id, p) =>\n 4       for {\n 5         cnt <- repo.updateProduct(p)\n 6         res = cnt match {\n 7           case 0 => StatusCodes.NotFound.asLeft[Unit]\n 8           case _ => ().asRight[StatusCode]\n 9         }\n10       } yield res\n11   }\n

\n\n\n\n\n

The implementation is again very similar to our original one. Except that it is even a bit simpler. This is because we do not have to worry about wrongly encoded input (Remember our handleErrorWith directive?). The tapir library will by default return a “400 Bad Request” status if any provided input cannot be decoded.
Within the pattern match we use the status codes provided by tapir and map the returned values to the correct type which is an Either[StatusCode, Unit] because of our endpoint type. This results from our endpoint type signature being Endpoint[(ProductId, Product), StatusCode, Unit, Nothing]. This translates to having an input of both a ProductId and a Product and returning a StatusCode in the error case or Unit upon success.

\n\n

Now we only need to combine both routes and we’re set.

\n\n\n

combined product routes

1 @SuppressWarnings(Array(\"org.wartremover.warts.Any\"))\n2 val routes: HttpRoutes[F] = getRoute <+> updateRoute\n

\n\n\n\n\n

So, let us run our tests and see what happens.

\n\n\n

1 [info]   when PUT /product/ID\n2 [info]     when request body is invalid\n3 [info]     - must return 400 Bad Request *** FAILED ***\n4 [info]       TestFailedException was thrown during property evaluation.\n5 [info]         Message: Vector(...) was not empty\n6 [info]         Location: (ProductRoutesTest.scala:100)\n7 [info]         Occurred when passed generated values (\n8 [info]           id = b55df341-a165-40c4-87ba-3d1c5cfb2f0c\n9 [info]         )\n

\n\n\n\n\n

Well, not what we expected, or is it? To be honest I personally expected more errors but maybe I’m just doing this stuff for too long. ;-)
If we look into the error we find that because the encoding problems are now handled for us the response not only contains a status code of “400 Bad Request” but also an error message: “Invalid value for: body”. Because I’m fine with that I just adjust the test and let it be good. :-)

\n\n

Pretty awesome, we have already half of our endpoints done. So let’s move on to the remaining ones and finally see how to generate documentation and also a client for our API.

\n\n

\nProducts routes

\n\n\n

Tapir endpoint for creating a product

1 val createProduct: Endpoint[Product, StatusCode, Unit, Nothing] =\n2   endpoint.post\n3     .in(\"products\")\n4     .in(\n5       jsonBody[Product]\n6         .description(\"The product data which should be created.\")\n7     )\n8     .errorOut(statusCode)\n9     .out(statusCode(StatusCodes.NoContent))\n

\n\n\n\n\n

As we can see the endpoint definition for creating a product does not differ from the one that was used to update one. Except that we have a different path here and do not need to extract our ProductId from the URL path.

\n\n\n

http4s implementation of the create product endpoint

 1 val createRoute: HttpRoutes[F] =\n 2   ProductsRoutes.createProduct.toRoutes { product =>\n 3     for {\n 4       cnt <- repo.saveProduct(product)\n 5       res = cnt match {\n 6         case 0 => StatusCodes.InternalServerError.asLeft[Unit]\n 7         case _ => ().asRight[StatusCode]\n 8       }\n 9     } yield res\n10   }\n

\n\n\n\n\n

The implementation is again pretty simple. In case that the saveProduct function returns a zero we output a “500 Internal Server Error” because the product has not been saved into the database.

\n\n

Finally we have our streaming endpoint left, so let’s see how we can do this via tapir.

\n\n\n

Tapir endpoint for loading all products

1 // Our type is Endpoint[Unit, StatusCode, Stream[F, Byte], Stream[F, Byte]]\n2 def getProducts[F[_]] =\n3   endpoint.get\n4     .in(\"products\")\n5     .errorOut(statusCode)\n6     .out(\n7       streamBody[Stream[F, Byte]](schemaFor[Byte], tapir.MediaType.Json())\n8     )\n

\n\n\n\n\n

The first thing we can see is that we use a def instead of a val this time. This is caused by some necessities on the Scala side. If we want to abstract over a type parameter then we need to use a def here.
We also have set the last type parameter not to Nothing but to something concrete this time. This is because we actually want to stream something. ;-)
It is a bit annoying that we have to define it two times (once for the output type and once for the “stream” type). Much nicer would be something like Endpoint[I, E, Byte, Stream[F, _]] but currently this is not the way we can do it.
So we again specify the HTTP method (via get) and the path (which is “products”). The errorOut helper once again restricts our error output to the status code. Finally we set the output of the endpoint by declaring a streaming entity (via streamBody).

\n\n\n\n

But is is sufficient and we also directly specify the returned media type to be JSON.

\n\n\n

http4s implementation of the load all products endpoint (1)

 1 @SuppressWarnings(Array(\"org.wartremover.warts.Any\"))\n 2 val getRoute: HttpRoutes[F] = ProductsRoutes.getProducts.toRoutes {\n 3   val prefix = Stream.eval(\"[\".pure[F])\n 4   val suffix = Stream.eval(\"]\".pure[F])\n 5   val ps = repo.loadProducts\n 6     .groupAdjacentBy(_._1)\n 7     .map {\n 8       case (id, rows) => Product.fromDatabase(rows.toList)\n 9     }\n10     .collect {\n11       case Some(p) => p\n12     }\n13     .map(_.asJson.noSpaces)\n14     .intersperse(\",\")\n15   val result: Stream[F, String] = prefix ++ ps ++ suffix\n16   val bytes: Stream[F, Byte]    = result.through(fs2.text.utf8Encode)\n17   bytes\n18 }\n

\n\n\n\n\n

Again our implementation is quite the same compared to the original one. Except that in the end we convert our stream of String into a stream of Byte using the utf8Encode helper from the fs2 library.

\n\n\n\n\n

1   found   : fs2.Stream[F,Byte]\n2   required: Unit => F[Either[tapir.model.StatusCode,fs2.Stream[F,Byte]]]\n3      (which expands to)  Unit => F[Either[Int,fs2.Stream[F,Byte]]]\n4      bytes\n5      ^\n

\n\n\n\n\n

Damn, so close. But let’s keep calm and think. Or ask around on the internet. Which is totally fine. Actually it is all in the compiler error message.

\n\n\n\n\n

http4s implementation of the load all products endpoint (2)

1 @SuppressWarnings(Array(\"org.wartremover.warts.Any\"))\n2 val getRoute: HttpRoutes[F] = ProductsRoutes.getProducts.toRoutes {\n3   // ...\n4   val result: Stream[F, String] = prefix ++ ps ++ suffix\n5   val bytes: Stream[F, Byte]    = result.through(fs2.text.utf8Encode)\n6   val response: Either[StatusCode, Stream[F, Byte]] = Right(bytes)\n7   (_: Unit) => response.pure[F]\n8 }\n

\n\n\n\n\n

We first convert our response explicitly into the right side of an Either because the left side is used for the error case. Afterwards we provide the needed Unit => ... function in which we lift our response value via pure into the context of F.
So let’s go crazy and simply combine our routes like in the previous part and run the test via testOnly *.ProductsRoutesTest on the sbt console.

\n\n\n

1 [info] All tests passed.\n

\n\n\n\n\n

Yes! Very nice, it seems like we are done with implementing our routes via tapir endpoints.

\n\n

\nDocumentation via OpenAPI

\n\n

So, we can now look at documenting our API via OpenAPI using the tooling provided by tapir. But first we should actually modify our main application entry point to provide the documentation for us.

\n\n\n

Provide Swagger UI and documentation

 1 // ...\n 2   productRoutes  = new ProductRoutes(repo)\n 3   productsRoutes = new ProductsRoutes(repo)\n 4   docs = List(\n 5     ProductRoutes.getProduct,\n 6     ProductRoutes.updateProduct,\n 7     ProductsRoutes.getProducts,\n 8     ProductsRoutes.createProduct\n 9   ).toOpenAPI(\"Pure Tapir API\", \"1.0.0\")\n10   docsRoutes = new SwaggerHttp4s(docs.toYaml)\n11   routes     = productRoutes.routes <+> productsRoutes.routes\n12   httpApp    = Router(\n13     \"/\" -> routes,\n14     \"/docs\" -> docsRoutes.routes\n15   ).orNotFound\n16 // ...\n

\n\n\n\n\n

We use the toOpenAPI helper provided by tapir which generates a class structure describing our API from a list of given endpoints. Additionally we use the SwaggerHttp4s helper which includes the Swagger UI for simple documentation browsing. All of it is made available under the /docs path. So calling http://localhost:57344/docs with your browser should open the UI and the correct documentation.
But while browsing there we can see that it provides our models and endpoints but documentation could be better. So what can we do about it?
The answer is simple: Use the helpers provided by tapir to add additional information to our endpoints.

\n\n

\nProviding example data

\n\n

Besides functions like description or name tapir also provides example which will result in having concrete examples in the documentation. To use this we must construct example values of the needed type. A Product example could look like this.

\n\n\n

Example Product type for the API documentation

 1 val example = Product(\n 2   id = java.util.UUID.randomUUID,\n 3   names = NonEmptySet.one(\n 4       Translation(\n 5         lang = \"de\",\n 6         name = \"Das ist ein Name.\"\n 7       )\n 8     ) ++\n 9     NonEmptySet.one(\n10       Translation(\n11         lang = \"en\",\n12         name = \"That's a name.\"\n13       )\n14     ) ++\n15     NonEmptySet.one(\n16       Translation(\n17         lang = \"es\",\n18         name = \"Ese es un nombre.\"\n19       )\n20     )\n21 )\n

\n\n\n\n\n

We can now use it in our product endpoint description.

\n\n\n

Documented tapir endpoints (1)

 1 val getProduct = endpoint.get\n 2   .in(\n 3     \"product\" / path[ProductId](\"id\")\n 4       .description(\"The ID of a product which is a UUID.\")\n 5       .example(example.id)\n 6   )\n 7   .errorOut(statusCode)\n 8   .out(\n 9     jsonBody[Product]\n10     .description(\"The product associated with the given ID.\")\n11     .example(example)\n12   )\n13   .description(\n14     \"Returns the product specified by the ID given in the URL path.\n15     If the product does not exist then a HTTP 404 error is returned.\"\n16   )\n

\n\n\n\n\n

As you can see we make use of description and example here. Also the path parameter id is described that way.

\n\n\n

Documented tapir endpoints (2)

 1 val updateProduct =\n 2   endpoint.put\n 3     .in(\n 4       \"product\" / path[ProductId](\"id\")\n 5         .description(\"The ID of a product which is a UUID.\")\n 6         .example(example.id)\n 7     )\n 8     .in(\n 9       jsonBody[Product]\n10         .description(\"The updated product data which should be saved.\")\n11         .example(example)\n12     )\n13     .errorOut(statusCode)\n14     .out(\n15       statusCode(StatusCodes.NoContent)\n16         .description(\"Upon successful product update no content is returned.\")\n17     )\n18     .description(\n19       \"Updates the product specified by the ID given in the URL path.\n20       The product data has to be passed encoded as JSON in the request body.\n21       If the product does not exist then a HTTP 404 error is returned.\"\n22     )\n

\n\n\n\n\n

Here we also add a description to the simple status code output explaining explicitly that no content will be returned upon success. While the 204 status code should be enough to say this you can never be sure enough. ;-)
We’ll skip the create endpoint because it looks nearly the same as the update endpoint. Instead let’s take a look at our streaming endpoint.

\n\n\n

Documented tapir endpoints (3)

 1 def getProducts[F[_]] =\n 2   endpoint.get\n 3     .in(\"products\")\n 4     .errorOut(statusCode)\n 5     .out(\n 6       streamBody[Stream[F, Byte]](schemaFor[Byte], tapir.MediaType.Json())\n 7         .example(examples.toList.asJson.spaces2)\n 8     )\n 9     .description(\n10       \"Return all existing products in JSON format as a stream of bytes.\"\n11     )\n

\n\n\n\n\n

This time we need to provide our example as a string because of the nature (read type) of our endpoint. We use the non empty list of examples that we created (you can look it up in ProductsRoutes.scala) and convert it into a JSON string.
If we now visit our swagger endpoint we’ll see nice examples included in the documentation. Pretty cool, especially because many people will look at the examples not at the specification. This might be because (too) many of us have seen an API not fulfilling its specification. This shouldn’t happen in our case because we’re deriving it, yeah! But nonetheless examples are very nice to have. :-)

\n\n\n\n

If we take a closer look at our model descriptions then we might see that we could do better in some cases. I’m thinking of our ID fields being simple strings instead of UUIDs and the language code which is also defined as a simple string. So let’s get going and clean that up!

\n\n

\nRefining the generated documentation

\n\n

Looking at the intermediate model for our API documentation (see the OpenAPI class structure in the tapir library) we realise that modifying such a deeply nested case class structure might result in some really messy code. I mean we have probably all been there at some point in our life as developer. ;-)

\n\n\n\n

Yes, we can! Confronted with big and nested structures we should pick a tool from our functional programming toolbox which is called optics¹⁰.
Don’t be scared by the name or all that mathematics, there exist some usable libraries for it. In our case we will pick Monocle¹¹ which provides profunctor optics for Scala. The basic idea of optics is to provide pure functional abstractions for the manipulation of immutable objects. Because of their pure nature they are composable which results in code which is more flexible and can more easily be reasoned about.

\n\n

Now that we have that cleared let’s make a plan what we actually want to do.

\n\n

Adjust the URL parameter descriptions of {id} to mark them as kind of UUID.
Adjust the id attribute of our Product model to mark it as kind of UUID.
Adjust the lang attribute of our Translation model to mark it as an ISO-639-1 language code.

\n\n\n\n

If we take a look at the code within the tapir library, we see that the Schema and SchemaFor code which is used for codecs does not yet support a dedicated UUID type. There is a detour for it using Schema.SString.

\n\n\n\n

Now we look a bit further into the OpenAPI code and find that it supports several interesting attributes which we might use. For now we will stick to the attribute pattern of the Schema class in that part. It is intended to hold a pattern (read regular expression) which describes the format of a string type.
Okay, so we need to define some (or better exactly 2) regular expressions. But hey wait, we already have one for our language code! :-)

\n\n\n\n

Well, using some shapeless¹² magic we might use an implicit Witness which should be provided by our refined type.

\n\n\n

Idea to reuse regular expression from refined type

1 def extractRegEx[S <: String](implicit ws: Witness.Aux[S]): String =\n2   ws.value\n

\n\n\n\n\n

The code above is a rough idea so don’t count on it. We’ll see later on if I was right or did suffer from the hallucination of actually understanding what I am doing. ;-)
In the hope that this is settled we need one additional regular expression for a UUID. These are defined in RFC-4122¹³ and ignoring the special edge case of a “NIL UUID” we come up with the following solution.

\n\n\n

Regular expression for matching UUIDs

1 ^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[089ab][0-9a-f]{3}-[0-9a-f]{12}$\n

\n\n\n\n\n\n\n

According to the OpenAPI specification we are nailed down to Javascript regular expressions (read the ECMA 262 regular expression dialect¹⁴). Oh why cruel fate? Well, let us deal with that when we have our other puzzle pieces in order.

\n\n

Before we dive right in let’s play around a bit to get used to this fancy optics thing.

\n\n\n

1 val docs: OpenAPI = ???\n2 val paths: Lens[OpenAPI, ListMap[String, PathItem]] =\n3   GenLens[OpenAPI](_.paths)\n4 val test = (paths composeLens at(\"/product/{id}\")).get(d)\n

\n\n\n\n\n

We defined our first Lens via the GenLens macro. It is supposed to give us the path definitions from an OpenAPI object. In the last part we use the compose functionality to query a specific item from the paths which we address via a string because it is a ListMap using string keys.

\n\n\n

1 [error] ...scala: ambiguous implicit values:\n2 [error]  both method atMap in object At of type \n3            [K, V]=> monocle.function.At[Map[K,V],K,Option[V]]\n4 [error]  and method atSet in object At of type \n5            [A]=> monocle.function.At[Set[A],A,Boolean]\n6 [error]  match expected type monocle.function.At[S,String,A]\n7 [error]     val x  = (paths composeLens at(\"/product/{id}\")).get(d)\n8 [error]                                   ^\n9 [error] one error found\n

\n\n\n\n\n

Oh no, the compiler yells at us! Some investigation leads to the conclusion that there is a type class instance missing for ListMap. So we build one. Luckily for us this is practically identical to the one for Map.

\n\n\n

At type class instance for ListMap

1 implicit def atListMap[K, V]: At[ListMap[K, V], K, Option[V]] = At(\n2   i => Lens((_: ListMap[K, V]).get(i))(optV => map => \n3     optV.fold(map - i)(v => map + (i -> v))\n4   )\n5 )\n

\n\n\n\n\n

And now our code compiles, hooray! :-)
While we’re at it we also provide an instance for Index on ListMap.

\n\n\n

Index type class instance for ListMap

1 implicit def listMapIndex[K, V]: Index[ListMap[K, V], K, V] = Index.fromAt\n

\n\n\n\n\n\n\n

Okay, back to work. Thinking a bit about our problem we realise that we need a couple of lenses which we can compose to modify the needed parts of the documentation structure.

\n\n\n

Generating lenses

 1 val paths: Lens[OpenAPI, ListMap[String, PathItem]] =\n 2   GenLens[OpenAPI](_.paths)\n 3 val getOps: Lens[PathItem, Option[Operation]] =\n 4   GenLens[PathItem](_.get)\n 5 val putOps: Lens[PathItem, Option[Operation]] =\n 6   GenLens[PathItem](_.put)\n 7 val operationParams: Lens[Operation, List[OpenAPI.ReferenceOr[Parameter]]] =\n 8   GenLens[Operation](_.parameters)\n 9 val pathParams: Lens[PathItem, List[OpenAPI.ReferenceOr[Parameter]]] =\n10   GenLens[PathItem](_.parameters)\n11 val parameterSchema: Lens[Parameter, OpenAPI.ReferenceOr[Schema]] =\n12   GenLens[Parameter](_.schema)\n13 val schemaPattern: Lens[Schema, Option[String]] =\n14   GenLens[Schema](_.pattern)\n

\n\n\n\n\n

Well this is quite a lot but let’s break it apart piece by piece. In general we use the GenLens macro of Monocle to create the lenses for us.
First we create a lens which returns the defined path items which is a ListMap. We later use the at function of the type class to grab a concrete entry from it. This “concrete” entry will be a PathItem that contains more information. Next are some lenses which will return an Operation from the aforementioned PathItem. Depending on the type of the operation (GET, POST, etc.) we return the appropriate entry.
Now we need to grab the parameters used for the endpoints which can either be collected directly from a PathItem or an Operation. These are both lists of the type Parameter. Okay, I’m lying straight to your face here. In fact they are ReferenceOr[Parameter] which means an Either[Reference, Parameter]. But in our use case we only have them as parameters so we’ll ignore this for now.
Last but not least we need to grab the Schema of a parameter and from that one the pattern field. The schema describing the parameter is also an Either[Reference, Schema] which we will ignore too in this case. Now we can play around with our lenses and various combinations.

\n\n\n

Some examples

 1 // Delete the entry at \"/product/{id}\"\n 2 (paths composeLens at(\"/product/{id}\")).set(None)(docs)\n 3 // Replace the path parameters with an empty list at \"/product/{id}\"\n 4 (paths composeLens at(\"/product/{id}\") composeOptional possible \n 5   composeLens pathParams)\n 6     .set(List.empty)(docs)\n 7 // Traverse through all schemas in all path parameters at \"/product/{id}\"\n 8 (paths composeLens at(\"/product/{id}\") composeOptional possible \n 9   composeLens pathParams composeTraversal each composeOptional \n10   possible composeLens parameterSchema)\n11     .getAll(docs)\n12 // Set the pattern field in all schemas in all path parameters\n13 // at \"/product/{id}\"\n14 (paths composeLens at(\"/product/{id}\") composeOptional possible \n15   composeLens pathParams composeTraversal each composeOptional \n16   possible composeLens parameterSchema composeOptional possible \n17   composeLens schemaPattern)\n18     .set(Option(\"Optics are soo cool!\"))(docs)\n

\n\n\n\n\n

This can be confusing at a first look but it is actually very powerful and clean for modifying deeply nested structures. The current API is a bit verbose but there is hope.

\n\n\n\n

So what we can take away from this is that we can update our pattern field within all parameters using the following code. Before you ask: We can update all parameters because we have only one. ;-)

\n\n\n

Update the pattern attribute in the API documentation

1 (paths composeLens at(\"/product/{id}\") composeOptional possible \n2   composeLens pathParams composeTraversal each composeOptional \n3   possible composeLens parameterSchema composeOptional possible \n4   composeLens schemaPattern)\n5     .set(Option(\"Fancy UUID regex here!\"))(docs)\n

\n\n\n\n\n

Seems we can (as good as) check off the first point on our list. Leaving us with modifying the pattern field of the actual model schemas. To get to these we have to define some more lenses.

\n\n\n

Additional lenses for content schemas

1 val components: Lens[OpenAPI, Option[Components]] =\n2   GenLens[OpenAPI](_.components)\n3 // type Lens[Components, ListMap[String, OpenAPI.ReferenceOr[Schema]]]\n4 val componentsSchemas =\n5   GenLens[Components](_.schemas)\n6 // type Lens[Schema, ListMap[String, OpenAPI.ReferenceOr[Schema]]]\n7 val schemaProperties =\n8   GenLens[Schema](_.properties)\n

\n\n\n\n\n

The power of lenses allows us to traverse all of our structures and modify all affected models at once. So we will use the functionality provided by the Each type class here. Let’s try something like the following code.

\n\n\n

Traverse each path item

1 (paths composeTraversal each composeLens getOps).getAll(docs)\n

\n\n\n\n\n

Looks good so far but it results in a compiler error.

\n\n\n

1 [error] ...: diverging implicit expansion for type cats.kernel.Order[A]\n2 [error] starting with method catsKernelStdOrderForSortedSet in trait \n3         LowPrioritySortedSetInstancesBinCompat1\n4 [error]       (paths composeTraversal each composeLens getOps).getAll(docs)\n5 [error]                               ^\n

\n\n\n\n\n

This looks like an error from a binary incompatible cats version. But hey, this time the compiler is lying to us. Maybe you have guessed it already: we’re missing another type class instance. This time the one for Each which we need to traverse our ListMap structure. So let’s write one! :-)

\n\n\n

Each type class instance for ListMap

 1 implicit def listMapTraversal[K, V]: Traversal[ListMap[K, V], V] = \n 2   new Traversal[ListMap[K, V], V] {\n 3     def modifyF[F[_]: Applicative](f: V => F[V])(s: ListMap[K, V]): \n 4       F[ListMap[K, V]] =\n 5         s.foldLeft(Applicative[F].pure(ListMap.empty[K, V])) {\n 6           case (acc, (k, v)) =>\n 7             Applicative[F].map2(f(v), acc)((head, tail) => \n 8               tail + (k -> head))\n 9       }\n10   }\n11 \n12 implicit def listMapEach[K, V]: Each[ListMap[K, V], V] =\n13   Each(listMapTraversal)\n

\n\n\n\n\n

As noted above these things will hopefully be in the next Monocle release together with a much nicer API. :-D
The information we need to modify is within the components field of the generated documentation. While we could traverse all the stuff (paths, operations) these only hold references which are of no use for us. So let’s update our Product model.

\n\n\n

Update Product model description via lenses

1 (components composeOptional possible composeLens componentsSchemas \n2  composeLens at(\"Product\") composeOptional possible composeOptional \n3  possible composeLens schemaProperties composeLens at(\"id\") \n4  composeOptional possible composeOptional possible composeLens \n5  schemaPattern)\n6   .set(Option(\"Our UUID regex here!\"))(docs)\n

\n\n\n\n\n

This is quite a lot but we actually only instruct our optics how to traverse down the structure. In general we need to take care of our possible field types here. Due to the nature of the generated structure having a lot of Option and Either fields we need way more boilerplate here. But as mentioned it is really not that complicated. We compose our lenses via composeLens but may need things like composeOptional possible to compose on a defined Option[T] which we have sometimes to duplicate when having nested occurrences of these. The same instructions can be used to zoom in on the right side of an Either.

\n\n\n

Update Translation model description via lenses

1 (components composeOptional possible composeLens componentsSchemas \n2  composeLens at(\"Translation\") composeOptional possible composeOptional \n3  possible composeLens schemaProperties composeLens at(\"lang\") \n4  composeOptional possible composeOptional possible composeLens \n5  schemaPattern)\n6   .set(Option(\"Our language code regex here!\"))(docs)\n

\n\n\n\n\n

The modification for the Translation model looks quite the same so we could possibly make a function out of it which would get some parameters and return the updated structure.

\n\n\n\n

Oh yeah, right we thought about extracting the regular expression directly from the refined type to avoid code duplication. So let’s try to use the function defined earlier on like this val langRegex = extractRegEx[LanguageCode].

\n\n\n

1 [error] : type arguments [com.wegtam.books.pfhais.tapir.models.LanguageCode] \n2   do not conform to method extractRegEx's type parameter bounds [S <: String]\n3 [error]     val langRegex = extractRegEx[LanguageCode]\n4 [error]                                 ^\n

\n\n\n\n\n

Okay, seems like I really don’t understand what I’m doing. ;-) Luckily for us the Scala community has nice, smart and helpful people in it. The solution is to write a type class which will support us extracting the desired parameter.

\n\n\n

Type class to extract the regular expression

 1 import eu.timepit.refined.api._\n 2 import eu.timepit.refined.string._\n 3 import shapeless.Witness\n 4 \n 5 trait RefinedExtract[T] {\n 6   def regex: String\n 7 }\n 8 \n 9 object RefinedExtract {\n10   implicit def instance[T, S <: String](\n11       implicit ev: String Refined MatchesRegex[S] =:= T,\n12       ws: Witness.Aux[S]\n13   ): RefinedExtract[T] = new RefinedExtract[T] { val regex = ws.value }\n14 }\n

\n\n\n\n\n

This allows us to have the desired effect in just this small piece of code.

\n\n\n

Extract and transform regular expression from LanguageCode

1 val typeRegex = implicitly[RefinedExtract[LanguageCode]].regex\n2 // convert to Javascript regular expression\n3 val langRegex = \"/\" + typeRegex + \"/\"\n

\n\n\n\n\n

Now if we take a look at our API documentation it looks better. Although we can see that the URL parameter pattern information is not used. But as mentioned before there is a lot of work going on at the tapir site currently and this will also get fixed then.

\n\n\n\n

However don’t forget about optics because their usage goes far beyond what we have done here.

\n\n

\nMoving to Scala 3…

\n\n

When I wrote this book version 2.12 was the thing and 2.13 was not yet released. While I did enable cross-building for 2.13 after it was released the next big version was still far away. But then it happened and Scala 3 was released. However it took some time until libraries and frameworks moved to support it. As I am writing this some still have not added support for it because in some areas this involves a significant amount of work.

\n\n

However, I always wanted to do an update to Scala 3 and while I might still find time to write a complete book about that, I chose not to wait longer although some libraries I would like to use are still not ported. Therefore the topic of this chapter will be to update our small service (the tapir version) to Scala 3 while dropping some not yet ported libraries.

\n\n

But instead of jumping right into the middle of it we might be better of to look at our options and plan our migration accordingly.

\n\n

The first step should be to switch to Scala 2.13 as major version and update all dependencies to their latest versions. This alone will be quite some work but it will ease the migration to Scala 3 for which we will try to use some tools which are available.

\n\n

Since version 1.5 the sbt build tool supports Scala 3 directly so there is no more need to add the sbt-dotty plugin to your build. Additionally it supports a new syntax for dependencies which will allow us to use 2.13 libraries in 3 and vice versa.

\n\n\n

Using Scala 2.13 libraries in Scala 3 projects

1 libraryDependency +=\n2   (\"my.domain\" %% \"my-lib\" % \"x.y.z\").cross(CrossVersion.for3Use2_13)\n

\n\n\n\n\n

The example above instructs sbt to use a Scala 2.13 library for Scala 3. If you want to do the opposite then you have to use CrossVersion.for2_13Use3 instead which will make sbt use a Scala 3 library for Scala 2.13.

\n\n

Furthermore there is the Scala-3-Migrate plugin for sbt which supports on a variety of topics when migration a project to Scala 3.

\n\n

So the second step would be to use the Scala-3-Migrate plugin to guide our migration to Scala 3. During this phase we will see what can be kept, what can be used with some restrictions and what has to be dropped.

\n\n

\nStep 1: Updating to 2.13.x

\n\n

The currently recommended version to start a migration from is 2.13.7 so we will target this Scala version for updating our project. In the source code you can see that I simply copied the tapir folder of our project and named it tapir-scala-3 to not mess with our existing code.

\n\n

First steps include updating sbt to a recent version as well as updating the sbt-plugins that we are using to their latest versions. Also some changes are made in regard to the compiler plugins. The kind-projector plugin needs a different way to be specififed (see cross CrossVersion.full in build.sbt) and the monadic-for plugin stays for now but will have to be removed when we’re on Scala 3. And while at it the migration plugin to support us is added as well:

\n\n\n

Add the scala3-migrate plugin to sbt

1 addSbtPlugin(\"ch.epfl.scala\" % \"sbt-scala3-migrate\" % \"0.5.0\")\n

\n\n\n\n\n

Now we switch the default version for Scala to 2.13.7 and try to compile the project. We run into some missing dependencies errors which will force our hand into upgrading several dependencies. In addition we stumble upon the matter that the compiler flag -Xlint:nullary-override has been dropped so we remove it or comment it out.

\n\n\n\n

Furthermore to reduce the clutter in our build file we remove support for Scala 2.12 and the related compiler options. In the case that you have to support older versions of Scala (cross compilation) things get more complicated. In our case we can move completely to Scala 3. :-)

\n\n

\nDetails and some compiling issues

\n\n

So what was done until now?

\n\n

include kind-projector plugin via CrossVersion.full
switch to Scala 2.13.7 aus default version
remove Scala 2.12 and related settings
update doobie to 0.8.8
update http4s to 0.21.31
update tapir to 0.11.11
update circe to 0.14.1
remove dropped compiler flags (for 2.13!)
disable Xfatal-warnings\n

\n\n

So far compiling our main code resulted in nagging us to fix several issues with auto-application of missing brackets for function calls, the main culprit being unsafeRunSync here which has to be unsafeRunSync(). Also some unused variable issues were popping up and are fixed easily too.

\n\n

Now onwards to compiling the tests and we have some more issues. So far the integration tests compile fine but the unit tests spill out an error:

\n\n\n

Compiler errors in the tests

 1 [error] .../TestRepository.scala:26:51: trait Seq takes type parameters\n 2 [error]         val ns = p.names.toNonEmptyList.toList.to[Seq]\n 3 [error]                                                   ^\n 4 [error] .../TestRepository.scala:26:50: missing argument list for method to \n 5           in trait IterableOnceOps\n 6 [error] Unapplied methods are only converted to functions when a function type\n 7           is expected.\n 8 [error] You can make this conversion explicit by writing `to _` or `to(_)`\n 9           instead of `to`.\n10 [error]         val ns = p.names.toNonEmptyList.toList.to[Seq]\n11 [error]                                                  ^\n12 [error] .../TestRepository.scala:32:49: trait Seq takes type parameters\n13 [error]       val ns = p.names.toNonEmptyList.toList.to[Seq]\n14 [error]                                                 ^\n15 [error] .../TestRepository.scala:32:48: missing argument list for method to\n16           in trait IterableOnceOps\n17 [error] Unapplied methods are only converted to functions when a function type\n18           is expected.\n19 [error] You can make this conversion explicit by writing `to _` or `to(_)`\n20           instead of `to`.\n21 [error]       val ns = p.names.toNonEmptyList.toList.to[Seq]\n22 [error]                                                ^\n23 [error] four errors found\n

\n\n\n\n\n

This looks big at first but let us stay calm an read the error messages. So the “trait Seq takes takes type parameters” eh? The second one says something about “unapplied methods” but isn’t exactly helpful either.

\n\n\n\n

Well, we fire up a REPL of course and as we are (or should be) in sbt we can simply use the console command. The sbt console will only work if we fixed all our compilation errors in the main code.

\n\n\n

Check for possible function for converting List

1 scala> List(1, 2, 3).to<TAB>\n2 // This should show a list of possible functions.\n

\n\n\n\n\n

So it seems we are missing our plain old (.to[T]). While there is a .to() function it requires a collection factory. So what about .toSeq? We did not use it in the past because it converted into a mutable sequence. But what about now?

\n\n\n

Test List conversion in the REPL

 1 scala> val a: scala.collection.immutable.Seq[Int] = List(1, 2, 3).toSeq\n 2 val a: Seq[Int] = List(1, 2, 3)\n 3 scala> a.getClass\n 4 val res0: Class[_ <: Seq[Int]] = class scala.collection.immutable.$colon$colon\n 5 \n 6 scala> a.getClass.getCanonicalName\n 7 val res1: String = scala.collection.immutable.$colon$colon\n 8 \n 9 scala> a.getClass.getName\n10 val res2: String = scala.collection.immutable.$colon$colon\n

\n\n\n\n\n

Well, well this looks pretty good I’d say so let’s adjust the code. And quickly we get a big type error but the gist of it is:

\n\n\n

Compiler error about invariant types

1 [error] Note: List[...] <: Seq[...], but type F is invariant in type _.\n2 [error] You may wish to define _$$1 as +_$$1 instead. (SLS 4.5)\n3 [error]         ns.map(n => (p.id, n.lang, n.name)).pure[F]\n4 [error]                                                 ^\n5 [error] one error found\n

\n\n\n\n\n

Good news first: The original error is gone and we can even simplify the code around the second error source by removing the toSeq completely. But the remaining one is heavier. So let’s take a step back and take a deep breathe. If we take a look at our function signature we can see that it requires a Seq but what if we simply change it to List?

\n\n

So let us try it and see how far we get. First we have to change the type signature of the function loadProduct in Repository to have a List instead of a Seq in its return type. Afterwards the compiler will tell us exactly in which places we have to make changes. Furthermore we can also remove some imports (scala.collection.immutable.Seq) which are no longer needed.

\n\n

Okay, onwards to… Did you run the tests? ;-)

\n\n

While executing the tests we discover that some unit tests are failing and the integration tests look good. Additionally I get a warning that the Flyway library should be updated. But we save this for later. Let us take a look at our failing tests first. We can see that they error out because of an exception:

\n\n\n

The dreaded NoSuchMethodError after a dependency upgrade

1 java.lang.NoSuchMethodError: 'cats.data.Kleisli\n2   org.http4s.HttpRoutes$.apply(scala.Function1, cats.effect.Sync)'\n

\n\n\n\n\n

This does not look good and it seems to be not caught by the compiler. We should start our service to see if it happens there too. So after a sbt run we can see that it affects our main code also.

\n\n

Nice we just broke our service. Welcome to the world of software development! :-)

\n\n\n\n

Most likely we are in trouble due to updating dependencies and running into some binary incompatibility issues. The error message indicates that it might either be cats or http4s related. To get some more insights we should issue the sbt evicted command and take a look at the output. We find some messages about replaced versions.

\n\n\n

Output snippets from the sbt evicted command

1 * org.http4s:http4s-dsl_2.13:0.21.31 is selected over 0.21.0-M5\n2 ...\n3 * org.typelevel:cats-effect_2.13:2.5.1 is selected over {2.0.0, ...}\n4 ...\n5 * org.typelevel:cats-core_2.13:2.6.1 is selected over {2.0.0, ...}\n6 ...\n7 * org.http4s:http4s-blaze-server_2.13:0.21.31 is selected over 0.21.0-M5\n8 ...\n

\n\n\n\n\n

Now we need to perform some investigations regarding the libraries which means digging into changelog entries, release notes and bug reports which might support our idea of something was broken. The cats part of the equation looks fine but there were some changes in the http4s library which might be the cause for our problem here. As the older version (0.21.0-M5) is a pre-release this is something that is totally valid and should always be on our radar. The older version is a dependency of tapir so that means we have to upgrade tapir as well which means “More breaking changes, yeah!” ;-)

\n\n\n\n

But before we tackle this problem we might as well quickly update the Flyway library to get rid of the warning in our tests. Brave as we are we jump to the most recent release and also update the driver for PostgreSQL as well. But what is this?

\n\n\n

Type error after Flyway upgrade

1 [error] .../FlywayDatabaseMigrator.scala:35:21: type mismatch;\n2 [error]  found   : org.flywaydb.core.api.output.MigrateResult\n3 [error]  required: Int\n4 [error]       flyway.migrate()\n5 [error]                     ^\n6 [info] org.flywaydb.core.api.output.MigrateResult <: Int?\n7 [info] false\n8 [error] one error found\n

\n\n\n\n\n

You didn’t expect this to be easy, did you? ;-) But this doesn’t look like a big issue. The return type of the migrate function was changed upstream and the only decision we have to make is if we want to change our function return type accordingly and simply pass the information onwards or do we change the function a bit and still only return the number of applied migrations. I pick the lazy route this time and simply append .migrationsExecuted to the call to .migrate() and we’re done with it.

\n\n

Now onwards to our tapir update. Before we simply upgrade to the latest version we should give it some more thought. Tapir is still in a heavy development phase and might depend on pre-release versions of other libraries again. So we better look some things up. The file project/Versions.scala within the tapir source repository gives us our needed insights. If we do not want to upgrade http4s even higher then it seems we will have to pick a tapir 0.17.x release. Such a jump will likely include lots of breaking changes so another option would be to pick the lowest possible tapir release with a compatible http4s dependency.

\n\n\n\n

We can either upgrade to the highest tapir version with a still compatible http4s dependency. Or we try to do the “minimum viable upgrade” and pick the lowest possible tapir version with a compatible http4s dependency to reduce our changes to a minimum. Last but not least we have the option to upgrade to the latest tapir version and upgrade all other dependencies as well.

\n\n

The last option might be tempting but it will force us to upgrade not only http4s but other dependencies as well and we will likely head straight into “upgrade dependency hell” and may not even succeed.

\n\n

Of our other options we can pick either the version 0.17.20 or something from the 0.12.x line of the tapir releases. Please not that the artefact organization name for tapir has changed! If you simply change the version number you will get unresolved dependency errors.

\n\n

Upgrading software is not for the faint hearted so let’s be brave and try to update to 0.17.20. The line between bravery and stupidity is a bit hazy but we’ll see how we do. :-)

\n\n

The first thing we stumble upon is of course a ton of errors because the namespace for tapir changed. Because changing it is simple but tedious it screams for automation and therefore we’ll use a shell script¹.

\n\n\n

Shell script to fix import namespace errors

1 % for i in `find tapir-scala-3 -name \"*.scala\"`; do\n2 %   sed -i '' -e s/'import tapir'/'import sttp.tapir'/g $i\n3 % done\n

\n\n\n\n\n

This script is specific to the sed version used in the BSD operating systems! It is a simple loop being fed from a find command and uses sed to perform a search and replace operation directly in the file. The -i '' parameter ensures that no backup is saved (We are within in version control anyway.).

\n\n

Okay after fixing that and the change of StatusCode and StatusCodes from tapir to sttp we still get a log of errors which look quite intimidating. Deciding that bravery is all good and so we turn to plan B switching the tapir version to 0.12.28. ;-)

\n\n

We still get a bunch of errors now but they are less in number and seem mostly related to schema creation and derivation. Also not the most easy topic but as we get the same errors on 0.17.x plus a load more we might as well try to fix them. The first guess is that some code has been moved and indeed it seems that our Schema.SWhatever types are now under SchemaType.SWhatever so this should be fixed easily. Additionally we need to do small adjustments regarding changed signatures and use Schema(SchemaType.SWhatever) instead of Schema.SWhatever in some places.

\n\n\n

Total error count so far

1 ...\n2 [warn] two warnings found\n3 [error] 6 errors found\n

\n\n\n\n\n

Nice! We are down to a single digit number of errors. It looks like I didn’t fix the StatusCodes issue correctly so after some changes we are down to one error:

\n\n\n

Last remaining compiler error after the tapir upgrade

1 [error] .../ProductsRoutes.scala:117:54: not found: value tapir\n2 [error] streamBody[Stream[F, Byte]](schemaFor[Byte], tapir.MediaType.Json())\n3 [error]                                              ^\n4 [error] one error found\n

\n\n\n\n\n

After digging a bit through the tapir code we can see that we simply have to pass a CodecFormat.Json() here now. Hooray, it compiles! But before we become too confident, let us run some tests.

\n\n\n

All our tests passed

1 [info] All tests passed.\n

\n\n\n\n\n

This is good news and furthermore starting our service via sbt run looks good also. :-)

\n\n

Now we could move on to the next step or we might try updating some more dependencies. For starters we remove the wartremover plugin because it isn’t available for Scala 3 anyway. Besides the plugin we must remove the settings in the build.sbt and the annotations within the code (see the @SuppressWarnings annotations). As a bonus we get rid of some warnings about Any type inference which are false positives anyway. Next is the move to the Ember server for http4s from the Blaze one because Ember is the new default and recommended one. For this our main entry point in Tapir.scala has to be adjusted a bit.

\n\n

First we change from IOApp to IOApp.WithContext and implement the executionContextResource function. In addition we adjust our blocking thread pool to use 2 threads or half of the available processors.

\n\n\n

Adjustments in the main entry point

 1 object Tapir extends IOApp.WithContext {\n 2   val availableProcessors: Int =\n 3     Runtime.getRuntime().availableProcessors() / 2\n 4   val blockingCores: Int =\n 5     if (availableProcessors < 2) 2 else availableProcessors\n 6   val blockingPool: ExecutorService =\n 7     Executors.newFixedThreadPool(blockingCores)\n 8   val ec: ExecutionContext =\n 9     ExecutionContext.global\n10 \n11   override protected def executionContextResource: \n12     Resource[SyncIO, ExecutionContext] = Resource.eval(SyncIO(ec))\n13 \n14   def run(args: List[String]): IO[ExitCode] = {\n15     val blocker = Blocker.liftExecutorService(blockingPool)\n16     val migrator: DatabaseMigrator[IO] = new FlywayDatabaseMigrator\n17     // ...\n18       resource = EmberServerBuilder\n19         .default[IO]\n20         .withBlocker(blocker)\n21         .withHost(apiConfig.host)\n22         .withPort(apiConfig.port)\n23         .withHttpApp(httpApp)\n24         .build\n25       fiber = resource.use(_ => IO(StdIn.readLine())).as(ExitCode.Success)\n26     // ...\n

\n\n\n\n\n\n\n

The update of the refined library requires us to update pureconfig as well in one step but it just works after increasing the version numbers. The same can be said about logback, cats and kittens. For the latter we make some small adjustment to get rid of a deprecation warning.

\n\n

Some more changes are required for updating ScalaTest and ScalaCheck but they boil down to changing some imports and names (i.e. Matchers instead of MustMatchers) and the inclusion of the ScalaTestPlus library which acts as a bridge to ScalaCheck now.

\n\n\n\n

The things left to update look like they might be a bit more involving:

\n\n

Doobie (database layer)
http4s (might not be that difficult as we switched to Ember already)
Monocle (version 3.x brings huge improvements but will require many changes)
tapir (contains breaking changes and might introduce more dependency trouble)

\n\n

As mentioned before we shouldn’t simply dive in but check what is really needed. To gather the necessary information we move on to the next step.

\n\n

\nStep 2: Migrating to Scala 3

\n\n

We have prepared our battle ground and already included the sbt plugin so we can just issue the migrate-libs tapir command to get some output. This is quite a lot so let’s concentrate on the important parts. First there is some explanation on the top:

\n\n\n

Possible status flags for dependencies for Scala 3 migration

1 [info] X             : Cannot be updated to scala 3\n2 [info] Valid         : Already a valid version for Scala 3\n3 [info] To be updated : Need to be updated to the following version\n

\n\n\n\n\n

We should not see the X mark (usually red in the terminal) but here we are and I count two of them. So what do we have?

\n\n

The better-monadic-for plugin.
The pureconfig library.

\n\n

The first one is no problem because we can simply drop it and the underlying problem is supposed to be solved in Scala 3. But what about pureconfig? Well let’s worry later and process the output further. We have quite some Valid marks which is great! Several others have other notes on them so onward to take a closer look.

\n\n

The following dependencies are supposed to work with CrossVersion.for3Use2_13:

\n\n

Monocle
Refined

\n\n

Last but not least some dependencies need to be updated further to support Scala 3:

\n\n

Doobie
http4s
kittens
tapir

\n\n

So it looks like we won’t get away without doing major upgrades anyway. While at it we might as well add Monocle to our upgrade list because it looks like it will be quite some work either way.

\n\n

The attentive reader will have noted that the recommended dependency updates have pre-release version numbers and she’ll ask if we really should upgrade them or wait until proper releases have been published. And yes, she is right: Though shall not use pre-release software in production!

\n\n

For our example here however we do it for demonstrating the upgrade process. In production I would advise you to wait or maybe upgrade and test in a separate environment.

\n\n\n\n

So, tapir has dependencies on http4s and also cats-effect therefore it will surely influence http4s and also doobie which also uses cats-effect. So the first candidate should be kittens because it doesn’t affect the other dependencies. The next one will be Monocle because although maybe not necessary it also doesn’t mess up the other dependencies. While updating kittens is done by simply increasing the version number the Monocle part will likely be more involving. After increasing the version number for Monocle and changing also the artefact group and removing the laws package we are greeted by a number of deprecation warnings and one errors upon compilation. This doesn’t look too bad so maybe we are lucky after all, are we?

\n\n

For the deprecations there is an open issue for providing Scalafix rules for automatic code rewrite but it is not yet done² therefore we have to do it ourselves. But at the issue we find a nice list of deprecated methods and their replacements! As for the error message:

\n\n\n

Compiler error related to Monocle

1 [error] .../Tapir.scala:117:11: object creation impossible.\n2         Missing implementation for:\n3 [error]   def modifyA[F[_]](f: V => F[V])\n4             (s: scala.collection.immutable.ListMap[K,V])\n5             (implicit evidence$1: cats.Applicative[F]):\n6             F[scala.collection.immutable.ListMap[K,V]]\n7             // inherited from trait PTraversal\n8 [error]       new Traversal[ListMap[K, V], V] {\n9 [error]           ^\n

\n\n\n\n\n

This might look intimidating but actually it is just complaining about a missing implementation so we will have to adjust or rewrite the one we are providing. But wait! Didn’t we provide patches to Monocle for the missing instances for ListMap? Yes we did! So how about removing our custom instances?

\n\n\n

Monocle error is gone and only warnings remain

1 [warn] 62 warnings found\n2 [success] ...\n

\n\n\n\n\n

Nice! Always remember: It pays off to provide your custom extensions and patches upstream!

\n\n

However now we get a lot of errors if we try to compile our tests. So we need to investigate. But before we do that let’s fix all these deprecation warnings to get our code clean. Some things are pretty trivial but if we remove the possible command which has no replacement then our code does not compile any longer. We could just ignore it because it is a warning but it is a deprecation one therefore it definitely will come back later to bite us. However we ignore it for now and look at our weird compile error in the tests.

\n\n\n

Weird error after upgrading another dependency

1 [error] ... object scalatestplus is not a member of package org\n

\n\n\n\n\n

This is strange not only because Monocle has no apparent connection to our testing libraries. But doing our research we find an issue in the bugtracker of scalatestplus³ and applying the workaround from there (manually including a dependency to discipline-scalatest) solves our issue. Hooray! But to be honest: I have no idea what is going on behind the scenes here. Likely some dependency issues which cannot be resolved and are silently dropped or so. While we’re at it we simply upgrade our Scala version to 2.13.8.

\n\n

Ignoring the remaining deprecation warnings our tests are running fine and we take another look at the output of migrate-libs tapir within sbt. It seems we have to at least upgrade to tapir 0.18.x. The current stable version being 0.19.x we can see that it depends on http4s 0.23.x which in turn depends on cats-effect 3. Being a major rewrite version 3 of cats-effect will clash with our doobie version. So we will have to switch to the current pre-release version of it. But at least it is close to being released. :-)

\n\n

Because the dependencies are so much weaved together we have no choice but to update them all in one step. We won’t have compiling code either way and will likely get misleading error messages if we do them step by step. So let’s increase some version numbers, take a deep breath and parse some compiler errors. To summarise: We update doobie to 1.0.0-RC2, http4s to 0.23.10 and tapir to 0.19.4. Additionally we have to adjust the tapir swagger ui package because some packaging changed.

\n\n\n

Total error count in the main code after major upgrades

1 [warn] 5 warnings found\n2 [error] 33 errors found\n3 [error] (Compile / compileIncremental) Compilation failed\n

\n\n\n\n\n

Okay, that doesn’t look too bad. Remember, we fixed similar numbers already. But where to start?

\n\n

One of the libraries in the background that all others are using is cats-effect so maybe we should start with that one. Reading the migration guide we realise that there is a Scalafix migration which we could use for automatic conversion of our code. But it says: “Remember to run it before making any changes to your dependencies’ versions.” ;-)

\n\n

So then let us rollback our versions and take a stab at the migration to cats effect 3 via Scalafix. The guide to manually applying the migration is straightforward however the results are a bit underwhelming as nearly nothing is changed. But the migration guide has lots of additional information about changed type hierarchies and so on so we are not left in the dark. Therefore we re-apply our version upgrades again and go on fixing the compilation errors. For convenience we start at our main entry point which is in Tapir.scala.

\n\n

Concentrating on cats-effect first we need to adjust our IOApp.WithContext into a simply IOApp (Basically reverting our changes from some pages back.) and can remove some code which is not needed any longer. Afterwards we have fixed some errors and the ones that still show up seem to be related to tapir and http4s. On the http4s side it seems that we now need a proper Host class instead of our non-empty string. So one option would be do to something like this:

\n\n\n

Possible solution for the configuration type problem

1 host <- IO(\n2   com.comcast.ip4s.Host\n3     .fromString(apiConfig.host)\n4     .getOrElse(throw new RuntimeException(\"Invalid hostname!\"))\n5 )\n

\n\n\n\n\n

It will work but why do we have introduced properly typed configuration then? On the other hand we might have to drop pureconfig because the migrate plugin told us that there is no version of it for Scala 3 yet. However looking at the repository and bugtracker⁴ we can see that basic Scala 3 support is supposed to be there. So let’s try to do it the proper way first!

\n\n

While we’re at it we realise that we also need a Port type also instead of custom PortNumber and of course pureconfig needs to be provided with type classes which can read these types.

\n\n\n

A cleaner solution for the configuration type problem

 1 import com.comcast.ip4s.{ Host, Port }\n 2 import pureconfig._\n 3 import pureconfig.generic.semiauto._\n 4 \n 5 final case class ApiConfig(host: Host, port: Port)\n 6 \n 7 object ApiConfig {\n 8   implicit val hostReader: ConfigReader[Host] =\n 9     ConfigReader.fromStringOpt[Host](Host.fromString)\n10   implicit val portReader: ConfigReader[Port] =\n11     ConfigReader.fromStringOpt[Port](Port.fromString)\n12 \n13   implicit val configReader: ConfigReader[ApiConfig] =\n14     deriveReader[ApiConfig]\n15 }\n

\n\n\n\n\n

This is our new ApiConfig class (comments removed from the snippet) and it looks like it works because we have even less compiler errors now. :-)

\n\n

However there is a new one now for the last part of our for comprehension returning the fiber:

\n\n\n

Cats effect related type error in the main entry point

1 found   : cats.effect.IO[cats.effect.ExitCode]\n2 required: cats.effect.ExitCode\n

\n\n\n\n\n

This is fixed easily though by just changing the fiber = ... to fiber <- ... within the for comprehension. After that we take a look at the error we get from tapir. They are related to the swagger UI and API documentation stuff. Referring to the tapir documentation the changes are quite simple, we just change an import and the way we construct our documentation structure.

\n\n\n

Fix the swagger UI problems in the code

 1 import sttp.tapir.swagger.SwaggerUI\n 2 //...\n 3 docs = OpenAPIDocsInterpreter().toOpenAPI(\n 4   List(\n 5     ProductRoutes.getProduct,\n 6     ProductRoutes.updateProduct,\n 7     ProductsRoutes.getProducts,\n 8     ProductsRoutes.createProduct\n 9   ),\n10   \"Pure Tapir API\",\n11   \"1.0.0\"\n12 )\n13 updatedDocs = updateDocumentation(docs)\n14 docsRoutes  = Http4sServerInterpreter[IO]()\n15                 .toRoutes(SwaggerUI[IO](updatedDocs.toYaml))\n16 //...\n17 httpApp     = Router(\"/\" -> routes, \"/docs\" -> docsRoutes).orNotFound\n

\n\n\n\n\n

Another step done, nice! Let’s enjoy the moment and move on to the other errors which are in our optics part of the file where we define lenses on the OpenAPI structure of tapir which seems to have changed quite a lot. So for starters there is the ReferenceOr structure which simply moved to another place so we can just add an import and reference it directly instead of OpenAPI.ReferenceOr and some errors are gone. Others are about the internal structure for example we now get a Paths type instead of a ListMap on some attributes. But before we dive to deep into this one we might as well thing about refactoring our optics part a bit more because we basically kept our old approach and just changed it so far as to compile with the latest Monocle library. But what about actually utilising the shiny new features? ;-)

\n\n

But let’s save this for later because the really nice features are Scala 3 only. So we just stub our function out and make it simply return the parameter it receives to make it compile again.

\n\n\n

Workaround for the needed Monocle adjustments

1 private def updateDocumentation(docs: OpenAPI): OpenAPI = docs\n

\n\n\n\n\n

Some of the errors left are related to tapir schemas so let’s try them first because they are few and directly related to our data models. Instead of specifying everything manually we try the semi-automatic derivation this time. We soon realise that we still have to specify some instances but the code we need to make it compile looks cleaner than the one before:

\n\n\n

Changes to the tapir schema definitions for our models

 1 object Translation {\n 2   //...\n 3   implicit val schemaForLanguageCode: Schema[LanguageCode] =\n 4     Schema.string\n 5   implicit val schemaForProductName: Schema[ProductName] =\n 6     Schema.string\n 7   implicit val schemaFor: Schema[Translation] =\n 8     Schema.derived[Translation]\n 9 }\n10 object Product {\n11   //...\n12   implicit val schemaForProductId: Schema[ProductId] = Schema.string\n13 \n14   implicit def schemaForNeS[T](implicit a: Schema[T]):\n15     Schema[NonEmptySet[T]] = Schema(SchemaType.SArray(a)(_.toIterable))\n16 \n17   implicit val schemaFor: Schema[Product] = Schema.derived[Product]\n18 }\n

\n\n\n\n\n

So far so good. If we really nailed it we will only know later when it might blow up in our faces or not. ;-)

\n\n

Further on we need to replace the Sync type in our routing classes with Async to fix two more errors. The route creation changed so we have to use a Http4sServerInterpreter[F]().toRoutes(...) function now to create our routes. It still gives some errors but let’s look at our endpoint definitions first. The type signature for endpoints changes from [I, E, O, S] to [A, I, E, O, R]. Sorry for the abbreviation overkill here. The details can be looked up at the tapir docs but the gist is that we have an additional type (“security input”) at the start and instead of the “streaming type” we have a “capabilities type” now at the end. Because we don’t use the security input type we can set it to Unit or could also use the PublicEndpoint type alias which is provided by tapir. In the case of streaming endpoints the output type stays as before (Stream[F, Byte]) and the capabilities type at the end becomes Fs2Streams[F] or Any for all non streaming endpoints.

\n\n

For our streaming endpoint (getProducts) we get an error about the streamBody specification so we can adjust that one or replace it with the new streamTextBody directive. Both ways should work.

\n\n\n

Adjust the stream return type for our tapir endpoint

1 streamTextBody(Fs2Streams[F])(CodecFormat.Json(),\n2   Option(StandardCharsets.UTF_8))\n

\n\n\n\n\n

We are down to a single digit number of compiler errors for our main code. This looks not bad so let’s head on. The main issue now seems to stem from the toRoutes functionality.

\n\n\n

Type errors about the toRoutes function from tapir

 1 [error] overloaded method toRoutes with alternatives:\n 2 [error]   (serverEndpoints: List[sttp.tapir.server.ServerEndpoint[\n 3             sttp.capabilities.fs2.Fs2Streams[F],F]])\n 4               org.http4s.HttpRoutes[F] <and>\n 5 [error]   (se: sttp.tapir.server.ServerEndpoint[\n 6             sttp.capabilities.fs2.Fs2Streams[F],F])\n 7               org.http4s.HttpRoutes[F]\n 8 [error]  cannot be applied to (sttp.tapir.Endpoint[\n 9            Unit,ProductId,StatusCode,Product,Any])\n10 [error]     Http4sServerInterpreter[F]().toRoutes(...) { id =>\n11 [error]                                  ^\n

\n\n\n\n\n

For me this looks like the function will only accept streaming endpoints which doesn’t make sense and would have surely been mentioned in the documentation or some release notes of the tapir project. But we take a closer look and we see that it actually expects ServerEndpoint instances here, not Endpoint ones. Or to quote from the documentation:

\n\n

To interpret a single endpoint, or multiple endpoints as a server, the endpoint descriptions must be coupled with functions which implement the server logic. The shape of these functions must match the types of the inputs and outputs of the endpoint.

\n\n

So server logic is added to an endpoint via one of the functions starting with serverLogic of course. ;-)

\n\n

For our purpose we will use the default one (simply serverLogic). Let’s test it out on one route:

\n\n\n

Fix the toRoutes errors using the serverLogic functionality

 1 final class ProductRoutes[F[_]: Async] ... {\n 2   //...\n 3   private val getRoute: HttpRoutes[F] =\n 4     Http4sServerInterpreter[F]().toRoutes(ProductRoutes.getProduct\n 5       .serverLogic { id =>\n 6         for {\n 7           rows <- repo.loadProduct(id)\n 8           resp = Product\n 9             .fromDatabase(rows)\n10             .fold(StatusCode.NotFound.asLeft[Product])(_.asRight[StatusCode])\n11         } yield resp\n12       })\n13 \n14   private val updateRoute: HttpRoutes[F] =\n15     Http4sServerInterpreter[F]().toRoutes(ProductRoutes.updateProduct\n16       .serverLogic {\n17         case (_, p) =>\n18           for {\n19             cnt <- repo.updateProduct(p)\n20             res = cnt match {\n21               case 0 => StatusCode.NotFound.asLeft[Unit]\n22               case _ => ().asRight[StatusCode]\n23             }\n24           } yield res\n25       })\n26   //...\n27 }\n

\n\n\n\n\n

And it compiles fine! So we just need to move our logic into the serverLogic function part and we are set. Pretty cool but once we fix it we get another error from our main entry point:

\n\n\n

A new error from our main entry point

1 Tapir.scala:42:26: method executionContextResource overrides nothing\n

\n\n\n\n\n

However we can simply remove it and are done. Oh wait! We are skipped two things: first there is still the optics implementation left and second we want to fix this example problem in the one endpoint definition. Besides that we also get a lot of compilation errors for our tests. We turn there first an can very quickly fix our integration tests by removing the no longer used IO.contextShift from our DoobieRepositoryTest and by providing an implicit IORuntime within our BaseSpec class.

\n\n

It turns out that for our regular tests we can apply the same fix to the BaseSpec there and also remove some obsolete code and adjust our imports because the http4s library has now a type called ProductId which clashes with our own one. After changing the Effect type in our TestRepository to Async the only thing left seems to be the ScalaCheck generators for our ApiConfig but these are also fixed easily.

\n\n

So we fixed the compilation errors in our tests, but alas some tests are failing. :-(

\n\n

At least the integration tests look fine, so let’s take a look at the possible reasons for our failing tests. The failing ones are: ApiConfigTest, ProductRoutesTest and ProductsRoutesTest. The first one spills out the following message:

\n\n\n

Equality issue causing a failing test

1 ApiConfig(127.0.0.1,34019) was not equal to ApiConfig(127.0.0.1,34019)\n

\n\n\n\n\n

I don’t know about you dear reader but I have stumbled into equality issues frequently (not always though but regular) so this should be resolvable by changing the c must be(expected) line in the test.

\n\n\n

Fix for the equality issue in the ApiConfig test

1 ConfigSource.fromConfig(config).at(\"api\").load[ApiConfig] match {\n2   case Left(e)  => fail(s\"Parsing a valid configuration must succeed! ($e)\")\n3   case Right(c) => withClue(\"Config must be equal!\")(c === expected)\n4 }\n

\n\n\n\n\n

In addition we add an implicit instance for the Eq of cats into the companion object of the ApiConfig class.

\n\n\n

Instance for cats Eq and ApiConfig

1 implicit val eqApiConfig: Eq[ApiConfig] = Eq.instance { (a, b) =>\n2   a.host === b.host && a.port === b.port\n3 }\n

\n\n\n\n\n

This fixes it and we can turn to the two remaining one. We soon find that the error message returned by tapir (which we check in the tests) has changed and is now more detailed so we simply adjust the according line in both tests and we are done here.

\n\n

Nice, we have a compiling project again and if ignore our stubbed out Monocle function for now the migrate-libs task output some good looking results. So we execute the migrate-scalacOptions task next and take a look at the results. It uses the same notifications like the migrate-libs task to highlight problems. So far we have a couple of flags that the plugin could not recognize, quite a lot which are not valid any longer and some which we can use or have to rename.

\n\n

First we change our compilerSettings function and add a case for Scala 3.

\n\n\n

Compiler settings for Scala 3 in build.sbt

 1 //...\n 2 case Some((3, _)) =>\n 3   Seq(\n 4     \"-deprecation\",\n 5     \"-explain-types\",\n 6     \"-feature\",\n 7     \"-language:higherKinds\",\n 8     \"-unchecked\",\n 9     //\"-Xfatal-warnings\", // Disable for migration\n10     \"-Ycheck-init\",\n11     \"-Ykind-projector\"\n12   )\n13 //...\n

\n\n\n\n\n

Also we add a libraryDependencies setting into our commonSettings block to only activate plugins for Scala 2.

\n\n\n

Load compiler plugins only for Scala 2 in build.sbt

 1 //...\n 2 libraryDependencies ++= (\n 3   if (scalaVersion.value.startsWith(\"2\")) {\n 4     Seq(\n 5       compilerPlugin(\"com.olegpy\"    %% \"better-monadic-for\" % \"0.3.1\"),\n 6       compilerPlugin(\"org.typelevel\" % \"kind-projector\"      % \"0.13.2\" cross CrossV\\\n 7 ersion.full)\n 8     )\n 9   } else {\n10     Seq()\n11   }\n12 ),\n13 //...\n

\n\n\n\n\n

Next is settings some flags for dependencies of which we want the 2.13 version.

\n\n\n

Using the 2.13 version for some dependencies in build.sbt

1 library.pureConfig.cross(CrossVersion.for3Use2_13),\n2 library.refinedCats.cross(CrossVersion.for3Use2_13),\n3 library.refinedCore.cross(CrossVersion.for3Use2_13),\n4 library.refinedPureConfig.cross(CrossVersion.for3Use2_13),\n

\n\n\n\n\n

Last but not least the big topic “new syntax” is lurking around the corner. But we will first add two more compiler flags which should make our 2.13 code compile in Scala 3 and add Scala 3 to the crossScalaVersions setting.

\n\n\n

Two migration compiler flags and the 3 version in build.sbt

 1 //...\n 2 case Some((3, _)) =>\n 3   Seq(\n 4     \"-deprecation\",\n 5     \"-explain-types\",\n 6     \"-feature\",\n 7     \"-language:higherKinds\",\n 8     \"-unchecked\",\n 9     //\"-Xfatal-warnings\", // Disable for migration\n10     \"-Ycheck-init\",\n11     \"-Ykind-projector\",\n12     // Gives warnings instead of errors on most syntax changes.\n13     \"-source:3.0-migration\",\n14     // Resolve warnings via the compiler of possible.\n15     \"-rewrite\",\n16   )\n17 //..\n18 crossScalaVersions := Seq(scalaVersion.value, \"3.1.1\"),\n19 //...\n

\n\n\n\n\n

Time for a first test run! We switch to Scala 3 by using ++3.1.1 in the sbt shell and issue a clean followed by a compile. Please not that you should have reloaded or restarted your sbt instance before to make it reflect all of our changes.

\n\n\n

Errors about conflicting dependencies versions

 1 [error] Modules were resolved with conflicting cross-version suffixes in \n 2   ProjectRef(uri(\".../tapir-scala-3/\"), \"tapir\"):\n 3 [error]    org.scala-lang.modules:scala-xml _2.13, _3\n 4 [error]    org.typelevel:simulacrum-scalafix-annotations _3, _2.13\n 5 [error]    org.typelevel:cats-kernel _3, _2.13\n 6 [error]    eu.timepit:refined _2.13, _3\n 7 [error]    org.typelevel:cats-core _3, _2.13\n 8 [error] stack trace is suppressed; run last update for the full output\n 9 [error] (update) Conflicting cross-version suffixes in:\n10   org.scala-lang.modules:scala-xml,\n11   org.typelevel:simulacrum-scalafix-annotations,\n12   org.typelevel:cats-kernel,\n13   eu.timepit:refined,\n14   org.typelevel:cats-core\n15 [error]\n

\n\n\n\n\n

Looks like we have a problem here. The libraries that we include in the version for 2.13 depend on other which collide with transitive dependencies of others using Scala 3. Digging through the maven central we can see that there are artefacts published for Scala 3 for pureconfig and refined, so what about removing our cross version settings and try them directly?

\n\n

We soon find out that the refined module for pureconfig is not available for Scala 3 and in addition it seems that the generic derivation module is also not there. :-(

\n\n\n\n

We could call it a day and stick with Scala 2 for the time being. However maybe there is a strong need for an upgrade. A library which only works partially under Scala 2 or other reasons. So imagine that we refactor our service a bit by removing some dependencies and add some more boilerplate to make up for it. At first we could simply remove the CrossVersion settings and use only libraries which are released for Scala 3. This move leaves us with hundreds of compiler errors. Many of them seem to be related to refined.

\n\n

Because we have to start somewhere we try to create our ConfigReader instances for pureconfig manually to avoid the dependency to the derivation module. Luckily there are some helpers like the ConfigReader.forProductX methods which make this quite easy.

\n\n\n

A manual constructed reader for pureconfig

1 implicit val configReader: ConfigReader[DatabaseConfig] =\n2   ConfigReader.forProduct4(\"driver\", \"url\", \"user\", \"pass\")\n3     (DatabaseConfig(_, _, _, _))\n

\n\n\n\n\n

The code compiles again (on 2.13!) and the tests are looking good. On Scala 3 we run again into the problem that the refined pureconfig module is not available. So we could either drop our beloved refined types or we write manual readers for them. In Scala 3 we could use the opaque type aliases⁵ to get more type safety but they are much less powerful than refined types. Well, we’ll see how it goes, so at first we define companion objects for our refined types to gain some more functionality i.e. the from method to convert arbitrary types into refined ones.

\n\n\n

A companion object for a refined type providing more functionality

1 type DatabaseLogin = String Refined NonEmpty\n2 object DatabaseLogin extends RefinedTypeOps[DatabaseLogin, String]\n3   with CatsRefinedTypeOpsSyntax\n

\n\n\n\n\n

As you can see this is pretty simple, we don’t even need to implement something ourselves. Getting the instances for ConfigReader looks pretty simple too.

\n\n\n

ConfigReader instances for refined types - Take 1

1 implicit val loginReader: ConfigReader[DatabaseLogin] =\n2   ConfigReader.fromStringOpt(s => DatabaseLogin.from(s).toOption)\n3 implicit val passReader: ConfigReader[DatabasePassword] =\n4   ConfigReader.fromStringOpt(s => DatabasePassword.from(s).toOption)\n5 implicit val urlReader: ConfigReader[DatabaseUrl] =\n6   ConfigReader.fromStringOpt(s => DatabaseUrl.from(s).toOption)\n

\n\n\n\n\n

But wait we get the dreaded “ambiguous implicit values” compiler error now. Since our types are all refined from String under the hood, the compiler throws an error at us.

\n\n\n\n

Well, it seems that we don’t need to because the pureconfig module of refined is a thing. But we cannot use it as it is because the usage of reflection is a no-go with Scala 3. But we can try to copy the base function and make some adjustments.

\n\n\n

Copied and adjusted converter function from refined-pureconfig

 1 implicit def refTypeConfigConvert[F[_, _], T, P](\n 2     implicit configConvert: ConfigConvert[T],\n 3     refType: RefType[F],\n 4     validate: Validate[T, P]\n 5 ): ConfigConvert[F[T, P]] =\n 6   new ConfigConvert[F[T, P]] {\n 7     override def from(cur: ConfigCursor): ConfigReader.Result[F[T, P]] =\n 8       configConvert.from(cur) match {\n 9         case Left(es) => Left(es)\n10         case Right(t) =>\n11           refType.refine[P](t) match {\n12             case Left(because) =>\n13               Left(\n14                 ConfigReaderFailures(\n15                   ConvertFailure(\n16                     reason = CannotConvert(\n17                       value = cur.valueOpt.map(_.render()).getOrElse(\"none\"),\n18                       toType = \"a refined type\",\n19                       because = because\n20                     ),\n21                     cur = cur\n22                   )\n23                 )\n24               )\n25             case Right(refined) => Right(refined)\n26           }\n27       }\n28     override def to(t: F[T, P]): ConfigValue =\n29       configConvert.to(refType.unwrap(t))\n30   }\n

\n\n\n\n\n

So we copied it and simply omitted the type tag stuff which is based on reflection. This way we loose some information for our error message but we gain a hopefully Scala 3 compatible refined type reader for pureconfig. Many thanks again at this point to Frank S. Thomas the creator of the wonderful refined library and all the contributors!

\n\n

Good, let’s switch to Scala 3 (++3.1.1 in the sbt shell) and try a clean compile which will give us a lot of errors still. However first things first. We notice some refined related ones in the top and fix them. The notation changed and we finally can use literal types so there is no longer a need for this weird W.andsoon.T constructs.

\n\n

After fixing them we get 266 errors. Oh why cruel fate? But looking at them we can see that a lot of them come from our LanguageCodes class. As we remember that due to macro issues several things are not there yet in libraries under Scala 3 we are again faced with the decision to abandon refined types or find yet another workaround. To keep this chapter from growing exponentially I’ll pick the lazy (and dirty) workaround for this time. A simple nudge in your favourite editor should convert the code in the file to something like this:

\n\n\n

Workaround for missing refined macros

1 val all: Seq[LanguageCode] = Seq(\n2   LanguageCode.unsafeFrom(\"ad\"),\n3   LanguageCode.unsafeFrom(\"ae\"),\n4   LanguageCode.unsafeFrom(\"af\"),\n5   //...\n6 )\n

\n\n\n\n\n

This is not the nicest solution but it leaves us with only 17 errors to fix and they look like they have the same cause. So after fixing the same thing in our routing examples we are down to one error:

\n\n\n

1 [error] -- [E008] Not Found Error: .../Tapir.scala:47:31\n2 [error] 47 |      (apiConfig, dbConfig) <- IO {\n3 [error]    |                               ^\n4 [error]    |value withFilter is not a member of \n5 [error]    |  cats.effect.IO[(com.wegtam.books.pfhais.tapir.config.ApiConfig,\n6 [error]    |  com.wegtam.books.pfhais.tapir.config.DatabaseConfig\n7 [error]    |)]\n8 [error] one error found\n9 [error] one error found\n

\n\n\n\n\n

Well the only time I got that one was when I removed the better-monadic-for compiler plugin and it won’t be available for Scala 3 because it shouldn’t be needed. But we can solve it by de-composing our code into several chunks.

\n\n\n

Fixing the withFilter error in Tapir.scala

1 for {\n2   cfg       <- IO(ConfigFactory.load(getClass().getClassLoader()))\n3   apiConfig <- IO(ConfigSource.fromConfig(cfg).at(\"api\")\n4     .loadOrThrow[ApiConfig])\n5   dbConfig  <- IO(ConfigSource.fromConfig(cfg).at(\"database\")\n6     .loadOrThrow[DatabaseConfig])\n7   //...\n

\n\n\n\n\n

Awesome! We have our main code compiling under Scala 3 now! =)

\n\n

Compiling the tests suites greets us with a couple of errors though, so no celebrations just yet. Some of them are refined related like in the main code (missing macros) so we will apply our unsafeFrom workaround for them and get rid of them. Then we have an error about implicit values needing an explicit type which is a good practice anyway. The integration tests look similar and are easy to fix too. However we have two failing tests after compilation. One within each route test and both related to checking the some error response for malformed requests. So we simply adjust the error message we test for and we are done.

\n\n\n\n

This is awesome but according to the official migration guide we should also try to migrate our syntax. So let’s run the migrate-syntax tapir command in the sbt shell.

\n\n\n

Result of the migrate-syntax command

 1 [info]\n 2 [info] The syntax incompatibilities have been fixed in tapir / Test\n 3 [info]\n 4 [info]\n 5 [info] You can now commit the change!\n 6 [info] Then you can run the next command:\n 7 [info]\n 8 [info] migrate tapir\n 9 [info]\n10 [info]\n

\n\n\n\n\n

Looking at the source we can see that only some annotations have been added. And finally we run migrate tapir and it bails out with an error. I’ll spare you the gory details but I did not have any clue about what was going wrong. But we know that everything is fine under Scala 3 anyway. So what about simply ignoring the tools lamentations and move on with our lives? Sounds good? Yeah, to me too. :-)

\n\n

Before calling it a day we should switch back to 2.13 and do a clean compile to see some unused imports warnings be printed out. These are easy to fix and cleaner code is easier to maintain. Finally we adjust our build.sbt to make the switch permanent.

\n\n\n

Switch to Scala 3 by default

1   //...\n2   scalaVersion := \"3.1.1\",\n3   crossScalaVersions := Seq(scalaVersion.value),\n4   //...\n

\n\n\n\n\n

We can also remove some code related to Scala 2 (plugins and compiler settings).

\n\n\n\n

Well, we should change our scalafmt configuration because we surely do not want to get our shiny new Scala 3 code formatted according to Scala 2 syntax. ;-)

\n\n\n

A new configuration for scalafmt

 1 version        = 3.4.3\n 2 runner.dialect = scala3\n 3 style          = defaultWithAlign\n 4 # Other options...\n 5 danglingParentheses.preset = true\n 6 maxColumn                  = 120\n 7 newlines.forceBeforeMultilineAssign = def\n 8 project.excludeFilters     = [\".*\\\\.sbt\"]\n 9 rewrite.rules              = [Imports, RedundantBraces, RedundantParens]\n10 rewrite.imports.sort       = ascii\n11 rewriteTokens              = {\n12   ...\n13 }\n14 spaces.inImportCurlyBraces = true\n15 unindentTopLevelOperators  = true\n

\n\n\n\n\n

We used the opportunity to upgrade to the latest scalafmt and use a bit different settings. Running a scalafmtAll rewrites a lot of code, so we need to check if it still compiles. This looks fine so we are done, are we?

\n\n\n\n

Wait, we’ve been here before, haven’t we? Hint: remember that updateDocumentation function using optics that we have stubbed out? Sometimes I wonder how many times smaller (or bigger) things get dropped under the table during such migrations only to be re-implemented later with great effort for new. We start with adding the needed import for the new Monocle (import monocle.syntax.all._) and try some things out. First we have this nice implicit extractor for the regular expression which defines our refined LanguageCode type. This one doesn’t seem to work any longer. To keep things simple, we just write it down in the code.

\n\n

Onwards to the .focus macro which is a killer feature of the new Monocle release. Instead of writing and combining all our custom Lens instances we should now be able to do something like the following:

\n\n\n

The Monocle focus in action

1 docs.focus(_.components.some.parameters.each.schema.pattern)\n2   .replace(uuidRegex.some)\n

\n\n\n\n\n

Looks neat but we get an error that the function components is an overloaded one which seems is not supported. So it won’t be that easy.

\n\n

After poking around and asking for help (Please ask for help if you need it, there is absolutely nothing wrong with this.) we realise that we will have to generate lenses like before because the shiny focus macro cannot (yet) solve this for us. Sadly the GenLens macro also has problems with overloaded methods therefore we try to define Getter and Setter functionality for these manually. Some lenses can still be generated via GenLens like this GenLens[Operation](_.parameters) for example. For others we define code like the following:

\n\n\n

Manually define Getter and Setter

1 val componentsGetter = Getter[OpenAPI, Option[Components]](_.components)\n2 val componentsSetter = Setter[OpenAPI, Option[Components]](\n3   f => o => o.copy(components = f(o.components))\n4 )\n

\n\n\n\n\n

However this is even more cumbersome than the manual macros before so maybe there is another way… And it turns out that we have another optics library for Scala which is called Quicklens⁶. The fact that is comes from the same people that do the tapir project looks promising and indeed we do find a working implementation quickly.

\n\n\n

Using optics from Quicklens to update our documentation

 1 private def updateDocumentation(docs: OpenAPI): OpenAPI = {\n 2   // Our regular expressions.\n 3   val langRegex = ???\n 4   val uuidRegex = ???\n 5   // Update the documentation structure.\n 6   val updateProductId = docs\n 7     .modify(_.paths.pathItems.at(\"/product/{id}\").parameters.each\n 8     .eachRight.schema.at.eachRight.pattern)\n 9     .using(_ => uuidRegex.some)\n10   val updateModelProduct = updateProductId\n11     .modify(_.components.at.schemas.at(\"Product\").eachRight\n12     .properties.at(\"id\").eachRight.pattern)\n13     .using(_ => uuidRegex.some)\n14   val updateModelTranslation = updateModelProduct\n15     .modify(_.components.at.schemas.at(\"Translation\").eachRight\n16     .properties.at(\"lang\").eachRight.pattern)\n17     .using(_ => langRegex.some)\n18   updateModelTranslation\n19 }\n

\n\n\n\n\n

The .modify functionality looks very similar to .focus from Monocle and apart from the fact that the modifiers have different names we also do the same like in the Monocle code (read traverse through our structure). While Monocle would allow us to use more advanced features we are happy with Quicklens here because it fulfils our needs.

\n\n\n\n

Nothing! Go forth, ship your release and treat yourself for having ported your project to Scala 3. :-)

\n\n

Of course I cheated because there is always something left like the not working example for one of our ProductsRoutes endpoints and furthermore: We have used package objects in our code which have been dropped in Scala 3. Therefore we should refactor these also. So feel free to do the required changes as a final exercise.

\n\n\n\n

\nEpilogue

\n\n\n\n

I personally hope that you had some fun reading this, have learned something and got some ideas about what is possible in the realm of programming. For me it was fun to write this book and I also learned a lot while writing it. I want to thank all the people that got back to me with questions, hints and ideas about what might be missing.

\n\n

The next time some person points out that all this “academic functional nonsense” is for naught please remember some of the things you read here. We have seen several things that imply that we can use pure functional programming in our daily work:

\n\n

better understandable code
better testable code
better performing code (Yes!)
better type safety using refined types
deriving boilerplate code from our statically typed models and logic
easier working with deeply nested structures using optics

\n\n

There really is no excuse to stick to impure and messy stuff that is hard to maintain. But please remember also that functional programming is not done for the sake of itself. It is another and in my opinion better way to deliver value. In our field of work this (value) usually means some working software / program / application / whatever.

\n\n

Also please do not consider this book a complete knowledge trove. Several things might already be outdated, some topics have only been scratched on the surface and other things are missing completely.

\n\n

So, people of the baud and the electron: Go out there, venture forth, cross boundaries, use the knowledge which is there and last but not least - Have fun!

\nNotes

Foreword

1https://akka.io↩

\n\n

2http://slick.lightbend.com↩

\n\n

3https://typelevel.org/cats/↩

\n\n

4https://typelevel.org/cats-effect/↩

\n\n

5https://http4s.org/↩

\n\n

6https://tpolecat.github.io/doobie/↩

\n\n

7https://github.com/fthomas/refined↩

\n\n

8https://fs2.io/↩

\n\n

9https://github.com/softwaremill/tapir↩

\n\n

10https://github.com/julien-truffaut/Monocle↩

\n\n

Maybe there is another way

Impure implementation

1http://slick.lightbend.com/doc/3.3.1/database.html↩

\n\n

Pure implementation

1https://pureconfig.github.io/↩

\n\n

2https://github.com/typelevel/kittens↩

\n\n

3https://typelevel.org/cats-effect/typeclasses/sync.html↩

\n\n

4http://slick.lightbend.com/doc/3.3.1/sql.html↩

\n\n

5https://blog.acolyer.org/2019/07/03/one-sql-to-rule-them-all/↩

\n\n

6https://httpie.org/↩

\n\n

7https://github.com/http4s/http4s/issues/2371↩

\n\n

8https://www.wartremover.org/↩

\n\n

9https://typelevel.org/cats-effect/datatypes/ioapp.html↩

\n\n

What about tests?

1http://www.scalacheck.org/↩

\n\n

2http://www.scalatest.org/user_guide/selecting_a_style↩

\n\n

3https://rspec.info/↩

\n\n

4https://github.com/scalatest/scalatest/issues/1370↩

\n\n

Adding benchmarks

1https://jmeter.apache.org/↩

\n\n

2https://github.com/http4s/http4s/issues/2855↩

\n\n

Documenting your API

1https://swagger.io/↩

\n\n

2https://swagger.io/tools/swagger-ui/↩

\n\n

3https://swagger.io/tools/swagger-editor/↩

\n\n

4https://github.com/swagger-akka-http/swagger-akka-http↩

\n\n

5https://www.youtube.com/watch?v=snbsYyBS4Bs↩

\n\n

6http://julienrf.github.io/endpoints/↩

\n\n

7https://github.com/http4s/rho↩

\n\n

8https://github.com/softwaremill/tapir↩

\n\n

9https://www.servant.dev/↩

\n\n

10https://www.cs.ox.ac.uk/people/jeremy.gibbons/publications/poptics.pdf↩

\n\n

11https://github.com/julien-truffaut/Monocle↩

\n\n

12https://github.com/milessabin/shapeless↩

\n\n

13https://tools.ietf.org/html/rfc4122↩

\n\n

14https://www.ecma-international.org/ecma-262/5.1/#sec-7.8.5↩

\n\n

Moving to Scala 3…

1We could have used scalafix here but I consider this overkill for such a thing.↩

\n\n

2https://github.com/optics-dev/Monocle/issues/1001↩

\n\n

3https://github.com/scalatest/scalatestplus-scalacheck/issues/36↩

\n\n

4https://github.com/pureconfig/pureconfig/issues/970↩

\n\n

5https://docs.scala-lang.org/scala3/reference/other-new-features/opaques.html↩

\n\n

6https://github.com/softwaremill/quicklens↩

\n\n\n

\n\n

\n\n","tocStructure",[227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261],{"_79":73,"_42":69,"_70":-5,"_74":75,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":80,"_42":81,"_70":-5,"_74":82,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":73,"_42":87,"_70":-5,"_74":88,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":80,"_42":91,"_70":-5,"_74":92,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":96,"_70":-5,"_74":97,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":101,"_70":-5,"_74":102,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":104,"_70":-5,"_74":105,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":73,"_42":107,"_70":-5,"_74":108,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":80,"_42":111,"_70":-5,"_74":112,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":73,"_42":115,"_70":-5,"_74":116,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":80,"_42":119,"_70":-5,"_74":120,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":123,"_70":-5,"_74":124,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":126,"_70":-5,"_74":127,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":129,"_70":-5,"_74":130,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":80,"_42":132,"_70":-5,"_74":133,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":136,"_70":-5,"_74":137,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":123,"_70":-5,"_74":139,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":126,"_70":-5,"_74":141,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":143,"_70":-5,"_74":144,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":80,"_42":146,"_70":-5,"_74":147,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":150,"_70":-5,"_74":151,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":153,"_70":-5,"_74":154,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":80,"_42":156,"_70":-5,"_74":157,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":160,"_70":-5,"_74":161,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":163,"_70":-5,"_74":164,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":166,"_70":-5,"_74":167,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":80,"_42":169,"_70":-5,"_74":170,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":173,"_70":-5,"_74":174,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":176,"_70":-5,"_74":177,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":179,"_70":-5,"_74":180,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":80,"_42":182,"_70":-5,"_74":183,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":186,"_70":-5,"_74":187,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":95,"_42":189,"_70":-5,"_74":190,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":73,"_42":192,"_70":-5,"_74":193,"_72":73,"_98":-5,"_99":-5,"_85":-5},{"_79":73,"_42":196,"_70":-5,"_74":197,"_72":73,"_98":-5,"_99":-5,"_85":-5},"publicChapterContent","nextChapter","sections",[235],"routes/_store+/_layout","routes/_store+/_readers+/_layout","actionData","errors"]

\nThe state of the art

Table of Contents

\nForeword

\nAbout this book

\nCopyleft Notice

\nThanks

\nOur use case

\nService specification

\nData model

\nDatabase

\nThe products table

\nThe names table

\nHTTP API

\nThe state of the art

\nProblems

\nMaybe there is another way

\nImpure implementation

\nModels

\nDatabase layer

\nMigrations

\nSlick tables

\nAkka-HTTP routes

\nProblems with the solution

\nPure implementation

\nPure configuration handling

\nModels

\nDatabase layer

\nMigrations

\nDoobie

\nhttp4s routes

\nStreaming - Take 1

\nStreaming - Take 2

\nJSON trouble

\nStarting the application

\nWhat about tests?

\nTesting the impure service

\nGenerators

\nUnit Tests

\nTesting a JSON codec

\nMore tests

\nSome refactoring

\nIntegration Tests

\nTesting the repository

\nTesting the routes

\nTesting the pure service

\nUnit Tests

\nTesting our routes

\nIntegration Tests

\nTesting the FlywayDatabaseMigrator

\nAdding benchmarks

\nOur environment

\nSystem environment

\nService workstation

\nClient workstation

\nComparison

\nCreate 100.000 products

\nLoad 100.000 products

\nUpdate 100.000 products

\nBulk load all 100.000 products

\nSummary

\nDocumenting your API

\nThe lay of the land

\nUsing types to describe an API

\nA pure implementation using tapir.

\nBasics

\nProduct routes

\nProducts routes

\nDocumentation via OpenAPI

\nProviding example data

\nRefining the generated documentation

\nMoving to Scala 3…

\nStep 1: Updating to 2.13.x

\nDetails and some compiling issues

\nStep 2: Migrating to Scala 3

\nEpilogue

\nNotes

Foreword

Maybe there is another way

Impure implementation

Pure implementation