Testing Spark Applications
Testing Spark Applications
Writing Spark code is hard... well designed, performant Spark tests are even harder. You need a robust test suite to identify performance bottlenecks in your code and refactor with ease. This book teaches you how to write a beautiful test suite and how to run the tests whenever code is pushed to the master branch.
About the Book
The book discusses Scala testing basics with the Scalatest framework. It uses the spark-fast-tests library to demonstrate column equality testing and DataFrame equality testing. Spark tests can run slowly so the book provides several practical workflows to keep tests running quickly. Spark code frequently reads and writes to disk and the book covers how to write tests for code with I/O. Configuring a test suite properly can make it around 70% faster and this book explains the configuration options you should have on your radar. Complex transformations (e.g. aggregations) and column types (e.g. MapType, ArrayType, StructType, BinaryType) have special testing considerations that are addressed in separate chapters.
This book has a heavy emphasis on software engineering best practices and will teach you skills that are useful for any language or framework.
Table of Contents
-
Introduction
- Messy data
- Nightmare deploys
- Empower refactoring
- Tests encourage code that doesn’t have side effects
- Identifying code bottlenecks
- Test suites document behavior
- Technologies used
-
Testing Scala with Scalatest
- Writing a simple test
- Directory organization
- build.sbt
- More tests
- Running tests and configuring output
- assertThrows
- assertDoesNotCompile
- Other assertions
- Other test formats
- Test library alternatives
- Testing Spark applications
- Next steps
-
Column Equality Tests
- Custom DataFrame Transformation Refresher
- Spark project setup
- assertColumnEquality with spark-fast-tests
- Conclusion
-
Quieting Test Output
- Customizing test suite output
-
Creating DataFrames for Tests
- toDF
- createDataFrame
- createDF
- Including spark-daria in your projects
- Next steps
-
DataFrame Equality Tests
- Simple example
- assertSmallDataFrameEquality error messages
- Next steps
-
Running Tests
- Running from the SBT console
- Running a single test file
- Running a single test
- Best workflow
-
Approximate Equality
- Difference between double, float and decimal
- When assertColumnEquality falls short
- assertFloatTypeColumnEquality to the rescue
- assertApproximateDataFrameEquality
- Conclusion
-
Testing User Defined Functions
- Creating a UDF
- Testing a UDF
- Check the UDF fails with null input
- The billion dollar mistake
- Verifying test failure in the test suite
- Next steps
-
Testing Spark Column Functions
- Simple example
- How Spark functions handle null
- Important takeaway
- Why print DataFrames from the test suite?
- Next steps
-
Testing Filesystem Reads
- Untestable code
- Setting the path as a param
- Testing with the config pattern
- Elegant testing with dependency injection
- Abstracting custom transformation to a separate function
- Next steps
-
Testing Filesystem Writes
- Simple example
- Rude tests leave garbage behind
- Performance considerations
- Next steps
-
Identifying Bottlenecks
- Let’s find the bottleneck
- Benchmarking individual transformations
- Contrived but representative
- Conclusion
-
Organizing Tests
- Some pure Scala
- Poorly organized Spark tests
- Test suite rules to follow
- Good Spark test organization
- Tests should be descriptive and document behavior
- Quantifying performance difference
- Next steps
-
Test Suite Configuration
- Shuffle partitions
- javaOptions
- Conclusion
-
Testing Aggregations
- groupBy refresher
- groupBy with two columns
- groupBy with filters
- Conclusions
Authors have earned$10,098,681writing, publishing and selling on Leanpub, earning 80% royalties while saving up to 25 million pounds of CO2 and up to 46,000 trees.
Learn more about writing on Leanpub
The Leanpub 45-day 100% Happiness Guarantee
Within 45 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms
Free Updates. DRM Free.
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers), EPUB (for phones and tablets) and MOBI (for Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
Learn more about Leanpub's ebook formats and where to read them
Top Books
C++20
Rainer GrimmC++20 is the next big C++ standard after C++11. As C++11 did it, C++20 changes the way we program modern C++. This change is, in particular, due to the big four of C++20: ranges, coroutines, concepts, and modules.
The book is almost daily updated. These incremental updates ease my interaction with the proofreaders.
A Guide to Artificial Intelligence in Healthcare
Dr. Bertalan MeskoCan we stay human in the age of A.I.? To go even further, can we grow in humanity, can we shape a more humane, more equitable and sustainable healthcare? This e-book aims to prepare healthcare and medical professionals for the era of human-machine collaboration. Read our guide to understanding, anticipating and controlling artificial intelligence.
Atomic Kotlin
Bruce Eckel and Svetlana IsakovaFor both beginning and experienced programmers! From the author of the multi-award-winning Thinking in C++ and Thinking in Java together with a member of the Kotlin language team comes a book that breaks the concepts into small, easy-to-digest "atoms," along with exercises supported by hints and solutions directly inside IntelliJ IDEA!
Introducing EventStorming
Alberto BrandoliniThe deepest tutorial and explanation about EventStorming, straight from the inventor.
C++ Best Practices
Jason TurnerLevel up your C++, get the tools working for you, eliminate common problems, and move on to more exciting things!
Ansible for DevOps
Jeff GeerlingAnsible is a simple, but powerful, server and configuration management tool. Learn to use Ansible effectively, whether you manage one server—or thousands.
Everyday Rails - RSpecによるRailsテスト入門
Junichi Ito (伊藤淳一), AKIMOTO Toshiharu, 魚振江, and Aaron SumnerRSpecを使ってRailsアプリケーションに信頼性の高いテストを書く実践的なアドバイスを提供します。詳細で丁寧な説明は本書のオリジナルコンテンツです。また、説明には実際に動かせるサンプルアプリケーションも使用します。本書は2017年版にアップデートされ、RSpec 3.6やRails 5.1といった新しい環境に対応しています!さあ、自信をもってテストできるようになりましょう!
The Hundred-Page Machine Learning Book
Andriy BurkovEverything you really need to know in Machine Learning in a hundred pages.
Composing Software
Eric ElliottAll software design is composition: the act of breaking complex problems down into smaller problems and composing those solutions. Most developers have a limited understanding of compositional techniques. It's time for that to change.
R Programming for Data Science
Roger D. PengThis book brings the fundamentals of R programming to you, using the same material developed as part of the industry-leading Johns Hopkins Data Science Specialization. The skills taught in this book will lay the foundation for you to begin your journey learning data science. Printed copies of this book are available through Lulu.
Top Bundles
- #1
Software Architecture for Developers: Volumes 1 & 2 - Technical leadership and communication
2 Books
"Software Architecture for Developers" is a practical and pragmatic guide to modern, lightweight software architecture, specifically aimed at developers. You'll learn:The essence of software architecture.Why the software architecture role should include coding, coaching and collaboration.The things that you really need to think about before... - #2
All the Books of The Medical Futurist
6 Books
We put together the most popular books from The Medical Futurist to provide a clear picture about the major trends shaping the future of medicine and healthcare. Digital health technologies, artificial intelligence, the future of 20 medical specialties, big pharma, data privacy, digital health investments and how technology giants such as Amazon... - #3
Cisco CCNA 200-301 Complet
4 Books
Ce lot comprend les quatre volumes du guide préparation à l'examen de certification Cisco CCNA 200-301. - #4
Linux Administration Complet
4 Books
Ce lot comprend les quatre volumes du Guide Linux Administration :Linux Administration, Volume 1, Administration fondamentale : Guide pratique de préparation aux examens de certification LPIC 1, Linux Essentials, RHCSA et LFCS. Administration fondamentale. Introduction à Linux. Le Shell. Traitement du texte. Arborescence de fichiers. Sécurité... - #5
Learn Git, Bash, and Terraform the Hard Way
3 Books
Learn Git, Bash and Terraform using the Hard Way method.These technologies are essential tools in the DevOps armoury. These books walk you through their features and subtleties in a simple, gradual way that reinforces learning rather than baffling you with theory. - #6
PowerShell
3 Books
Buy every PowerShell book from Adam Bertram at a 20% discount! - #7
Software Architecture and Beautiful APIs
2 Books
There is no better way to learn how to design good APIs than to look at many existing examples, complementing the Software Architecture theory on API design. - #8
9 Books-Bundle: Shut Up and Code!
9 Books
"Shut up and code." Laughter in the audience. The hacker had just plugged in his notebook and started sharing his screen to present his super-smart Python script. "Shut up and code" The letters written in a white literal coding font on black background was the hackers' home screen background mantra. At the time, I was a first-year computer... - #9
CCDE Practical Studies (All labs)
3 Books
CCDE lab - #10
Modern C++ by Nicolai Josuttis
2 Books