Testing Spark Applications
Testing Spark Applications
Writing Spark code is hard... well designed, performant Spark tests are even harder. You need a robust test suite to identify performance bottlenecks in your code and refactor with ease. This book teaches you how to write a beautiful test suite and how to run the tests whenever code is pushed to the master branch.
About the Book
The book discusses Scala testing basics with the Scalatest framework. It uses the spark-fast-tests library to demonstrate column equality testing and DataFrame equality testing. Spark tests can run slowly so the book provides several practical workflows to keep tests running quickly. Spark code frequently reads and writes to disk and the book covers how to write tests for code with I/O. Configuring a test suite properly can make it around 70% faster and this book explains the configuration options you should have on your radar. Complex transformations (e.g. aggregations) and column types (e.g. MapType, ArrayType, StructType, BinaryType) have special testing considerations that are addressed in separate chapters.
This book has a heavy emphasis on software engineering best practices and will teach you skills that are useful for any language or framework.
About the Author
Table of Contents
-
Introduction
- Messy data
- Nightmare deploys
- Empower refactoring
- Tests encourage code that doesn’t have side effects
- Identifying code bottlenecks
- Test suites document behavior
- Technologies used
-
Testing Scala with Scalatest
- Writing a simple test
- Directory organization
- build.sbt
- More tests
- Running tests and configuring output
- assertThrows
- assertDoesNotCompile
- Other assertions
- Other test formats
- Test library alternatives
- Testing Spark applications
- Next steps
-
Column Equality Tests
- Custom DataFrame Transformation Refresher
- Spark project setup
- assertColumnEquality with spark-fast-tests
- Conclusion
-
Quieting Test Output
- Customizing test suite output
-
Creating DataFrames for Tests
- toDF
- createDataFrame
- createDF
- Including spark-daria in your projects
- Next steps
-
DataFrame Equality Tests
- Simple example
- assertSmallDataFrameEquality error messages
- Next steps
-
Running Tests
- Running from the SBT console
- Running a single test file
- Running a single test
- Best workflow
-
Approximate Equality
- Difference between double, float and decimal
- When assertColumnEquality falls short
- assertFloatTypeColumnEquality to the rescue
- assertApproximateDataFrameEquality
- Conclusion
-
Testing User Defined Functions
- Creating a UDF
- Testing a UDF
- Check the UDF fails with null input
- The billion dollar mistake
- Verifying test failure in the test suite
- Next steps
-
Testing Spark Column Functions
- Simple example
- How Spark functions handle null
- Important takeaway
- Why print DataFrames from the test suite?
- Next steps
-
Testing Filesystem Reads
- Untestable code
- Setting the path as a param
- Testing with the config pattern
- Elegant testing with dependency injection
- Abstracting custom transformation to a separate function
- Next steps
-
Testing Filesystem Writes
- Simple example
- Rude tests leave garbage behind
- Performance considerations
- Next steps
-
Identifying Bottlenecks
- Let’s find the bottleneck
- Benchmarking individual transformations
- Contrived but representative
- Conclusion
-
Organizing Tests
- Some pure Scala
- Poorly organized Spark tests
- Test suite rules to follow
- Good Spark test organization
- Tests should be descriptive and document behavior
- Quantifying performance difference
- Next steps
-
Test Suite Configuration
- Shuffle partitions
- javaOptions
- Conclusion
-
Testing Aggregations
- groupBy refresher
- groupBy with two columns
- groupBy with filters
- Conclusions
The Leanpub 45-day 100% Happiness Guarantee
Within 45 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms
Do Well. Do Good.
Authors have earned$10,888,441writing, publishing and selling on Leanpub, earning 80% royalties while saving up to 25 million pounds of CO2 and up to 46,000 trees.
Learn more about writing on Leanpub
Free Updates. DRM Free.
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers), EPUB (for phones and tablets) and MOBI (for Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
Learn more about Leanpub's ebook formats and where to read them
Top Books
- #1
Functional event-driven architecture: Powered by Scala 3
Gabriel VolpeExplore the event-driven architecture (EDA) in a purely functional way, mainly powered by Fs2 streams in Scala 3!
Leverage your functional programming skills by designing and writing stateless microservices that scale, powered by stateful message brokers.
- #2
Book of Monads
Alejandro Serrano MenaIn functional programming, at the heart of input/output, failure, state, logic, and much more, lies a powerful abstraction called monad. This book provides a journey from the very first concepts, to the myriad of monads available to programmers, down to the categorical foundations.
- #3
Ansible for DevOps
Jeff GeerlingAnsible is a simple, but powerful, server and configuration management tool. Learn to use Ansible effectively, whether you manage one server—or thousands.
- #4
Practical FP in Scala: A hands-on approach
Gabriel VolpeA practical book aimed for those familiar with functional programming in Scala who are yet not confident about architecting an application from scratch.
Together, we will develop a purely functional application using the best libraries in the Cats ecosystem, while learning about design patterns and best practices.
- #5
Functional Programming Made Easier
Charles ScalfaniA Functional Programming book from beginner to advanced without skipping a single step along the way.
In my 40 years of programming, I've felt that programming books always let me down, especially Functional Programming books. So, I wrote the book I wish I had 5 years ago.
Functional Programming will never be easy, but it can be easier.
- #6
入門 日本語自然言語処理
Masato Hagiwara and Paul O'Leary McCann日本語テキストを処理したい全てのプログラマ・エンジニアの方へ。分かち書きなどの基本から、自然言語生成などの最新の話題までをカバー。動かして学べるコードや、参照文献も付いています。言語学や機械学習の知識が無くても問題ありません。
- #7
Introducing EventStorming
Alberto BrandoliniThe deepest tutorial and explanation about EventStorming, straight from the inventor.
- #8
Visualise, document and explore your software architecture
Simon BrownA short guide to visualising, documenting and exploring your software architecture.
- #9
Introduction to Japanese Natural Language Processing
Masato Hagiwara and Paul O'Leary McCannA thorough guide for programmers working with Japanese text, covering fundamental issues like tokenization and recent research topics like generating natural language texts. Working examples are accompanied by extensive reference to allow problem solving even without a background in Japanese or Machine Learning.
- #10
C++20
Rainer GrimmC++20 is the next big C++ standard after C++11. As C++11 did it, C++20 changes the way we program modern C++. This change is, in particular, due to the big four of C++20: ranges, coroutines, concepts, and modules.
Top Bundles
- #1
Modern C++ Collection
3 Books
Get All about Modern C++C++ Standard Library, including C++20Concurrency with Modern C++, including C++20C++20Each book has about 200 complete code examples. Updates are included. When I update one of the books, you immediately get the updated bundle. You can expect significant updates to each new C++ standard (C++23, C++26, .. ) and also... - #2
Software Architecture for Developers: Volumes 1 & 2 - Technical leadership and communication
2 Books
"Software Architecture for Developers" is a practical and pragmatic guide to modern, lightweight software architecture, specifically aimed at developers. You'll learn:The essence of software architecture.Why the software architecture role should include coding, coaching and collaboration.The things that you really need to think about before... - #3
CCIE Service Provider Ultimate Study Bundle
2 Books
Piotr Jablonski, Lukasz Bromirski, and Nick Russo have joined forces to deliver the only CCIE Service Provider training resource you'll ever need. This bundle contains a detailed and challenging collection of workbook labs, plus an extensively detailed technical reference guide. All of us have earned the CCIE Service Provider certification... - #4
All the Books of The Medical Futurist
6 Books
We put together the most popular books from The Medical Futurist to provide a clear picture about the major trends shaping the future of medicine and healthcare. Digital health technologies, artificial intelligence, the future of 20 medical specialties, big pharma, data privacy, digital health investments and how technology giants such as Amazon... - #5
The Indie Python Extravaganza
5 Books
The Indie Python Extravaganza! A collection of books that will help you to improve your knowledge of the Python programming language one page at a time. Join four indie authors in a journey from the basics of Python to the structure of production-ready systems, going through the core features of the language, some intermediate projects and a... - #7
Abandon Your Job - Leaving the Rat Race with Python
8 Books
Now, get all the great Python and Freelancing books from finxter.com in one single bundle!Coffee Break Python - All Python BasicsCoffee Break Python Workbook - Practice the basics of PythonCoffee Break Python Mastery Workouts - Practice advanced Python conceptsCoffee Break Python Slicing - All about SlicingCoffee Break Python NumPy - Get started... - #10
Pattern-Oriented Memory Forensics and Malware Detection
2 Books
This training bundle for security engineers and researchers, malware and memory forensics analysts includes two accelerated training courses for Windows memory dump analysis using WinDbg. It is also useful for technical support and escalation engineers who analyze memory dumps from complex software environments and need to check for possible...
