![](/assets/ce75d53f5bec2ec1a47c2452c115659d.png)
Practical Machine Learning with Transformers
Practical Machine Learning with Transformers
About the Book
It has been possible for many decades now to achieve amazing feats with Machine Learning—there are almost too many to list that have contributed to making our lives better (or at the very least easier).
With the introduction of the transformer architecture and in particular a technique called _attention_ it has become possible though to develop models that can do a number of these tasks all at once, and discuss the results with you. You wouldn't have gotten very far trying to ask a spam filter what it was thinking (to use the term loosely), but now models like ChatGPT can even help you write your prompts that you use to get these results.
Machine Learning has always had quite a high bar for entry. Taking just Artificial Neural Networks as one sub-field, historically you would generally have needed to have good knowledge of calculus, linear algebra and regression analysis, in addition to programming. Then, to solve your problem, you would be quite likely to be building your training data from scratch. You would amass enough of it, learn how to train a model and repeat this process many times. Hopefully at the end of this process you would have something that you could put to good practical use.
In terms of not starting from scratch, there were some options for transfer learning, but this still required considerable effort, knowledge and accumulated experience to get right. Today this is still true, but I would argue that the bar has been very usefully lowered in terms of allowing anyone who persists at it to develop their own practical, useful models by standing on top of two recently-formed giants: the transformer architecture and its attention mechanism, and the myriad base models that are currently available as published by everyone from large companies down to individuals.
Now you can take a base model which may have cost some millions of dollars to train, and fine-tune it for some task, usefully leveraging the incredible depth and breadth of context and encoded knowledge that some of these models hold. In a number of cases the licensing is also sufficiently permissive to use the results commercially, meaning that in addition to research and hobby use you might actually build a business around your work in this field, if you wish to do so.
The purpose of this book is to lay out how we can solve practical problems using transformers and leveraging the incredible base models that we can access via the HuggingFace hub. A key aim here is to start with the practical and delve into the theoretical as an optional step—you can successfully train models and solve problems without knowing the underlying maths well. Not knowing the maths well can certainly hinder your progression in certain directions but solving practical problems using the transformers and pytorch libraries will always be open to you.
Being a Leanpub publication, this book is a work in progress and you are invited to ask questions and make requests. If some part is unclear then please let me know and I'll be happy to expand that section. If you find what looks to be an error then I will endeavour to fix it.
We'll start by focussing on natural language problems like classification (is the sentiment of this text positive, negative, or neutral?), question answering, translation and autocorrect. As transformers can also produce incredible results on vision tasks then we will look to explore that area also.
Table of Contents
- About this book
- How to study the material
- The Transformer Architecture
- Encoder / Decoder
- Decoder models vs Sequence-to-sequence models
- Using Transformers
- The HuggingFace pipeline
- Handling inputs
- Fine-tuning a Transformer
- Preparing the data
- An end-to-end example
- Loss Functions
- Binary cross entropy (log loss)
- Categorical cross entropy
- Triplet loss
- Background Material
- What are logits?
- About tensors
- Logistic regression
- The Softmax function
- About this book
The Leanpub 60 Day 100% Happiness Guarantee
Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
Now, this is technically risky for us, since you'll have the book or course files either way. But we're so confident in our products and services, and in our authors and readers, that we're happy to offer a full money back guarantee for everything we sell.
You can only find out how good something is by trying it, and because of our 100% money back guarantee there's literally no risk to do so!
So, there's no reason not to click the Add to Cart button, is there?
See full terms...
Earn $8 on a $10 Purchase, and $16 on a $20 Purchase
We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.
(Yes, some authors have already earned much more than that on Leanpub.)
In fact, authors have earnedover $13 millionwriting, publishing and selling on Leanpub.
Learn more about writing on Leanpub
Free Updates. DRM Free.
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
Learn more about Leanpub's ebook formats and where to read them