Table of Contents
Generator is a new concept introduced to JavaScript in ECMAScript 2015 (or ES6). Generators are the new powerful tools which should be in each JavaScript developer’s toolkit. Generators are mostly used in JavaScript frameworks and libraries. For example, Koa uses generators as middleware. Babel can use generators to transpile async/await
functions. But generators are not commonly used in application code yet. The main reason is that generators are not easy to understand and adopt in day-to-day development.
This book focuses on real day-to-day development scenarios which can benefit from using generators.
Most the code examples in this book are tested on Chrome browser and some of them are tested on NodeJS 6.7.0. These examples should also work on other browsers which support generators. Refer to this page for browser compatibility of generators.
I Generators basics
Before discussing actual usage of generators, we start from the basic concept of generators.
There are two different concepts related to generators.
- Generator function - A special kind of function which generates generator objects.
- Generator object - An instance of generator function.
Execution of generator objects can be suspended and resumed. In JavaScript, we have only limited control over execution of normal functions. Given a function, when it starts execution, by using ()
, apply
or call
, it will run to the end of the execution.
For a simple function sum
shown below, when it’s invoked using sum(1, 2)
, it starts execution and returns value 3
to the caller.
As JavaScript engine execution is single-threaded (not considering web worker here), during the execution of a function, there is no way to stop the execution. So if you accidentally create an infinite loop in your function, the whole application will be blocked.
1. Basic generators
Let’s start with a simple generator function. The difference between a generator function and a normal function declaration is the *
between function
and the function name.
Generator objects can return multiple values when next()
method is invoked. Those values are specified using yield
keyword. In the generator function above, three yield
expressions can generate three values 1
, 2
and 3
when next()
method of a generator object is invoked.
In the code above, invoking the generator function sample
generates a new generator object func
. Execution of generator object func
is initially suspended. When next
method is invoked on the func
object, it starts execution and runs to the first yield
expression and returns the value 1
to the caller. The return value is an object with two properties: value
and done
. value
contains the return value of yield
expression, done
can be used to check if there are more values to get. done
property is false
for the first three invocations of next
method. For the fourth invocation, done
property is set to true
, which means there are no values anymore.
1.1 Suspend & resume execution
The power of generators comes from the ability to suspend and resume execution of generator objects. Each generator object can be viewed as a state machine. Each instance of the same generator function maintains its own state. Invoking next()
on the generator object triggers state transition inside the object, which causes the object runs to the next yield
expression. This continues until no more yield
expressions found.
In the code below, two generator objects func1
and func2
maintain their own internal states. Invoking next()
on one object doesn’t affect the state of the other object.
1.2 Check types of generator functions and generator objects
We can use Object.prototype.toString
to check the types of generator functions and generator objects.
2. Pass values to next()
Let’s start from another simple generator function doMath
. If we just look at the code, we may think that after invoking next()
on the generator object, the value of x
should be 1
, the value of y
should be 11
and the value of z
should be 110
. It’s just simple math, right???
But the actual result doesn’t match what we would expect. As shown in the code below, the values are 1
, NaN
and NaN
.
The key to understanding the actual result is that value passed to next()
invocation is the actually used value of last yield
expression. Since we didn’t pass any argument when invoking next()
, so the value of each yield
expression is actually undefined
.
For the first next()
invocation, there is no last yield
expression, so the value is actually ignored. For the second next()
invocation, value of last yield
expression, i.e. yield 1
is set to undefined
, which sets x
to undefined
, then sets the result of yield x + 10
to NaN
. For the third next()
invocation, value of last yield
expression, i.e. yield x + 10
is set to undefined
, which sets y
to undefined
, then sets the result of yield y * 10
to NaN
.
Now we can try to pass a value when invoking next()
method on a generator object. In the code below, the second next()
invocation func.next(1)
passes 1
to the generator object, so value 1
is set as the value of yield 1
, which sets x
to 1
, then the result of this next()
will be 11
. For the third next()
invocation func.next(2)
, 2
is passed as the value of yield x + 10
, which sets y
to 2
, then the result of this next()
will be 20
.
3. return
in generators
In the generator function, we can also use return
statement. The returned value is also passed to the caller of a generator object’s next()
method. return
also finishes execution of generator object, i.e. done
property is set to true
. In the code below, the return value of second next(1)
invocation is the value of return
statement, i.e. x + 2
.
3.1 Infinite values
It’s possible for a generator object to generate an infinite number of values, i.e. done
property is always false
. For example, we can create a generator which generates infinite integer numbers starting from 0
. In this case, we can use return
to finish generator objects.
In the code below, loop
keeps generating incremental values in a while
loop. When a truthy value is passed to next()
as the value of shouldExit
, the last value is returned and generator object is finished.
As shown in the code below, three values are generated using next()
. The forth next(true)
invocation finishes the generator object func
.
4. Iterators & generators
From all the generators code above, you may wonder why we should use next()
to get values from the generator objects and deal with the nonintuitive return value format {value: 1, done: false}
. Meet iterators.
4.1 Iterators
Iterators are no strangers to developers. They already exist in different programming languages with similar names, e.g. Java Iterator, Ruby Enumerator and Python Iterator Types. Iterators can be used to iterate over items in a collection. Iterators maintain their own states regarding the current position in the target collection.
An iterator in ES6 is just an object which provides a next()
method to get next item in the current iteration. next()
method should return an object with two properties: value
and done
. So generator functions are actually factories of iterators.
4.2 Iterables
Iterables are objects which have property @@iterator
. The value of @@iterator
property is a function that returns an Iterator object.
A generator object conforms to both the Iterator and Iterable interfaces.
4.3 Iterate generator objects
As generators are iterable, we can use other ES6 language features to interact with generator objects easily. Following examples use values
generator function shown below.
for-of
loops
We can use for-of
loops to easily iterate all the values in a generator object.
Spread operator
Generator objects can also be used with spread operator.
Work with new collection types
Generator objects can be used to create new collection objects, e.g. Set
, WeakSet
, Map
and WeakMap
.
II Advanced generators
After introducing basic concepts of generators, we are now looking into more advanced features of generators.
The code in this chapter uses following debug
function to log values to the console.
5. Arguments of generator functions
Like other normal functions, generator functions can take arguments. These arguments can be used in yield
expressions inside the generator functions.
In the code below, seq
is a generator function with arguments start
and number
. start
means the start number of generated values and number
means the total number of generated values.
6. return
method
A generator object has a return
method to return given value and finish the generator. This behavior is similar with using return
statement inside of a generator.
Given the same values
generator function shown below,
We can see how invoking return
method finishes the generator object. The first next()
invocation returns the first value 'a'
, then func.return('d')
returns value 'd'
and finishes the generator, i.e. done
property is set to true
.
return
method can be invoked multiple times. Each invocation returns the value passed to return()
method.
7. throw
method
A generator object also has a throw
method to pass a value to it and trigger an exception to throw inside of the generator object. Both throw
and next
methods can send values to generator objects and change their behaviors. A value passed using next
is treated as the result of last yield
expression, but a value passed using throw
is treated as replacing last yield
expression with a throw
statement.
In the code below, when passing hello
to the generator object using throw('hello')
, an uncaught error is thrown and the generator object is finished. When func.throw('hello')
is invoked, the last yield
expression yield x + 1
is replaced with throw 'hello'
. Since the thrown object is not caught, it’s propagated to the JavaScript engine.
Although it’s possible to pass any types of values to throw()
, it’s recommended to pass Error
objects for better debugging, e.g. throw(new Error('boom!'))
.
We can use try-catch
in the generator function to handle errors. In the code below, when func.throw(new Error('boom!'))
is invoked, last yield
expression yield 2
is replaced with throw new Error('boom!')
. The thrown object is caught by try-catch
. So the execution continues until the next yield
expression yield 3
.
If the value passed by throw()
is caught and handled by the generator object, it can continue to generate all remaining values. Otherwise, it will finish with a uncaught error.
8. yield*
So far aforementioned generator objects only generate a single value using yield
expression one at a time. We can also use a yield*
expression to generate a sequence of values. When a yield*
expression is encountered, sequence generation of current generator object is delegated to another generator object or iterable object.
8.1 yield*
& iterable objects
In the code below, generator function oneToThree
uses yield* [1, 2, 3]
to generate three values: 1
, 2
and 3
, which has the same result as generator function sample
in basic generators. Using yield*
expression is more concise and easier to read.
We can use multiple yield*
expressions in a generator function, then values from each yield*
expression are generated in order.
8.2 yield*
& generator objects
We can also use other generator objects in yield*
expressions.
8.3 Value of yield*
yield*
is also an expression, so it’s evaluated to a value. The value of yield*
expression depends on its target, i.e. the expression after yield*
. The value is the last value generated by the iterable object or generator object, i.e. the value
property with done
set to true
.
If yield*
is used with iterable objects, then the evaluated value is always undefined
, because the last generated value is always {value: undefined, done: true}
.
If yield*
is used with generator objects, we can control the last generated value using return
inside of the generator functions.
9. Nested yield
and yield*
We can nest yield
and yield*
to create complex values generation.
9.1 Nested yield
In the code below, the inner yield
expression generates value 1
first, then the middle yield
expression generates value of yield 1
- undefined
, then the outer yield
expression generates value of yield yield 1
- undefined
.
9.2 Nested yield
and yield*
In the code below, generator oneToThree
first generates three values 1
, 2
and 3
, then its value undefined
is generated by yield
expression.
10. co
Generator functions can also be used to control code execution flow. By using yield
expressions, we can control when the execution of a generator object should be suspended. When the execution of a generator object is suspended, other code can have the chance to run and choose the best time to resume the execution. yield*
expressions allow the delegation to other generator objects or iterable objects, which can create complicated nested or recursive execution flows.
Generator functions are most useful when combining with Promise
s. As described in MDN,
The Promise object is used for asynchronous computations. A Promise represents a value which may be available now, or in the future, or never.
If the value of a yield
expression is a Promise
object, then we can suspend the execution of the generator object when waiting for the Promise
to be resolved. When the Promise
is fulfilled, we can resume the execution of the generator object with the fulfilled value as the value of the yield
expression. Otherwise, we can finish the generator with the rejected error.
To support this kind of scenarios, we need to use the library co. In the code below, timeoutToPromise
is a helper method that creates a Promise
object using setTimeout
. Generator function calculate
uses yield
expression and the Promise
object created by timeoutToPromise
. co(calculate, 1, 2)
turns the generator function calculate
into a Promise
object.
Below is an example of using co
with generator functions which have yield
expressions with other generator objects. value
is a generator function which takes the argument v
as the seed of generating two random values v1
and v2
. yield value(1)
in calculate
uses a generator object value(1)
as the target of yield
expression.
11. regenerator
If generator functions are not supported on the target platform, we can use regenerator to transpile generator functions into ES5. Babel also has a transform-regenerator plugin to perform the transformation. If you use Babel preset ES2015, then this plugin is already included.
This plugin can transform code using generators
into code using regenerator.
You can try it online.
III Real-world usages
We are going to see how generators in real-world projects.
12. Koa
Koa is next generation web framework for NodeJS. Its powerful middleware architecture is built on top of generators. We are going to see how Koa uses generators.
12.1 Koa basics
Koa is very easy to configure and use. Each application creates different middleware to handle requests and generate responses. Each middleware is a generator function and registered using use()
of Koa application. Middleware are processed in a chain with the same order as they are registered. Each middleware can access context information using this
, e.g. request
, response
, method
and url
.
The code below is a simple Koa application. It registers two middleware generator functions, the first one is used to log request processing time and the second one is used to set the response body to Hello World
.
Each middleware generator function can take an extra argument which represents the next middleware in the chain. If a middleware generator function needs to intercept execution of downstream middleware in the chain, it can perform certain tasks first, then call yield
to delegate to other middleware, then perform other tasks after downstream middleware finish. The logging middleware generator function in the code above demonstrates this pattern. It records start time when a request comes in, then it calls yield next
for delegation, finally it records the finish time and calculates the duration.
After accessing the http://localhost:3000
, the console log looks like below:
12.2 koa-compose
koa-compose
is the small library which does the composition of middleware generator functions. Its source code is very simple, only 29 sloc. compose
is the main method to compose middleware. The argument middleware
is an array of middleware generator functions in the order of registration. The return value of compose
method is a generator function with argument next
. next
is an optional generator function which is the last middleware in the chain.
Let’s go through the generator function code line by line. The first line if (!next) next = noop();
sets next
to a do-nothing generator function noop
if it’s null
. i
is the loop variable for array middleware
starting from the last middleware in the array. In the while
loop, the generator function of each middleware is invoked with the current value of next
as the argument, the returned generator object is set as the new value of next
. Then yield*
is used to delegate to final next
generator object.
We’ll see how middleware are used in the sample application of Koa basics. The middleware
array contains two generator functions, log
and setBody
. In the while
loop, generator function setBody
is invoked first with the argument next
set to noop
and next
is set to the generator object of setBody
. Then generator function log
is invoked with the argument next
set to the generator object of setBody
and next
is set to the generator object of log
. The last yield* next
expression delegates to the generator object of log
.
The returned generator function of compose
is turned into a regular function that returns a Promise
using co.wrap
from co. The wrapped function is the actual request handler. When a request comes in, the generator object of log
starts execution first and runs until the yield next
, so the start time is recorded. next
is a generator object of setBody
, invoking yield next
triggers the execution of setBody
and set the response body. Finally, the generator object of log
resumes execution and calculate the duration.
13. Babel
Babel is a JavaScript compiler which allows developers to use future JavaScript features. Babel has different plugins to transform JavaScript code written with the latest standards into a version which is supported on today’s platforms.
13.1 Transform async/await
Babel has a async to generator plugin which transforms async
functions into generator functions. We’ll use a simple NodeJS application to demonstrate the usage of this Babel plugin.
The code below shows the .babelrc
file.
Given JavaScript code shown below,
After applying the plugin, the output is shown as below.
You can also view the transformed result online.
The transformation is straightforward and relies on a helper method _asyncToGenerator
. async
function is transformed into generator function and await
is transformed into yield
. The _asyncToGenerator
helper is responsible for transforming generator functions into a regular function that returns a Promise
.
From the source code of asyncToGenerator
, we can see that it transforms a generator function into a Promise
chain.
IV Usage scenarios
In this chapter, we are going to see some common usage scenarios for generators.
14. Sequence generation
Generator functions are very useful when generating complex sequences. We can encapsulate the generation logic in the function and shield the consumer from internal details.
In the code below, generator function numbers
has a complicated logic about generating values in the sequence.
For more complicated scenarios, we can also use yield*
to combine sequences. Suppose we have a system which stores users information in both file system and database, we can use following code to return a sequence of all users.
15. Fail-fast task queue
We can use generator functions to create a simple fail-fast task queue and avoid recursive calls. The task queue is fail-fast, so subsequent tasks shouldn’t be executed when a task failed.
We use the following code in file createTask.js
to create tasks using setTimeout
and Promise
. The task fails when value
is greater than or equals to 5
.
The values used for testing are simple numbers.
We can implement the task queue using recursive calls.
We can also implement it using generator functions and co
.