Web Scraping
Web scraping, the automated extraction of content from web pages, is a powerful technique for research, data analysis, and building intelligent applications. Before we dive into the code, it is important to discuss responsible and legal web scraping practices. Always read and respect a site’s robots.txt file and its terms of service before scraping. Limit the rate of your requests so you do not place undue burden on web servers; a delay of a second or two between requests is good practice. Prefer using public APIs when they are available, and only scrape content that is publicly accessible. Be aware that some jurisdictions have laws (such as the Computer Fraud and Abuse Act in the United States or the GDPR in Europe) that restrict automated data collection. When in doubt, contact the site owner for permission. In my own work I routinely email web site owners to explain how I plan to use their data, and this approach has served me well. If you treat web scraping the way you would treat visiting someone’s home, politely, and with respect for their resources, you will stay on solid ethical ground.
In this chapter we build three example scripts, found in the src/webscraping/ directory: html-headers.lisp, page-text.lisp, and page-markdown.lisp. They demonstrate a progression from simple HTML inspection to full content extraction. All three share a helper file, utils.lisp, which each example loads first. The examples use the Drakma HTTP client to fetch pages and the Plump HTML parser to build a DOM tree. The header example also uses CLSS, a CSS-selector engine for querying parsed HTML. The cleanup code uses CL-PPCRE, a fast regular-expression library. Running any script from the command line loads Quicklisp and its dependencies through utils.lisp, so no setup is required.
Shared Utilities: utils.lisp
1 ;;; Shared helpers for the web-scraping examples in this directory.
2 ;;; Each example loads this file first:
3 ;;; (load (merge-pathnames #P"utils.lisp" (or *load-pathname* #P"")))
4
5 (ql:quickload '(:drakma :plump :cl-ppcre) :silent t)
6
7 ;; DEFPARAMETER rather than DEFCONSTANT so re-loading these examples in one
8 ;; core does not signal a constant-redefinition error.
9 (defparameter +whitespace+ '(#\Space #\Tab #\Newline #\Return)
10 "Characters trimmed from the ends of extracted text.")
11
12 (defun trim (string)
13 (string-trim +whitespace+ string))
14
15 (defun whitespace-only-p (string)
16 (every (lambda (char) (member char +whitespace+)) string))
17
18 (defun kw (name)
19 "Return the keyword symbol for NAME, e.g. \"H1\" => :H1, or NIL."
20 (when name (find-symbol (string-upcase name) :keyword)))
21
22 (defun text-node-p (node)
23 (typep node 'plump:text-node))
24
25 (defun element-node-p (node)
26 (typep node 'plump:element))
27
28 (defun fetch-html (url)
29 "Fetch URL, returning the body as a string, or NIL on failure.
30 Drakma's default User-Agent is blocked by many sites, so we set our own.
31 When a server omits the charset, Drakma decodes the body with
32 *DRAKMA-DEFAULT-EXTERNAL-FORMAT* (Latin-1), which garbles UTF-8 text."
33 (handler-case
34 (drakma:http-request url
35 :user-agent "Mozilla/5.0 (compatible; CL-scraping-example)"
36 :connection-timeout 10
37 :redirect 5)
38 (error (c)
39 (format *error-output* "~&Could not fetch ~A: ~A~%" url c)
40 nil)))
41
42 (defun node-text (node)
43 "Text of NODE's subtree, skipping comments.
44 Do not use PLUMP:TEXT on an element for this: its deep walk also collects
45 comment contents, so hidden text leaks into the output."
46 (with-output-to-string (s)
47 (labels ((walk (node)
48 (cond
49 ((text-node-p node) (write-string (plump:text node) s))
50 ((typep node 'plump:nesting-node)
51 (loop for child across (plump:children node)
52 do (walk child))))))
53 (walk node))))
54
55 ;; NOTE ON REGEX STRINGS: Common Lisp string tokens have no Python-style
56 ;; \n or \t escapes. Inside "...", a backslash merely makes the next
57 ;; character literal, so "\n" is the single character #\n (code 110), not
58 ;; a newline, and "[ \t]" is a class matching space and the letter t.
59 ;; Regex escapes therefore need double backslashes ("\\s+", "[ \\t]+"),
60 ;; and real newline characters must be built from #\Newline (string
61 ;; #\Newline) or ~% (format nil "~A~%") instead of "\n".
62
63 (defun normalize-spaces (string)
64 "Collapse every run of whitespace, newlines included, into a single space.
65 Block tags supply the newlines during assembly, so marker tokens (such as
66 the old __H1__ hack) are not needed to protect them from CLEAN-WHITESPACE."
67 (cl-ppcre:regex-replace-all "\\s+" string " "))
68
69 (defun raw-text (node)
70 "Text of NODE's subtree that collapses spaces/tabs but keeps newlines,
71 and skips comments. For <pre>/<code>, where line structure is content."
72 (cl-ppcre:regex-replace-all "[ \\t]+" (node-text node) " "))
73
74 (defun escape-markdown (string)
75 "Escape characters that Markdown would otherwise interpret as syntax."
76 (with-output-to-string (s)
77 (loop for char across string
78 do (when (find char "*_[]`\\") (write-char #\\ s))
79 (write-char char s))))
80
81 (defun clean-whitespace (text &key strip-indent)
82 "Normalize newlines, drop whitespace-only lines, and collapse runs of
83 blank lines to one. With STRIP-INDENT also remove leading spaces per line
84 (safe for plain text; Markdown needs them for nested-list indentation).
85 Text nodes are assumed to be normalized already by NORMALIZE-SPACES, so no
86 space-collapse pass is needed here."
87 ;; Build replacement strings from #\Newline; see the escape warning above.
88 (let ((two-newlines (format nil "~A~A" (string #\Newline) (string #\Newline))))
89 (when strip-indent
90 (setf text (cl-ppcre:regex-replace-all "(?m)^[ \\t]+" text "")))
91 (setf text (cl-ppcre:regex-replace-all "(?m)^[ \\t]+$" text ""))
92 (setf text (cl-ppcre:regex-replace-all "(?m)[ \\t]+$" text ""))
93 ;; Match CRLF and lone CR as single units so Windows-sourced line
94 ;; endings collapse the same way Unix ones do.
95 (setf text (cl-ppcre:regex-replace-all "(?:\\r\\n|\\r|\\n){3,}" text two-newlines))
96 (trim text)))
A few decisions in this file are worth calling out.
fetch-html sets a custom User-Agent because Drakma’s default is blocked by many sites. It catches errors and returns NIL so one failed page does not stop a larger job. Note that when a server omits the charset from its Content-Type header, Drakma decodes the body with Latin-1, which garbles UTF-8 text; see the practice problems at the end of this chapter for one way to recover.
node-text walks the subtree by hand instead of calling plump:text on an element. The deep walk inside plump:text also collects comment contents, so text hidden behind HTML comments leaks into the output.
The comment block above normalize-spaces records a trap that catches many people who port regex code to Common Lisp. Lisp string tokens have no \n or \t escapes, so regex strings need double backslashes ("\\s+") and real newlines must come from #\Newline or ~%.
Earlier versions of these examples wrapped headings in marker tokens like __H1__ to protect their spacing from the whitespace-collapse pass. That trick is gone. Each text node now goes through normalize-spaces, which collapses all whitespace including newlines, and the block tags themselves add the newlines back during assembly. clean-whitespace therefore needs no markers.
Extracting HTML Headers
Our first example, html-headers.lisp, is the simplest: fetch a web page and print the text content of every heading tag, h1 through h6. This is useful for quickly surveying the structure of a page, what sections it contains and how the content is organized.
1 (unless (fboundp 'fetch-html)
2 (load (merge-pathnames #P"utils.lisp" (or *load-pathname* #P""))))
3 (ql:quickload :clss :silent t)
4
5 (defun fetch-and-print-headers (&optional (url "https://markwatson.com"))
6 "Fetch URL and print the text of h1-h6 tags in document order.
7 One comma-separated selector beats six round trips through the DOM, and
8 NODE-TEXT keeps HTML comments out of the printed headers."
9 (format t "Fetching ~A...~%" url)
10 (let ((html (fetch-html url)))
11 (when html
12 (let ((nodes (clss:select "h1,h2,h3,h4,h5,h6" (plump:parse html))))
13 (if (zerop (length nodes))
14 (format t "No h1-h6 headers found.~%")
15 (loop for node across nodes
16 for text = (trim (normalize-spaces (node-text node)))
17 unless (string= text "")
18 do (format t " [~A] ~A~%" (plump:tag-name node) text)))))))
19
20 (fetch-and-print-headers)
The flow is straightforward. fetch-html downloads the raw HTML as a string, plump:parse turns it into a DOM tree, and one clss:select call with the comma-separated selector "h1,h2,h3,h4,h5,h6" finds every heading in a single pass. CLSS returns nodes in document order, so the output follows the order in which the headings appear on the page. Each heading prints with its tag name in brackets, which preserves the hierarchy in the flat output.
The (unless (fboundp 'fetch-html) ...) guard appears in all three examples. It loads utils.lisp relative to the file being loaded, so the script works from any working directory, and it skips the load when you have already loaded the utilities in your running image.
You can run this example from the command line:
1 sbcl --load html-headers.lisp --eval "(sb-ext:exit)"
Here is a snippet of the output when run against my personal site:
1 Fetching https://markwatson.com...
2 [h1] Mark Watson
3 [h2] Read My Books — Many for Free
4 [h2] Connect & Interests
5 [h3] Loving Lisp Is Being Rewritten
6 [h3] Writings & Blogs
7 [h3] Open Source
8 [h3] Consulting
9 [h4] Additional eBooks on Leanpub
10 [h4] Traditional Publications (Springer-Verlag, McGraw-Hill, Morgan Kaufmann)
This tiny script is already quite useful. You could extend it to crawl a list of URLs and build a table of contents for an entire site.
Extracting Page Content as Plain Text
Our second example, page-text.lisp, goes beyond headers and extracts the full readable text of a web page, stripping out scripts, styles, navigation, footers, and other boilerplate. The result is clean plain text suitable for natural language processing, summarization, or feeding into an LLM.
1 (unless (fboundp 'fetch-html)
2 (load (merge-pathnames #P"utils.lisp" (or *load-pathname* #P""))))
3
4 (defun text-boilerplate-p (tag)
5 "Elements dropped for plain-text extraction. The Markdown example keeps
6 nav/header/footer because their link lists carry meaning; here we want only
7 the article body. Named differently from MD-BOILERPLATE-P so both examples
8 can be loaded in one core without clobbering each other."
9 (member tag '("script" "style" "head" "nav" "header" "footer" "iframe" "noscript")
10 :test #'string-equal))
11
12 (defun get-element-spacing (tag text)
13 "Wrap TEXT in newlines appropriate for TAG. CLEAN-WHITESPACE collapses
14 over-long runs afterward, so each block only needs enough newlines to
15 separate it from its neighbors."
16 (case (kw tag)
17 ((:h1 :h2 :h3 :h4 :h5 :h6 :p :blockquote :ul :ol :pre)
18 (format nil "~%~A~%~%" text))
19 ((:li :tr)
20 (format nil "~A~%" text))
21 (:br
22 (format nil "~%"))
23 ((:div :article :section :aside :main)
24 (if (and (plusp (length text))
25 (char= #\Newline (char text (1- (length text)))))
26 text
27 (format nil "~A~%" text)))
28 (t text)))
29
30 (defun children-text (node)
31 (with-output-to-string (s)
32 (loop for child across (plump:children node)
33 do (write-string (get-clean-text child) s))))
34
35 (defun get-clean-text (node)
36 "Recursively collect text from NODE, skipping boilerplate tags and
37 inserting line breaks for block tags. <pre>/<code> keep their internal
38 newlines; all other text is whitespace-normalized at the node level."
39 (cond
40 ((text-node-p node)
41 (normalize-spaces (plump:text node)))
42 ((element-node-p node)
43 (let ((tag (plump:tag-name node)))
44 (if (text-boilerplate-p tag)
45 ""
46 (let ((text (if (member (kw tag) '(:pre :code))
47 (raw-text node)
48 (children-text node))))
49 (get-element-spacing tag text)))))
50 ((typep node 'plump:nesting-node)
51 (children-text node))
52 (t "")))
53
54 (defun fetch-and-print-text (&optional (url "https://markwatson.com"))
55 "Fetch URL and print cleaned-up text content."
56 (format t "Fetching ~A...~%" url)
57 (let ((html (fetch-html url)))
58 (when html
59 (format t "~A~%"
60 (clean-whitespace (get-clean-text (plump:parse html))
61 :strip-indent t)))))
62
63 (fetch-and-print-text)
The text-boilerplate-p function filters out tags that contain non-content material: script, style, head, nav, header, footer, iframe, and noscript. get-element-spacing maps each HTML element to the appropriate whitespace treatment: headings, paragraphs, and lists get blank lines around them, list items and table rows get one newline after them, and container elements like div get a newline only when their text does not already end with one.
get-clean-text walks the tree recursively. It calls normalize-spaces on each text node, which collapses every run of whitespace, including source-formatting newlines inside a paragraph, into a single space. Block tags then supply the real newlines. The kw helper converts a tag name to a keyword so the dispatch code can use case instead of long chains of string-equal tests. The one exception to the collapse-everything rule: <pre> and <code> content goes through raw-text, which keeps newlines because line structure is part of the content of those elements.
The plain-text version wants only the article body, so it drops nav, header, and footer. The Markdown example in the next section keeps them. Because both examples can be loaded into the same Lisp image, the two boilerplate predicates carry different names, text-boilerplate-p and md-boilerplate-p, so neither clobbers the other.
The final call passes :strip-indent t to clean-whitespace, which removes leading spaces from each line. Those spaces carry no meaning in plain text.
You can run this example from the command line:
1 sbcl --load page-text.lisp --eval "(sb-ext:exit)"
Sample output (truncated):
1 Fetching https://markwatson.com...
2 Mark Watson
3
4 AI Generalist
5
6 Self-funded open source Artificial Intelligence researcher and author. Engineering AI systems, knowledge graphs, and distributed architectures for organizations like Google, Mind AI, Capital One, Disney, SAIC, Baby List, Olive and CompassLabs.
7
8 40+ Years in AI
9 20+ Books Published
10 55 US Patents
11
12 Explore My Books Connect with Me
13
14 Work Experience
15 Mostly AI projects since 1985. Large scale distributed systems for DARPA and PacBell. Knowledge Graphs at Google. Master Software Engineer and managed an AI team at Capital One. 20 years in the defense industry at SAIC/Leidos.
16
17 Books
18 Read My Books — Many for Free
19
20 All of my recent books are available on Leanpub. Most can be read online at no cost.
Converting a Web Page to Markdown
Our third and most complete example, page-markdown.lisp, converts a full web page into well-formed Markdown. This is useful for feeding web content into large language models, which process Markdown far more effectively than raw HTML.
1 (unless (fboundp 'fetch-html)
2 (load (merge-pathnames #P"utils.lisp" (or *load-pathname* #P""))))
3
4 (defun md-boilerplate-p (tag)
5 "Elements removed before Markdown conversion. Unlike page-text.lisp,
6 nav/header/footer are kept: their link lists are often the point of the
7 conversion. Named apart from TEXT-BOILERPLATE-P so both examples can be
8 loaded in one core without clobbering each other."
9 (member (kw tag) '(:script :style :head :noscript :iframe) :test #'eq))
10
11 (defun md-heading (level inner)
12 (format nil "~%~A ~A~%~%"
13 (make-string level :initial-element #\#) (trim inner)))
14
15 (defun whitespace-child-p (child)
16 "Between <li> tags, text nodes are source indentation, not content."
17 (and (text-node-p child) (whitespace-only-p (plump:text child))))
18
19 (defun html-to-markdown (node &optional (depth 0) (ordered nil) (index 1))
20 "Recursively convert the plump HTML node NODE into Markdown.
21 DEPTH counts enclosing lists (nested items indent two spaces per level),
22 ORDERED is true inside an <ol>, and INDEX is an <li>'s position within its
23 list. Text nodes are Markdown-escaped and whitespace-normalized up front;
24 block tags contribute the newlines, so CLEAN-WHITESPACE needs no marker
25 tokens to protect spacing."
26 (cond
27 ((text-node-p node)
28 (normalize-spaces (escape-markdown (plump:text node))))
29 ((element-node-p node)
30 (let ((tag (kw (plump:tag-name node))))
31 (cond
32 ((md-boilerplate-p tag) "")
33 ((member tag '(:ul :ol) :test #'eq)
34 (with-output-to-string (s)
35 (let ((item 0))
36 (loop for child across (plump:children node)
37 unless (whitespace-child-p child)
38 do (incf item)
39 (write-string (html-to-markdown child (1+ depth)
40 (eq tag :ol) item)
41 s)))))
42 (t
43 (let ((inner (if (member tag '(:pre :code) :test #'eq)
44 (raw-text node)
45 (with-output-to-string (s)
46 (loop for child across (plump:children node)
47 do (write-string (html-to-markdown child depth
48 ordered index)
49 s))))))
50 (case tag
51 (:h1 (md-heading 1 inner))
52 (:h2 (md-heading 2 inner))
53 (:h3 (md-heading 3 inner))
54 (:h4 (md-heading 4 inner))
55 (:h5 (md-heading 5 inner))
56 (:h6 (md-heading 6 inner))
57 ((:p :blockquote)
58 (format nil "~%~A~%~%" (trim inner)))
59 (:br
60 (format nil "~%~A" inner))
61 ((:strong :b)
62 (format nil "**~A**" inner))
63 ((:em :i)
64 (format nil "*~A*" inner))
65 (:code
66 (format nil "`~A`" (trim inner)))
67 (:pre
68 (format nil "~%~%```~%~A~%```~%~%" (trim inner)))
69 (:a
70 (let ((href (plump:attribute node "href")))
71 (if (and href (plusp (length (trim inner))))
72 (format nil "[~A](~A)" (trim inner) href)
73 inner)))
74 (:img
75 (let ((src (plump:attribute node "src"))
76 (alt (escape-markdown (or (plump:attribute node "alt") ""))))
77 (if src
78 (format nil "" alt src)
79 "")))
80 (:li
81 (let ((indent (make-string (* 2 (max 0 (1- depth)))
82 :initial-element #\Space))
83 (marker (if ordered (format nil "~D. " index) "* ")))
84 (format nil "~A~A~A~%" indent marker (trim inner))))
85 ((:div :article :section :aside :main :tr :table :td :th)
86 (format nil "~%~A~%~%" inner))
87 (t inner)))))))
88 ((typep node 'plump:nesting-node)
89 (with-output-to-string (s)
90 (loop for child across (plump:children node)
91 do (write-string (html-to-markdown child depth) s))))
92 (t "")))
93
94 (defun fetch-and-print-markdown (&optional (url "https://markwatson.com"))
95 "Fetch URL and print the content converted to Markdown."
96 (format t "Fetching ~A...~%" url)
97 (let ((html (fetch-html url)))
98 (when html
99 (format t "~A~%"
100 (clean-whitespace (html-to-markdown (plump:parse html)))))))
101
102 (fetch-and-print-markdown)
html-to-markdown walks the DOM tree recursively. For each element node it first converts all children to Markdown (the inner string), then wraps the result in the appropriate Markdown syntax based on the tag: # prefixes for headings, **...** for bold, *...* for italic, [text](url) for links,  for images, * for list items, and triple backticks for code blocks. md-heading builds the heading prefix with make-string, so one helper serves all six heading levels.
Three details separate this converter from a naive tag-to-syntax mapping.
First, html-to-markdown carries three extra arguments: depth counts enclosing lists so nested items indent two spaces per level, ordered records whether the current list is an <ol>, and index is an item’s position in its list, which produces the 1., 2., 3. markers of ordered lists. The :ul/:ol branch increments a counter for each non-whitespace child and passes that count down as the item’s index. whitespace-child-p skips the text nodes that sit between <li> tags in the source HTML; they are indentation, not content, and counting them would offset every list number.
Second, every text node passes through escape-markdown, which backslash-escapes characters like *, _, and brackets. Without this step, a page that sells “20% off limited time” would turn part of its own text into Markdown emphasis.
Third, this example keeps nav, header, and footer elements that the plain-text version drops, because on many sites the link lists in those regions are the point of the conversion. It also differs in one cleanup setting: fetch-and-print-markdown calls clean-whitespace without :strip-indent, because leading spaces are how Markdown represents nested list indentation.
You can run this example from the command line:
1 sbcl --load page-markdown.lisp --eval "(sb-ext:exit)"
Sample output (truncated):
1 Fetching https://markwatson.com...
2 [Mark Watson](#) * [Books](#books)
3 * [Connect](#connect)
4 * [Flagstaff Arizona](/flagstaff.html)
5 * [Blogspot](https://mark-watson.blogspot.com/)
6 * [Substack](https://marklwatson.substack.com)
7 * [GitHub](https://github.com/mark-watson)
8 * [Leanpub](https://leanpub.com/u/markwatson)
9
10 
11 # Mark Watson
12
13 ****AI Generalist****
14
15 Self-funded open source Artificial Intelligence researcher and author. Engineering AI systems, knowledge graphs, and distributed architectures for organizations like **Google**, **Mind AI**, **Capital One**, **Disney**, **SAIC**, **Baby List**, **Olive** and **CompassLabs**.
Wrap Up
The scripts in this chapter form a practical toolkit for extracting information from the web using Common Lisp. The header extractor gives you a quick structural overview, the plain-text extractor yields clean readable content, and the Markdown converter produces richly formatted output ideal for downstream processing. The shared utils.lisp file shows how to factor out the concerns that every scraper faces: fetching with a proper User-Agent, skipping HTML comments, and normalizing whitespace.
Here are some project ideas that build on this web scraping code:
- Build a personal knowledge base. Scrape articles and blog posts you read frequently, convert them to Markdown, and store them in a local file system or database for full-text search. This is especially powerful when combined with the embedding and vector search techniques covered in later chapters.
- Create a site-structure analyzer. Extend the header extraction script to crawl an entire site (following internal links) and build a hierarchical table of contents. This is invaluable for auditing large documentation sites or wikis.
- Feed web content to an LLM. Use the Markdown converter to scrape a page and pass the cleaned output directly to an LLM API (such as the OpenAI, Ollama, or Gemini interfaces covered elsewhere in this book) for summarization, question answering, or translation.
- Monitor pages for changes. Run the plain-text extractor on a schedule and diff successive snapshots to detect when a page’s content changes, useful for tracking product prices, news updates, or government filings.
- Extract structured data. Adapt the CSS-selector technique from the header example to pull specific data fields (prices, dates, names) from pages with consistent HTML structure, and export the results as CSV or JSON.
- Combine with the Lightpanda browser client. For pages that require JavaScript rendering, use the Lightpanda interface from the previous chapter to fetch the fully rendered HTML, then pass that HTML through the text or Markdown extraction functions developed here.
Optional Practice Problems
Extraction of Hyperlinks into an Association List: In page-text.lisp, hyperlink tags
<a>are processed by simply extracting their inner text content. Write a functionextract-all-linksthat parses the DOM tree using CLSS and extracts allhrefattributes, converting them into an association list of(anchor-text . url). Filter out empty anchors or relative links, resolving relative paths against the base URL.Robots.txt Parser and Rate Limiting Compliance: To comply with the responsible scraping guidelines mentioned in the introduction of web-scraping.md, write a utility function
allowed-by-robots-pinutils.lispor as a helper. Fetch and parse the/robots.txtfile for a given URL, check if the current user agent is allowed to access the target path, and read anyCrawl-delaydirective to sleep dynamically before making requests via Drakma.Markdown Table Converter: The
html-to-markdownfunction in page-markdown.lisp renders table cells as separated blocks, but does not produce GitHub Flavored Markdown (GFM) tables. Extend the:tableand:trbranches to collect rows, detect header cells (<th>), and emit a correctly formatted GFM table with a header separator row.User-Agent Rotation and Request Header Customization:
fetch-htmlin utils.lisp sends a single fixed User-Agent. Change it to randomly select a User-Agent string from a pre-defined list of modern browser headers (Chrome, Firefox, Safari) and to add custom request headers likeAccept-LanguageorRefererto the Drakma request.Recursive Site Crawler with Level Limits (DFS/BFS): Build a recursive site crawler in page-text.lisp that starts from a seed URL, extracts all internal links (using your link extraction function), and recursively scrapes pages up to a maximum depth limit (e.g. depth 2). Store the scraped pages as individual Markdown files, keeping track of visited URLs in a hash table to avoid infinite recursion.
Automatic Encoding Detection and Recovery:
fetch-htmlin utils.lisp notes that Drakma falls back to Latin-1 when a server omits the charset, which garbles UTF-8 text. Write a wrapper functionfetch-html-with-encoding-recoverythat requests the body as bytes (:want-stream nilcombined with a Latin-1 external format, ordrakma:http-requestwith:external-formatoverridden), searches the raw bytes for<meta charset="...">tags, and re-decodes the byte array usingflexi-streams:octets-to-stringwith the detected encoding.