Using MongoDB, Solr NoSQL Data Stores

Non-relational data stores are commonly used for applications that don’t need either full relational algebra or must scale.

The MongoDB example code is in the file src/loving_snippets/mongo_news.lisp. The Solr example code is in the subdirectories src/solr_examples.

Note for the fifth edition: The Common Lisp cl-mongo library is now unsupported for versions of MongoDB later than 2.6 (released in 2016). You can install an old version of MongoDB for macOS or for Linux. I have left the MongoDB examples in this section but I can’t recommend that you use cl-mongo and MongoDB for any serious applications.

Brewer’s CAP theorem states that a distributed data storage system comprised of multiple nodes can be robust to two of three of the following guarantees: all nodes always have a Consistent view of the state of data, general Availablity of data if not all nodes are functioning, and Partition tolerance so clients can still communicate with the data storage system when parts of the system are unavailable because of network failures. The basic idea is that different applications have different requirements and sometimes it makes sense to reduce system cost or improve scalability by easing back on one of these requirements.

A good example is that some applications may not need transactions (the first guarantee) because it is not important if clients sometimes get data that is a few seconds out of date.

MongoDB allows you to choose consistency vs. availability vs. efficiency.

I cover the Solr indexing and search service (based on Lucene) both because a Solr indexed document store is a type of NoSQL data store and also because I believe that you will find Solr very useful for building systems, if you don’t already use it.

The following discussion of MongoDB is based on just my personal experience, so I am not covering all use cases. I have used MongoDB for:

• Small clusters of MongoDB nodes to analyze social media data, mostly text mining and sentiment analysis. In all cases for each application I ran MongoDB with one write master (i.e., I wrote data to this one node but did not use it for reads) and multiple read-only slave nodes. Each slave node would run on the same server that was usually performing a single bit of analytics.
• Multiple very large independent clusters for web advertising. Problems faced included trying to have some level of consistency across data centers. Replica sets were used within each data center.
• Running a single node MongoDB instance for low volume data collection and analytics.

One of the advantages of MongoDB is that it is very “developer friendly” because it supports ad-hoc document schemas and interactive queries. I mentioned that MongoDB allows you to choose consistency vs. availability vs. efficiency. When you perform MongoDB writes you can specify some granularity of what constitutes a “successful write” by requiring that a write is performed at a specific number of nodes before the client gets acknowledgement that the write was successful. This requirement adds overhead to each write operation and can cause writes to fail if some nodes are not available.

The MongoDB online documentation is very good. You don’t have to read it in order to have fun playing with the following Common Lisp and MongoDB examples, but if you find that MongoDB is a good fit for your needs after playing with these examples then you should read the documentation. I usually install MongoDB myself but it is sometimes convenient to use a hosting service. There are several well regarded services and I have used MongoHQ.

At this time there is no official Common Lisp support for accessing MongoDB but there is a useful project by Alfons Haffmans’ cl-mongo that will allow us to write Common Lisp client applications and have access to most of the capabilities of MongoDB.

The file src/mongo_news.lisp contains the example code used in the next three sessions.

The following repl listing shows the cl-mongo APIs for creating a new document, adding elements (attributes) to it, and inserting it in a MongoDB data store:

In this example, three string attributes were added to a new document before it was saved.

We will start by fetchng and pretty-printing all documents in the collection articles and fetching all articles a list of nested lists where the inner nested lists are document URI, title, and text:

Output for these two functions looks like:

By reusing the function article-results->lisp-data defined in the last section, we can also search for JSON documents using regular expressions matching attribute values:

I set the limit to return a maximum of ten documents. If you do not set the limit, this example code only returns one search result. The following repl listing shows the results from calling function search-articles-text:

I find using MongoDB to be especially effective when experimenting with data and code. The schema free JSON document format, using interactive queries using the mongo shell, and easy to use client libraries like clouchdb for Common Lisp will let you experiment with a lot of ideas in a short period of time. The following listing shows the use of the interactive mongo shell. The database news is the database used in the MongoDB examples in this chapter; you will notice that I also have other databases for other projects on my laptop:

Line 1 of this listing shows starting the mongo shell. Line 4 shows how to list all databases in the data store. In line 13 I select the database “news” to use. Line 15 prints out the names of all collections in the current database “news”. Line 18 prints out all documents in the “articles” collection. You can read the documentation for the mongo shell for more options like selective queries, adding indices, etc.

When you run a MongoDB service on your laptop, also try the admin interface on http://localhost:28017/.

The Lucene project is one of the most widely used Apache Foundation projects. Lucene is a flexible library for preprocessing and indexing text, and searching text. I have personally used Lucene on so many projects that it would be difficult to count them. The Apache Solr Project adds a network interface to the Lucene text indexer and search engine. Solr also adds other utility features to Lucene:

• While Lucene is a library to embed in your programs, Solr is a complete system.
• Solr provides good defaults for preprocessing and indexing text and also provides rich support for managing structured data.
• Provides both XML and JSON APIs using HTTP and REST.
• Supports faceted search, geospatial search, and provides utilities for highlighting search terms in surrounding text of search results.
• If your system ever grows to a very large number of users, Solr supports scaling via replication.

I hope that you will find the Common Lisp example Solr client code in the following sections helps you make Solr part of large systems that you write using Common Lisp.

Download a binary Solr distribution and un-tar or un-zip this Solr distribution, cd to the distribution directory, then cd to the example directory and run:

You can access the Solr Admin Web App at http://localhost:8983/solr/#/. This web app can be seen in the following screen shot:

There is no data in the Solr example index yet, so following the Solr tutorial instructions:

You will learn how to add documents to Solr directly in your Common Lisp programs in a later section.

Assuming that you have a fast Internet connection so that downloading Solr was quick, you have hopefully spent less than five or six minutes getting Solr installed and running with enough example search data for the Common Lisp client examples we will play with. Solr is a great tool for storing, indexing, and searching data. I recommend that you put off reading the official Solr documentation for now and instead work through the Common Lisp examples in the next two sections. Later, if you want to use Solr then you will need to carefully read the Solr documentation.

The Solr REST Interface Documentation documents how to perform search using HTTP GET requests. All we need to do is implement this in Common Lisp which you will see is easy.

Assuming that you have Solr running and the example data loaded, we can try searching for documents with, for example, the word “British” using the URL http://localhost:8983/solr/select?q=British. This is a REST request URL and you can use utilities like curl or wget to fetch the XML data. I fetched the data in a web browser, as seen in the following screen shot of a Firefox web browser (I like the way Firefox formats and displays XML data):

The attributes in the returned search results need some explanation. We indexed several example XML data files, one of which contained the following XML element that we just saw as a search result:

So, the search result has the same attributes as the structured XML data that was added to the Solr search index. Solr’s capability for indexing structured data is a superset of just indexing plain text. If for example we were indexing news stories, then example input data might look like:

With this example, a search result that returned this document as a result would return attributes id, title, and text, and the values of these three attributes.

By default the Solr web service returns XML data as seen in the last screen shot. For our examples, I prefer using JSON so we are going to always add a request parameter wt=json to all REST calls. The following screen shot shows the same data returned in JSON serialization format instead of XML format of a Chrome web browser (I like the way Chrome formats and displays JSON data with the JSONView Chrome Browser extension):

You can read the full JSON REST Solr documentation later, but for our use here we will use the following search patterns:

• http://localhost:8983/solr/select?q=British+One&wt=json - search for documents with either of the words “British” or “one” in them. Note that in URIs that the “+” character is used to encode a space character. If you wanted a “+” character you would encode it with “%2B” and a space character is encoded as “%20”. The default Solr search option is an OR of the search terms, unlike, for example, Google Search.
• http://localhost:8983/solr/select?q=British+AND+one&wt=json - search for documents that contain both of the words “British” and “one” in them. The search term in plain text is “British AND one”.

As we sawearlier in Network Programming it is fairly simple to use the drakma and cl-json Common Lisp libraries to call REST services that return JSON data. The function do-search defined in the next listing (all the Solr example code is in the file src/solr-client.lisp) constructs a query URI as we saw in the last section and uses the Drackma library to perform an HTTP GET operation and the cl-json library to parse the returned string containing JSON data into Lisp data structures:

This example code does return the search results as Lisp list data; for example:

I might modify the search function to return just the fetched documents as a list, discarding the returned Solr meta data:

There are a few more important details if you want to add Solr search to your Common Lisp applications. When there are many search results you might want to fetch a limited number of results and then “page” through them. The following strings can be added to the end of a search query:

• &rows=2 this example returns a maximum of two “rows” or two query results.
• &start=4 this example skips the first 4 available results

A query that combines skipping results and limiting the number of returned results looks like this:

In the last example we relied on adding example documents to the Solr search index using the directions for setting up a new Solr installation. In a real application, in addition to performing search requests for indexed documents you will need to add new documents from your Lisp applications. Using the Drakma we will see that it is very easy to add documents.

We need to construct a bit of XML containing new documents in the form:

You can specify whatever field names (attributes) that are required for your application. You can also pass multiple <doc></doc> elements in one add request. We will want to specify documents in a Lisp-like way: a list of cons values where each cons value is a field name and a value. For the last XML document example we would like an API that lets us just deal with Lisp data like:

One thing to note: the attribute names and values must be passed as strings. Other data types like integers, floating point numbers, structs, etc. will not work.

This is nicer than having to use XML, right? The first thing we need is a function to convert a list of cons values to XML. I could have used the XML Builder functionality in the cxml library that is available via Quicklisp, but for something this simple I just wrote it in pure Common Lisp with no other dependencies (also in the example file src/solr-client.lisp) :

The macro with-output-to-string on line 2 of the listing is my favorite way to generate strings. Everything written to the variable stream inside the macro call is appended to a string; this string is the return value of the macro.

The following function adds documents to the Solr document input queue but does not actually index them:

You have noticed in line 3 that I am accessing a Solr server running on localhost and not a remote server. In an application using a remote Solr server you would need to modify this to reference your server; for example:

For efficiency Solr does not immediately add new documents to the index until you commit the additions. The following function should be called after you are done adding documents to actually add them to the index:

Notice that all we need is an empty element <commit></commit> that signals the Solr server that it should index all recently added documents. The following repl listing shows everything working together (I am assuming that the contents of the file src/solr-client.lisp has been loaded); not all of the output is shown in this listing:

Solr has a lot of useful features that we have not used here like supporting faceted search (drilling down in previous search results), geolocation search, and looking up indexed documents by attribute. In the examples I have shown you, all text fields are indexed but Solr optionally allows you fine control over indexing, spelling correction, word stemming, etc.

Solr is a very capable tool for storing, indexing, and searching data. I have seen Solr used effectively on projects as a replacement for a relational database or other NoSQL data stores like CouchDB or MongoDB. There is a higher overhead for modifying or removing data in Solr so for applications that involve frequent modifications to stored data Solr might not be a good choice.

There are more convenient languages than Common Lisp to use for accessing MongoDB. To be honest, my favorites are Ruby and Clojure. That said, for applications where the advantages of Common Lisp are compelling, it is good to know that your Common Lisp applications can play nicely with MongoDB.

I am a polyglot programmer: I like to use the best programming language for any specific job. When we design and build systems with more than one programming language, there are several options to share data:

• Use foreign function interfaces to call one language from another from inside one process.
• Use a service architecture and send requests using REST or SOAP.
• Use shared data stores, like relational databases, MongoDB, CouchDB and Solr.

Hopefully this chapter and the last chapter will provide most of what you need for the last option.