Information Gathering Using DBPedia Lookup
Wikipedia is a great source of information. As you may know, you can download a data dump of all Wikipedia data with or without version information and comments. When I want fast access to the entire Wikipedia set of English language articles I use the DBPedia Lookup service for finding information on entities like people, companies, etc.
The source code for this chapter in in the directory loving-common-lisp/src/dbpedia.
To load and run an example, try:
1 (ql:quickload "dbpedia")
2 (dbpedia:dbpedia-lookup "berlin")
Wikipedia is a great resource to have on hand but I am going to show you in this section how to access the Semantic Web version or Wikipedia, DBPedia using the DBPedia Lookup Service in the next code listing that shows the contents of the example file dbpedia-lookup.lisp:
1 (in-package #:dbpedia)
2
3 (defstruct dbpedia-data uri label description)
4
5 (defun dbpedia-lookup (search-string)
6 (let* ((response-body nil)
7 (response-status nil)
8 (response-headers nil)
9 (xml nil)
10 ret)
11 (multiple-value-setq
12 (response-body response-status response-headers)
13 (drakma:http-request
14 "https://lookup.dbpedia.org/api/search"
15 :parameters (list (cons "query" search-string)) ; drakma URL-encodes this
16 :method :get
17 :accept "application/xml"))
18 (unless (= response-status 200)
19 (error "DBpedia lookup failed with status ~A" response-status))
20 (let ((xml-str (typecase response-body
21 (string response-body)
22 (vector (babel:octets-to-string response-body))
23 (t (error "Invalid or empty response body from DBpedia")))))
24 (setf xml (s-xml:parse-xml-string xml-str)))
25 (dolist (r (cdr xml))
26 (let ((uri (cadr (find "uri" (cdr r)
27 :key (lambda (el) (and (consp el) (symbol-name (first el))))
28 :test #'string-equal)))
29 (label (cadr (find "Label" (cdr r)
30 :key (lambda (el) (and (consp el) (symbol-name (first el))))
31 :test #'string-equal)))
32 (desc (cadr (find "Description" (cdr r)
33 :key (lambda (el) (and (consp el) (symbol-name (first el))))
34 :test #'string-equal))))
35 (push
36 (make-dbpedia-data
37 :uri uri
38 :label label
39 :description (and desc (string-trim '(#\Space #\NewLine #\Tab) desc)))
40 ret)))
41 (reverse ret)))
42
43 ;; (dbpedia:dbpedia-lookup "berlin")
The :parameters argument passes the search string to Drakma, which URL-encodes it and appends it to the URL as the query parameter. This handles spaces and characters such as & that would otherwise break the URL.
I am only capturing the attributes for DBPedia URI, label and description in this example code. If you print the variable response-body after the call to DBPedia Lookup returns, you can see other attributes that you might want to capture in your applications.
The following diagram shows the high-level architecture of the information gathering tools developed in this chapter:
Here is a sample call to the function dbpedia:dbpedia-lookup (only some of the returned data is shown):
1 * (ql:quickload "dbpedia")
2 * (dbpedia:dbpedia-lookup "berlin")
3
4 (#S(DBPEDIA-DATA
5 :URI "http://dbpedia.org/resource/Berlin"
6 :LABEL "Berlin"
7 :DESCRIPTION
8 "Berlin is the capital city of Germany and one of the 16 states of Germany.
9 With a population of 3.5 million people, Berlin is Germany's largest city
10 and is the second most populous city proper and the eighth most populous
11 urban area in the European Union. Located in northeastern Germany, it is
12 the center of the Berlin-Brandenburg Metropolitan Region, which has 5.9
13 million residents from over 190 nations. Located in the European Plains,
14 Berlin is influenced by a temperate seasonal climate.")
15 ...)
Wikipedia, and the DBPedia linked data for of Wikipedia are great sources of online data. If you get creative, you will be able to think of ways to modify the systems you build to pull data from DPPedia. One warning: Semantic Web/Linked Data sources on the web are not available 100% of the time. If your business applications depend on having the DBPedia always available then you can follow the instructions on the DBPedia web site to install the service on one of your own servers.
Wrap Up
DBPedia Lookup is a convenient way to get information on entities in Wikipedia and DBPedia. In the chapter Web Scraping we look at ways to pull data directly off web sites.