Information Gathering Using DBPedia Lookup
Wikipedia is a great source of information. As you may know, you can download a data dump of all Wikipedia data with or without version information and comments. When I want fast access to the entire Wikipedia set of English language articles I use the DBPedia Lookup service for finding information on entities like people, companies, etc.
The source code for this chapter in in the directory loving-common-lisp/src/dbpedia.
To load and run an example, try:
1 (ql:quickload "dbpedia")
2 (dbpedia:dbpedia-lookup "berlin")
Wikipedia is a great resource to have on hand but I am going to show you in this section how to access the Semantic Web version or Wikipedia, DBPedia using the DBPedia Lookup Service in the next code listing that shows the contents of the example file dbpedia-lookup.lisp:
1 (in-package #:dbpedia)
2
3 ;; utility from http://cl-cookbook.sourceforge.net/strings.html#manip:
4 (defun replace-all (string part replacement &key (test #'char=))
5 "Returns a new string in which all the occurrences of the part
6 is replaced with replacement."
7 (with-output-to-string (out)
8 (loop with part-length = (length part)
9 for old-pos = 0 then (+ pos part-length)
10 for pos = (search part string
11 :start2 old-pos
12 :test test)
13 do (write-string string out
14 :start old-pos
15 :end (or pos (length string)))
16 when pos do (write-string replacement out)
17 while pos)))
18
19 (defstruct dbpedia-data uri label description)
20
21 (defun dbpedia-lookup (search-string)
22 (let* ((s-str (replace-all search-string " " "+"))
23 (s-uri
24 (concatenate
25 'string
26 "https://lookup.dbpedia.org/api/search?query="
27 s-str))
28 (response-body nil)
29 (response-status nil)
30 (response-headers nil)
31 (xml nil)
32 ret)
33 (multiple-value-setq
34 (response-body response-status response-headers)
35 (drakma:http-request
36 s-uri
37 :method :get
38 :accept "application/xml"))
39 (unless (= response-status 200)
40 (error "DBpedia lookup failed with status ~A" response-status))
41 (let ((xml-str (typecase response-body
42 (string response-body)
43 (vector (babel:octets-to-string response-body))
44 (t (error "Invalid or empty response body from DBpedia")))))
45 (setf xml (s-xml:parse-xml-string xml-str)))
46 (dolist (r (cdr xml))
47 (let ((uri (cadr (find "uri" (cdr r)
48 :key (lambda (el) (and (consp el) (symbol-name (first el))))
49 :test #'string-equal)))
50 (label (cadr (find "Label" (cdr r)
51 :key (lambda (el) (and (consp el) (symbol-name (first el))))
52 :test #'string-equal)))
53 (desc (cadr (find "Description" (cdr r)
54 :key (lambda (el) (and (consp el) (symbol-name (first el))))
55 :test #'string-equal))))
56 (push
57 (make-dbpedia-data
58 :uri uri
59 :label label
60 :description (and desc (string-trim '(#\Space #\NewLine #\Tab) desc)))
61 ret)))
62 (reverse ret)))
63
64 ;; (dbpedia:dbpedia-lookup "berlin")
I am only capturing the attributes for DBPedia URI, label and description in this example code. If you uncomment line 41 and look at the entire response body from the call to DBPedia Lookup, you can see other attributes that you might want to capture in your applications.
The following diagram shows the high-level architecture of the information gathering tools developed in this chapter:
Here is a sample call to the function dbpedia:dbpedia-lookup (only some of the returned data is shown):
1 * (ql:quickload "dbpedia")
2 * (dbpedia:dbpedia-lookup "berlin")
3
4 (#S(DBPEDIA-DATA
5 :URI "http://dbpedia.org/resource/Berlin"
6 :LABEL "Berlin"
7 :DESCRIPTION
8 "Berlin is the capital city of Germany and one of the 16 states of Germany.
9 With a population of 3.5 million people, Berlin is Germany's largest city
10 and is the second most populous city proper and the eighth most populous
11 urban area in the European Union. Located in northeastern Germany, it is
12 the center of the Berlin-Brandenburg Metropolitan Region, which has 5.9
13 million residents from over 190 nations. Located in the European Plains,
14 Berlin is influenced by a temperate seasonal climate.")
15 ...)
Wikipedia, and the DBPedia linked data for of Wikipedia are great sources of online data. If you get creative, you will be able to think of ways to modify the systems you build to pull data from DPPedia. One warning: Semantic Web/Linked Data sources on the web are not available 100% of the time. If your business applications depend on having the DBPedia always available then you can follow the instructions on the DBPedia web site to install the service on one of your own servers.
Wrap Up
DBPedia Lookup is a convenient way to get information on entities in Wikipedia and DBPedia. In the chapter Web Scraping we look at ways to pull data directly off web sites.