\n Preface\n
- \n Book Outline\n
- \n If You Did Not Buy This Book\n
\n
\n Network Programming Techniques for the Internet of Things\n
- \n Motivation for IoT\n
- \n Running the example programs\n
- \n Design Pattern\n
- \n Directory Lookups\n
- \n User Data Protocol Network Programming\n
- \n Multicast/Broadcast Network Programming\n
- \n Wrap Up on IoT\n
\n
\n Natural Language Processing Using OpenNLP\n
- \n Using OpenNLP Pre-Trained Models\n
- \n Training a New Categorization Model for OpenNLP\n
- \n Using Our New Trained Classification Model\n
- \n Using the OpenNLP Parsing Model\n
\n
\n Machine Learning Using Apache Spark\n
- \n Setting Up Spark On Your Laptop\n
- \n Hello Spark - a Word Count Example\n
- \n Introducing the Spark MLlib Machine Learning Library\n
- \n MLlib Logistic Regression Example Using University of Wisconsin Cancer Database\n
- \n MLlib SVM Classification Example Using University of Wisconsin Cancer Database\n
- \n MLlib K-Means Example Program\n
- \n Converting Text to Numeric Feature Vectors\n
- \n Using K-Means to Cluster Wikipedia Articles\n
- \n Using SVM for Text Classification\n
- \n Using word2vec To Find Similar Words In Documents\n
- \n Chapter Wrap Up\n
\n
\n Anomaly Detection Machine Learning Example\n
- \n Motivation for Anomaly Detection\n
- \n Math Primer for Anomaly Detection\n
- \n AnomalyDetection Utility Class\n
- \n Example Using the University of Wisconsin Cancer Data\n
\n
\n Deep Learning Using Deeplearning4j\n
- \n Deep Belief Networks\n
- \n Deep Belief Example\n
- \n Deep Learning Wrapup\n
\n
\n Web Scraping Examples\n
- \n Motivation for Web Scraping\n
- \n Using the jsoup Library\n
- \n Wrap Up\n
\n
\n Linked Data\n
- \n Example Code\n
- \n Overview of RDF and SPARQL\n
- \n SPARQL Query Client\n
- \n DBPedia Entity Lookup\n
- \n Annotate Text with DBPedia Entity URIs\n
- \n Resolving Named Entities in Text to Wikipedia URIs\n
- \n Combining Data from Public and Private Sources\n
- \n Wrap Up for Linked Data\n
\n
\n Java Strategies for Working with Cloud Data: Knowledge Management-Lite\n
- \n Motivation for Knowledge Management\n
- \n Using Google Drive Cloud Takeout Service\n
- \n Using Postgres as a Local Document Store with Text Search\n
- \n Wrap Up\n
\n
\n Book Wrap Up\n

Preface

\n\n

Please note that this book was published in the fall of 2015 and updated July 2016 for the latest version 2.0 of Apache Spark machine learning library.

\n\n

I have been programming since high school and since the early 1980s I have worked on artificial intelligence, neural networks, machine learning, and general web engineering projects. Most of my professional work is reflected in the examples in this book. These examples programs were also chosen based on their technological importance, i.e. the rapidly changing technical scene of big data, the use of machine learning in systems that touch most parts of our lives, and networked devices. I then narrowed the list of topics based on a public survey I announced on my blog. Many thanks to the people who took the time to take this survey. It is my hope that the Java example programs in this book will be useful in your projects. Hopefully you will also have a lot of fun working through these examples!

\n\n

Java is a flexible language that has a huge collection of open source libraries and utilities. Java gets some criticism for being a verbose programming language. I have my own coding style that is concise but may break some of the things you have learned about “proper” use of the language. The Java language has seen many upgrades since its introduction over 20 years ago. This book requires and uses the features of Java 8 so please update to the latest JDK if you have not already done so. You will also need to have maven installed on your system. I also provide project files for the free Community Version of the IntelliJ IDE.

\n\n

Everything you learn in this book can be used with some effort in the alternative JVM languages Clojure, JRuby, and Scala. In addition to Java I frequently use Clojure, Haskell, and Ruby in my work.

\n\n

Book Outline

\n\n

This book consists of eight chapters that I believe show the power of the Java language to good effect:

\n\n

Network programming techniques for the Internet of Things (IoT)
Natural Language Processing using OpenNLP including using existing models and creating your own models
Machine learning using the Spark mllib library
Anomaly Detection Machine Learning
Deep Learning using Deeplearning4j
Web Scraping
Using rich semantic and linked data sources on the web to enrich the data models you use in your applications
Java Strategies for Knowledge Management-Lite using Cloud Data Resources

\n\n

The first chapter on IoT is a tutorial on network programming techniques for IoT development. I have also used these same techniques for multiplayer game development and distributed virtual reality systems, and also in the design and implementation of a world-wide nuclear test monitoring system. This chapter stands on its own and is not connected to any other material in this book.

\n\n

The second chapter shows you how to use the OpenNLP library to train your own classifiers, tag parts of speech, and generally process English language text. Both this chapter and the next chapter on machine learning using the Spark mllib library use machine learning techniques.

\n\n

The fourth chapter provides an example of anomaly detection using the University of Wisconsin cancer database. The fifth chapter is a short introduction to pulling plain text and semi-structured data from web sites.

\n\n

The last two chapters are for information architects or developers who would like to develop information design and knowledge management skills. These chapters cover linked data (semantic web) and knowledge management techniques.

\n\n

The source code for the examples can be found at https://github.com/mark-watson/power-java and are all released under the Apache 2 license. I have tried to use only existing libraries in the examples that are either Apache 2 or MIT style licensed. In general I prefer Free Software licenses like GPL, LGPL, and AGPL but for examples in a book where I expect readers to sometimes reuse entire example programs or at least small snippets of code, a license that allows use in commercial products makes more sense.

\n\n

There is a subdirectory in this github repository for each chapter, each with its own maven pom.xml file to build and run the examples.

\n\n

The five chapters are independent of each other so please feel free to skip around when reading and experimenting with the sample programs.

\n\n

This book is available for purchase at https://leanpub.com/powerjava.

\n\n

You might be interested in other books that I have self-published via leanpub:

\n\n

My older books published by Springer-Verlag, McGraw-Hill, Morgan Kaufman, APress, Sybex, M&T Press, and J. Wiley are listed on the books page of my web site.

\n\n

One of the major themes of this book is machine learning. In addition to my general technical blog I have a separate blog that contains information on using machine learning and cognition technology: blog.cognition.tech and an associated website supporting cognition technology.

\n\n

If You Did Not Buy This Book

\n\n

I frequently find copies of my books on the web. If you have a copy of this book and did not buy it please consider paying the minimum purchase price of $4 at leanpub.com/powerjava.

\n\n\n

Network Programming Techniques for the Internet of Things

\n\n

This chapter will show you techniques of network programming relevant to developing Internet of Things (IoT) projects and products using local TCP/IP and UDP networking. We will not cover the design of hardware or designing IoT user experiences. Specifically, we will look at techniques for using local directory services to publish and look up available services and techniques for efficiently communicating using UDP, multicast and broadcast.

\n\n

This chapter is a tutorial on network programming techniques that I believe you will find useful for developing IoT applications. The material on User Data Protocol (UDP) and multicast is also useful for network game development.

\n\n

I am not covering some important material: the design of user experience and devices, and IoT devices that use local low power radios to connect cooperating devices. That said, it is worth thinking about what motivates the development of IoT devices and we will do this in the next section.

\n\n

There are emerging standards for communication between IoT devices and open source projects like TinyOS and Contiki that are C language based, not Java based, so I won’t discuss them. Oracle supports the Java ME Embedded profile that is used in some IoT products but in this chapter I want to concentrate network programming techniques and example programs that run on stock Java (including Android devices).

\n\n

Motivation for IoT

\n\n

We are used to the physical constraints of using computing devices. When I was in high school in the mid 1960s and took a programming class at a local college I had to make a pilgrimage to the computer center, wait for my turn to use a keypunch machine, walk over to submit my punch cards, and stand around and wait and eventually get a printout and my punch cards returned to me. Later, interactive terminals allowed me to work in a more comfortable physical environment. Jumping ahead almost fifty years I can now use my smartphone to SSH into my servers, watch movies, and even use a Java IDE. As I write this book, perhaps 70% of Internet use is done on mobile devices.

\n\n

IoT devices complete this process of integrating computing devices more into our life with fewer physical constraints on us. Our future will include clothing, small personal items like pens, cars, furniture, etc., all linked in a private and secure data fabric. Unless you are creating content (e.g., writing software, editing pictures and video, or performing a knowledge engineering task) it is likely that you won’t think too much about the devices that you interact with and in general will prefer mobile devices. This is a similar experience to driving a car where tasks like braking, steering, etc. are “automatic” and don’t require much conscious thought.

\n\n

We are physical creatures. Many of us gesture with our hands while talking, move things around on our desk while we are thinking, and generally use our physical environment. Amazing products will combine physical objects and computing and information management.

\n\n

Before diving into techniques for communication between IoT devices, we will digress briefly in the next section to see how to run the sample programs for this chapter.

\n\n

Running the example programs

\n\n

If you want to try running the example programs let’s set that up before going through the network programming techniques and code later in this chapter. Assuming that you have cloned the github repository for the examples in this book, go to the subdirectory internet_of_things, open two shell windows and use the following commands to run the examples in the same order that they are presented (these commands are in the README.md file so you can copy and paste them). These commands have been split to multiple lines to fit the page width of this book:

\n\n

Build

1     mvn clean compile package assembly:single\n

\n\n\n\n

Run service discovery

1     java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n2 \t     com.markwatson.iot.CreateTestService\n3     java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar /\n4 \t     com.markwatson.iot.LookForTestService\n

\n\n\n\n

Run UDP experiments:

 1     java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n 2 \t     com.markwatson.iot.UDPServer\n 3     java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n 4 \t     com.markwatson.iot.UDPClient\n 5 \n 6     # Multicast experiments:\n 7     java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n 8 \t     com.markwatson.iot.MulticastServer\n 9     java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n10 \t     com.markwatson.iot.MulticastClient\n

\n\n\n\n

Line 2 compiles and builds the examples. For each of the example programs you will run two programs on the command line. For example for the service discovery example on lines 5 and 6 (which would be entered on one line in a terminal window) we start a test service and lines 7 and 8 run the service test client. Similarly, lines 10 through 20 show how to run the services and test clients for the UDP experiments and the multicast experiments. I repeat these instructions with each example as we look at the code but I thought that you might enjoy running all three examples before we look at the implementations of the examples later in this chapter.

\n\n

In each example, run the two example programs in different shell windows. For more fun, run the examples using different laptops or any device that can run Java code.

\n\n

The following figure shows the project for this chapter in the Community Edition of IntelliJ:

\n\n\n

This chapter starts with a design pattern that you might use for writing IoT applications. Then the following three major sections show you how to use directory lookups, use User Data Protocol (UDP) instead of TCP/IP, and then multi-cast/broadcast.

\n\n

The material in this chapter is fairly elementary but it seems like many developers don’t have experience with lower level network programming techniques. My hope is that you will understand when you might want to use a lower level protocol like UDP instead of TCP and when you might find directory lookups useful in your applcations.

\n\n

Design Pattern

\n\n

The pattern that the examples in the following sections support is simple (in principle): small devices register services that they offer and look for services offered in other devices. Devices will use User Data Protocol (UDP) and multi-cast to communicate for using and providing services. Once a consumer has discovered a service privider (host, IP address, port, and service types) a consumer can also open a direct TCP/IP socket connection to a provider to use a newly discovered service.

\n\n

The following diagram shows several IoT use cases:

\n\n

A consumer uses the JmDNS library to broadcast on the local network a request for services provided.
The JmDNS library on the provider side hears a service description request and broadcasts available services.
The JmDNS library on the consumer side listens for the broadcast from the provider.
Using a service description, the consumer makes a direct socket connection to the provider for a service.
The provider periodically broadcasts updated data for its service.
The consumer listens for broadcasts of updated data and uses new data as it is available.

\n\n\n

\"Consumer — Consumer and Provider Use Cases

\n\n\n

This figure shows a few use cases. You can use these interaction patterns of interactions between IoT devices using application specific software you write.

\n\n

Directory Lookups

\n\n

In the next two sub-sections we will be using the Java JmDNS library that supports a local multi-cast Domain Name Service (DNS) and supports service discovery. JmDNS is compatible with Apple’s Bonjour discovery service that is also available for installation on Microsoft Windows.

\n\n

JmDNS has been successfully used in Android apps if you change the defaults on your phone to accept multi-cast (since Android version 4.1 Network Service Discovery (NDS)). Android development is outside the scope of this chapter and I refer you to the NDS documentation and example programs if you are interested. I tried the example Android NsdChat application on my Galaxy S III (Android 4.4.2) and Note 4 (Android 5.0.1) and this example app would be a good place for you to start if you are interested in using Android apps to experiment with IoT development.

\n\n

Create and Register a Service

\n\n

The example program in this section registers a service using the JmsDNS Java library that listens on port 1234. In the next section we look at another example program that connects to this service.

\n\n

 1 package com.markwatson.iot;\n 2 \n 3 import javax.jmdns.*;\n 4 import java.io.IOException;\n 5 \n 6 public class CreateTestService {\n 7   static public void main(String[] args) throws IOException {\n 8     JmDNS jmdns = JmDNS.create();\n 9     jmdns.registerService(\n10         ServiceInfo.create(\"_http._tcp.local.\", \"FOO123.\", 1234,\n11             0, 0, \"any data for FOO123\")\n12     );\n13     // listener for service FOO123:\n14     jmdns.addServiceListener(\"_http._tcp.local.\", new Foo123Listener());\n15   }\n16 \n17   static class Foo123Listener implements ServiceListener {\n18 \n19     public void serviceAdded(ServiceEvent serviceEvent) {\n20       System.out.println(\"FOO123 SERVICE ADDED in class CreateTestService\");\n21     }\n22 \n23     public void serviceRemoved(ServiceEvent serviceEvent) {\n24       System.out.println(\"FOO123 SERVICE REMOVED in class CreateTestService\");\n25     }\n26 \n27     public void serviceResolved(ServiceEvent serviceEvent) {\n28       System.out.println(\"FOO123 SERVICE RESOLVED in class CreateTestService\");\n29     }\n30   }\n31 }\n

\n\n\n\n

The service is created in lines 10 and 11 defining the service name (notice that a period is added to the end of the service name FOO123), the port that the server listens on (1234 in this case), and the last argument is any string data that you also want to pass to clients looking up information on service FOO123. When you run the client in the next section you will see that the client receives this information for the local DNS lookup.

\n\n

Running this example produces the following output (edited to fit the page width):

\n\n

> java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n       com.markwatson.iot.CreateTestService\nFOO123 SERVICE ADDED in class CreateTestService\n

\n\n\n\n

Lookup a Service by Name

\n\n

If you have the service created by the example program in the last section running then the following example program will lookup the service.

\n\n

 1 package com.markwatson.iot;\n 2 \n 3 import javax.jmdns.*;\n 4 import java.io.IOException;\n 5 \n 6 // reference: https://github.com/openhab/jmdns\n 7 \n 8 public class LookForTestService {\n 9 \n10   static public void main(String[] args) throws IOException {\n11     System.out.println(\"APPLICATION entered registerService\");\n12     JmDNS jmdns = JmDNS.create();\n13     // try to find a service named \"FOO123\":\n14     ServiceInfo si = jmdns.getServiceInfo(\"_http._tcp.local.\", \"FOO123\");\n15     System.out.println(\"APPLICATION si: \" + si);\n16     System.out.println(\"APPLICATION APP: \" + si.getApplication());\n17     System.out.println(\"APPLICATION DOMAIN: \" + si.getDomain());\n18     System.out.println(\"APPLICATION NAME: \" + si.getName());\n19     System.out.println(\"APPLICATION PORT: \" + si.getPort());\n20     System.out.println(\"APPLICATION getQualifiedName: \" + si.getQualifiedName());\n21     System.out.println(\"APPLICATION getNiceTextString: \" + si.getNiceTextString(\\\n22 ));\n23   }\n24 }\n

\n\n\n\n

We use the JmsDNS library in lines 12 and 14 to find a service by name. In line 14 the first argument “_http._tcp.local.” indicates that we are searching the local network (for example, your home network created by a router from your local ISP).

\n\n

Running this example produces the following output (edited to fit the page width):

\n\n

> java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n       com.markwatson.iot.LookForTestService\nAPPLICATION entered registerService\nAPPLICATION si: [ServiceInfoImpl@1595428806\n                  name: 'FOO123._http._tcp.local.' address: '/192.168.0.14:1234'\n                  status: 'DNS: Marks-MacBook-Air-local.local.\n                  state: probing 1 task: null', has data empty]\nAPPLICATION APP: http\nAPPLICATION DOMAIN: local\nAPPLICATION NAME: FOO123\nAPPLICATION PORT: 1234\nAPPLICATION getQualifiedName: FOO123._http._tcp.local.\nAPPLICATION getNiceTextString: any data for FOO123\n

\n\n\n\n

Do you want to find all local services? Then you can also create a listener that will notify you of all local HTTP services. There is an example program in the JmsDNS distribution on the path src/sample/java/samples/DiscoverServices.java written by Werner Randelshofer. I changed the package name to com.markwatson.iot and added it to the git repository for this book. You can try it using the following commands:

\n\n

mvn clean compile package assembly:single\njava -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n         com.markwatson.iot.CreateTestService\njava -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n         com.markwatson.iot.DiscoverServices\n

\n\n\n\n

While it is often useful to find all local services, I expect that for your own projects you will be writing both service and client applications that cooperate on your local network. In this case you know the service names and can simply look them up as in the example in this section.

\n\n

User Data Protocol Network Programming

\n\n

You are most likely familiar with using TPC/IP in network programming. TCP/IP provides some guarantees that the sender is notified on failure to deliver data to a receiving program. User Data Protocol (UDP) makes no such guarantees for notification of message delivery and is very much more efficient than TCP/IP. On local area networks data delivery is more robust to failure compared to the Internet and since we are programming for devices on a local network it makes some sense to use a lower level and more efficient protocol.

\n\n

Example UDP Server

\n\n

The example program in this section implements a simple service that listens on port 9005 for UDP packets that are client requests. The UDP packets received from clients are assumed to be text and this example slightly modifies the text and returns it to the client.

\n\n

 1 package com.markwatson.iot;\n 2 \n 3 import java.io.IOException;\n 4 import java.net.*;\n 5 \n 6 public class UDPServer {\n 7   final static int PACKETSIZE = 256;\n 8   final static int PORT = 9005;\n 9 \n10   public static void main(String args[]) throws IOException {\n11     DatagramSocket socket = new DatagramSocket(PORT);\n12     while (true) {\n13       DatagramPacket packet =\n14          new DatagramPacket(new byte[PACKETSIZE], PACKETSIZE);\n15       socket.receive(packet);\n16       System.out.println(\"message from client: address: \" + packet.getAddress() +\n17           \" port: \" + packet.getPort() + \" data: \" +\n18           new String(packet.getData()));\n19       // change message content before sending it back to the client:\n20       packet.setData((\"from server - \" + new String(packet.getData())).getBytes(\\\n21 ));\n22       socket.send(packet);\n23     }\n24   }\n25 }\n

\n\n\n\n

In line 11 we create a new DatagramSocket listener on port 9005. On lines 13 and 14 we create a new empty DatagramPacket that is used in line 16 to hold data from a client. The example code prints out packet data, changes the text from the client, and returns the modified text to the client.

\n\n

Running this example produces the following output (edited to fit the page width):

\n\n

1 > java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n2         com.markwatson.iot.UDPServer\n3 message from client: address: /127.0.0.1 port: 63300\n4                      data: Hello Server1431540275433\n

\n\n\n\n

Note that you must also run the example program from the next section to get the output in lines 4 and 5. This is why I listed the commands at the beginning of the chapter for running all of the examples: I thought that if you ran all of the examples according to these earlier directions, the example programs would be easier to understand.

\n\n

Example UDP Client

\n\n

The example program in this section implements a simple client for the UDP server in the last section. A sample text string is sent to the server and the response is printed.

\n\n

 1 package com.markwatson.iot;\n 2 \n 3 import java.io.IOException;\n 4 import java.net.*;\n 5 \n 6 public class UDPClient {\n 7 \n 8   static String INET_ADDR = \"localhost\";\n 9   static int PORT = 9005;\n10   static int PACKETSIZE = 256;\n11 \n12   public static void main(String args[]) throws IOException {\n13     InetAddress host = InetAddress.getByName(INET_ADDR);\n14     DatagramSocket socket = new DatagramSocket();\n15     byte[] data = (\"Hello Server\" + System.currentTimeMillis()).getBytes();\n16     DatagramPacket packet = new DatagramPacket(data, data.length, host, PORT);\n17     socket.send(packet);\n18     socket.setSoTimeout(1000); // 1000 milliseconds\n19     packet.setData(new byte[PACKETSIZE]);\n20     socket.receive(packet);\n21     System.out.println(new String(packet.getData()));\n22     socket.close();\n23   }\n24 }\n

\n\n\n\n

Running this example produces the following output (edited to fit the page width):

\n\n

> java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n         com.markwatson.iot.UDPClient\nfrom server - Hello Server1431540275433\n

\n\n\n\n

I have used UDP network programming in many projects and I will mention two of them here. In the 1980s I was an architect and developer on the NMRD project that used 38 seismic research stations around the world to collect data that we used to determine if any country was conducting nuclear explosive tests. We used a guaranteed messaging system for meta control data but used low level UDP for much of the raw data transfer. The other application was communcations library for networked racecar and hovercraft games that ran on a local network. UDP is very good for local network game development because it is efficient and the chance of losing data packets is very low and even if packets are lost game play is usually not affected. Remember that when using UDP you are not guaranteed delivery of data and not guaranteed notification of failure!

\n\n

Multicast/Broadcast Network Programming

\n\n

The examples in the next two sections are similar to the examples in the last two sections with one main difference: a server periodically broadcasts data using UDP. The server does not care if any client is listening or not, it simply broadcasts data. If any clients are listening they will probably receive the data. I say that they will probably receive the data because UDP provides no delivery guarantees.

\n\n

There are specific INET addresses that can be used for local multicast in the range of addresses between 224.0.0.0 and 224.0.0.255. We will use 224.0.0.5 in the two example programs. These are local addresses that are invisible to the the Internet.

\n\n

Multicast Broadcast Server

\n\n

The following example is very similar to that seen two sections ago (Java class UDPServer) except we do not wait for a client request.

\n\n

 1 package com.markwatson.iot;\n 2 \n 3 import java.io.IOException;\n 4 import java.net.*;\n 5 \n 6 public class MulticastServer {\n 7 \n 8   final static String INET_ADDR = \"224.0.0.5\";\n 9   final static int PORT = 9002;\n10 \n11   public static void main(String[] args)\n12       throws IOException, InterruptedException {\n13     InetAddress addr = InetAddress.getByName(INET_ADDR);\n14     DatagramSocket serverSocket = new DatagramSocket();\n15     for (int i = 0; i < 10; i++) {\n16       String message = \"message \" + i;\n17 \n18       DatagramPacket messagePacket =\n19           new DatagramPacket(message.getBytes(),\n20               message.getBytes().length, addr, PORT);\n21       serverSocket.send(messagePacket);\n22 \n23       System.out.println(\"broadcast message sent: \" + message);\n24       Thread.sleep(500);\n25     }\n26   }\n27 }\n

\n\n\n\n

In line 13 we create an INET address for “224.0.0.5” and in line 14 create a DatagramSocket that is reused for broadcasting 10 messages with a time delay of 1/2 a second between messages. The broadcast messages are defined in line 16 and the code in lines 18 to 21 creates a new DatagramPacket with the message text data and gets broadcast to the local network in line 21.

\n\n

Running this example produces the following output (output edited for brevity):

\n\n

> java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar com.markwatson.iot.\\\nMulticastServer\nbroadcast message sent: message 0\nbroadcast message sent: message 1\n ...\nbroadcast message sent: message 8\nbroadcast message sent: message 9\n

\n\n\n\n

Unlike the UDPServer example two sections ago, you will get this output regardless of whether you are running the listing client in the next section.

\n\n

Multicast Broadcast Listener

\n\n

The example in this section listens for broadcasts by the server example in the last section. The broadcast server and clients need to agree on the INET address and a port number (9002 in these examples).

\n\n

 1 package com.markwatson.iot;\n 2 \n 3 import java.io.IOException;\n 4 import java.net.DatagramPacket;\n 5 import java.net.InetAddress;\n 6 import java.net.MulticastSocket;\n 7 import java.net.UnknownHostException;\n 8 \n 9 public class MulticastClient {\n10 \n11   final static String INET_ADDR = \"224.0.0.5\";\n12   final static int PORT = 9002;\n13 \n14   public static void main(String[] args) throws IOException {\n15     InetAddress address = InetAddress.getByName(INET_ADDR);\n16     byte[] buf = new byte[512]; // more than enough for any UDP style packet\n17     MulticastSocket clientSocket = new MulticastSocket(PORT);\n18     // listen for broadcast messages:\n19     clientSocket.joinGroup(address);\n20 \n21     while (true) {\n22       DatagramPacket messagePacket = new DatagramPacket(buf, buf.length);\n23       clientSocket.receive(messagePacket);\n24 \n25       String message = new String(buf, 0, buf.length);\n26       System.out.println(\"received broadcast message: \" + message);\n27     }\n28   }\n29 }\n

\n\n\n\n

The client creates an InetAddress in line 15 and allocates storage for the received text messages in line 16. We create a MulticastSocket on port 9002 in line 17 and in line 19 join the broadcast server’s group. In lines 21 through 27 we loop forever listening for broadcast messages and printing them out.

\n\n

Running this example produces the following output (output edited for brevity):

\n\n

> java -cp target/iot-1.0-SNAPSHOT-jar-with-dependencies.jar \\\n          com.markwatson.iot.MulticastClient\nreceived broadcast message: message 0\nreceived broadcast message: message 1\n ...\nreceived broadcast message: message 8\nreceived broadcast message: message 9\n

\n\n\n\n

Wrap Up on IoT

\n\n

Even though I personally have some concerns about security and privacy for IoT applications and products, I am enthusiastic about the possibilities of making our lives simpler and more productive with small networked devices. When we work and play we want as much as possible to not think about our computing devices so that we can concentrate on our activities.

\n\n

I have barely scratched the surface on useful IoT technologies in this chapter but I hope that I have shown you network programming techniques that you can use in your projects (IoT and also networked game programming). As I write this book there are companies supplying IoT hardware development kits and deciding on one kit and software framework and creating your own projects is a good way to proceed.

\n\n\n

Natural Language Processing Using OpenNLP

\n\n

I have worked in the field of Natural Language Processing (NLP) since the early 1980s. Many more people are interested in the field of NLP and the techniques have changed drastically. NLP has usually been considered part of the field of artificial intelligence (AI) and in the 1980s and 1990s there was more interest in symbolic AI, that is the manipulation of high level symbols that are meaningful to people because of our knowledge of the real world. As an example, the symbol car means something to us but to AI software this symbol in itself has no meaning except for possible semantic relationships to other symbols like driver and road.

\n\n

There is still much work in NLP that deals with words as symbols: syntactic and semantic parsers being two examples. What has changed is a strong reliance on statistical and machine learning techniques. In this chapter we will use the open source (Apache 2 license) OpenNLP project that uses machine learning to create models of language. Currently, OpenNLP has support for Danish, German, English, Spanish, Portuguese, and Swedish. I include in the github repository some trained models for English that are used in the examples in this chapter. You can download models for other languages at the web page for OpenNLP 1.5 models (we are using version 1..6.0 of OpenNLP in this book which uses the version 1.5 models).

\n\n

I use OpenNLP for some of the NLP projects that I do for my consulting customers. I have also written my own NLP toolkit KBSPortal.com that is a commercial product. For customers who can use the GPL license I sometimes use the Stanford NLP libraries that, like OpenNLP, are written in Java.

\n\n

The following figure shows the project for this chapter in the Community Edition of IntelliJ:

\n\n\n

We will use pre-trained models for tokenizing text, recognizing the names of organizations, people, locations, and parts of speech for words in input text. We will also train a new model (the file opennlp/models/en-newscat.bin in the github repository) for recognizing the category of input text. The section on training new maximum entropy classification models using your own training data is probably the material in this chapter that you will use the most in your own projects. We will train one model to recognize the categories of COMPUTERS, ECONOMY, HEALTH, and POLITICS. You should then have the knowledge for training your own models using your own training texts for the categories that you need for your applications. We will also use both some pre-trained models that are included with the OpenNLP distribution and the classification mode that we will soon create in the next chapter when we learn how to perform scalable machine learning algorithms using the Apache Spark platform where we look at techniques for processing very large collections of text to discover information.

\n\n

After building a classification model we finish up this chapter with an interesting topic: statistically parsing sentences to discover the most probable linguistic structure of each sentence in input text. We will not use parsing in the rest of this book so you may skip the last section of this chapter if you are not currently interested in parsing sentences into linguistic components like noun and verp phrases, proper nouns, nouns, adjectives, etc.

\n\n

Using OpenNLP Pre-Trained Models

\n\n

Assuming that you have cloned the github repository for this book, you can fetch the maven dependencies, compile the code, and run the unit tests using the command:

\n\n

1     mvn clean build test\n

\n\n\n\n

The model files, including the categorization model you will learn to build later in this chapter, are found in the subdirectory models. The unit tests in src/test/java/com/markwatson/opennlp/NLPTest.java provide examples for using the code we develop in this chapter. The Java example code for tokenization (splitting text into individual words), splitting sentences, and recognizing organizations, locations, and people in text is all in the Java class NLP. You can look at the source code in the repository for this book. Here I will just show a few snippets of the code to make clear how to load and use pre-trained models.

\n\n

I use static class initialization to load the model files:

\n\n

 1   static public Tokenizer tokenizer = null;\n 2   static public SentenceDetectorME sentenceSplitter = null;\n 3   static POSTaggerME tagger = null;\n 4   static NameFinderME organizationFinder = null;\n 5   static NameFinderME locationFinder = null;\n 6   static NameFinderME personNameFinder = null;\n 7 \n 8   static {\n 9 \n10     try {\n11       InputStream organizationInputStream =\n12 \t    new FileInputStream(\"models/en-ner-organization.bin\");\n13       TokenNameFinderModel model =\n14 \t    new TokenNameFinderModel(organizationInputStream);\n15       organizationFinder = new NameFinderME(model);\n16       organizationInputStream.close();\n17 \n18       InputStream locationInputStream =\n19 \t    new FileInputStream(\"models/en-ner-location.bin\");\n20       model = new TokenNameFinderModel(locationInputStream);\n21       locationFinder = new NameFinderME(model);\n22       locationInputStream.close();\n23 \n24       InputStream personNameInputStream =\n25 \t    new FileInputStream(\"models/en-ner-person.bin\");\n26       model = new TokenNameFinderModel(personNameInputStream);\n27       personNameFinder = new NameFinderME(model);\n28       personNameInputStream.close();\n29 \n30       InputStream tokienizerInputStream =\n31 \t    new FileInputStream(\"models/en-token.bin\");\n32       TokenizerModel modelTokenizer = new TokenizerModel(tokienizerInputStream);\n33       tokenizer = new TokenizerME(modelTokenizer);\n34       tokienizerInputStream.close();\n35 \n36       InputStream sentenceInputStream =\n37 \t    new FileInputStream(\"models/en-sent.bin\");\n38       SentenceModel sentenceTokenizer = new SentenceModel(sentenceInputStream);\n39       sentenceSplitter = new SentenceDetectorME(sentenceTokenizer);\n40       tokienizerInputStream.close();\n41 \n42       organizationInputStream = new FileInputStream(\"models/en-pos-maxent.bin\");\n43       POSModel posModel = new POSModel(organizationInputStream);\n44       tagger = new POSTaggerME(posModel);\n45 \n46     } catch (IOException e) {\n47       e.printStackTrace();\n48     }\n49   }\n

\n\n\n\n

The first operation that you will usually start with for processing natural language text is breaking input text into individual words and sentences. Here is the code for using the tokenizing code that separates text stored as a Java String into individual words:

\n\n

1   public static String[] tokenize(String s) {\n2     return tokenizer.tokenize(s);\n3   }\n

\n\n\n\n

Here is the similar code for breaking text into individual sentences:

\n\n

1   public static String[] sentenceSplitter(String s) {\n2     return sentenceSplitter.sentDetect(s);\n3   }\n

\n\n\n\n

Here is some sample code to use sentenceSplitter:

\n\n

1     String sentence =\n2       \"Apple Computer, Microsoft, and Google are in the \" +\n3       \" tech sector. Each is very profitable.\";\n4     String [] sentences = NLP.sentenceSplitter(sentence);\n5     System.out.println(\"Sentences found:\\n\" + Arrays.toString(sentences));\n

\n\n\n\n

In line 4 the static method NLP.sentenceSplitter returns an array of strings. In line 5 I use a common Java idiom for printing arrays by using the static method Arrays.toString to convert the array of strings into a List<String> object. The trick is that the List class has a toString method that formats list nicely for printing.

\n\n

Here is the output of this code snippet (edited for page width and clarity):

\n\n

1 Sentences found:\n2   [\"Apple Computer, Microsoft, and Google are in the tech sector.\",\n3    \"Each is very profitable.\"]\n

\n\n\n\n

The code for finding organizations, locations, and people’s names is almost identical so I will only show the code in the next listing for recognizing locations. Please look at the methods companyNames and personNames in the class com.markwatson.opennlp.NLP to see the implementations for finding the names of companies and people.

\n\n

 1   public static Set<String> locationNames(String text) {\n 2     return locationNames(tokenizer.tokenize(text));\n 3   }\n 4 \n 5   public static Set<String> locationNames(String tokens[]) {\n 6     Set<String> ret = new HashSet<String>();\n 7     Span[] nameSpans = locationFinder.find(tokens);\n 8     if (nameSpans.length == 0) return ret;\n 9     for (int i = 0; i < nameSpans.length; i++) {\n10       Span span = nameSpans[i];\n11       StringBuilder sb = new StringBuilder();\n12       for (int j = span.getStart(); j < span.getEnd(); j++)\n13         sb.append(tokens[j] + \" \");\n14       ret.add(sb.toString().trim().replaceAll(\" ,\", \",\"));\n15     }\n16     return ret;\n17   }\n

\n\n\n\n

The public methods in the class com.markwatson.opennlp.NLP are overriden to take either a single string value which gets tokenized inside of the method and also a method that takes as input text that has already been tokenized into a String tokens[] object. In the last example the method starting on line 1 accepts an input string and the overriden method starting on line 5 accepts an array of strings. Often you will want to tokenize text stored in a single input string into tokens and reuse the tokens for calling several of the public methods in com.markwatson.opennlp.NLP that can take input that is already tokenized. In line 2 we simply tokenize the input text and call the method that accepts tokenized input text.

\n\n

In line 6 we create a HashSet<String> object that will hold the return value of a set of location names. The NameFinderME object locationFinder returns an array of Span objects. The SPan class is used to represnt a sequence of adjacent words. The **Span class has a public static attribute length and instance methods getstart and getEnd that return the indices of the beginning and ending (plus one) index of a span in the origianl input text.

\n\n

Here is some sample code to use locationNames along with the output (edited for page width and clarity):

\n\n

1    String sentence =\n2      \"Apple Computer is located in Cupertino, California and Microsoft \" +\n3      \"is located in Seattle, Washington. He went to Oregon\";\n4    Set<String> names = NLP.locationNames(sentence);\n5    System.out.println(\"Location names found: \" + names);\n6 \n7 Location names found: [Oregon, Cupertino, Seattle, California, Washington]\n

\n\n\n\n

Note that the pre-trained model does not recognize when city and state names are associated.

\n\n

Training a New Categorization Model for OpenNLP

\n\n

The OpenNLP class DoccatTrainer can process specially formatted input text files and produce categorization models using maximum entropy which is a technique that handles data with many features. Features that are automatically extracted from text and used in a model are things like words in a document and word adjacency. Maximum entropy models can recognize multiple classes. In testing a model on new text data the probablilities of all possible classes add up to the value 1 (this is often refered to as “softmax”). For example we will be training a classifier on four categories and the probablilities of these categories for some test input text add up to the value of one:

\n\n

[[COMPUTERS, 0.1578880260493374], [ECONOMY, 0.2163099374638936],\n [HEALTH, 0.35546925520388845], [POLITICS, 0.27033278128288035]]\n

\n\n\n\n

The format of the input file for training a maximum entropy classifier is simple but has to be correct: each line starts with a category name, followed by sample text for each category which must be all on one line. Please note that I have already trained the model producing the model file models/en-newscat.bin so you don’t need to run the example in this section unless you want to regenerate this model file.

\n\n

The file sample_category_training_text.txt contains four lines, defining four categories. Here are two lines from this file (I edited the following to look better on the printed page, but these are just two lines in the file):

\n\n

COMPUTERS Computers are often used to access the Internet and meet basic busines\\\ns functions.\nECONOMY The Austrian School (also known as the Vienna School or the Psychologica\\\nl School ) is a Schools of economic thought|school of economic thought that emph\\\nasizes the spontaneous organizing power of the price mechanism.\n

\n\n\n\n

Here is one training example each for the categories COMPUTERS and ECONOMY.

\n\n

You must format the training file perfectly. As an example, if you have empty (or blank) lines in your input training file then you will get an error like:

\n\n

Computing event counts...  java.io.IOException: Empty lines, or lines with\n                           only a category string are not allowed!\n

\n\n\n\n

The OpenNLP documentation has examples for writing custom Java code to build models but I usually just use the command line tool; for example:

\n\n

bin/opennlp DoccatTrainer -model models/en-newscat.bin -lang en \\\n                          -data sample_category_training_text.txt \\\n                          -encoding UTF-8\n

\n\n\n\n

The model is written to the relative file path models/en-newscat.bin. The training file I am using is tiny so the model is trained in a few seconds. For serious applications, the more training text the better! By default the DoccatTrainer tool uses the default text feature generator which uses word frequencies in documents but ignores word ordering. As I mention in the next section, I sometimes like to mix word frequency feature generation with 2gram (that is, frequencies of two adjacent words). In this case you cannot simply use the DoccatTrainer command line tool. You need to write a little Java code yourself that you can plug another feature generator into using the alternative API:

\n\n

public static DoccatModel train(String languageCode,\n                                ObjectStream<DocumentSample> samples,\n                                int cutoff, int iterations,\n                                FeatureGenerator... featureGenerators)\n

\n\n\n\n

As I also mention in the next section, the last argument would look like:

\n\n

public DocumentCategorizerME(DoccatModel model,\n           new FeatureGenerator[]{new BagOfWordsFeatureGenerator(),\n                                  new NGramFeatureGenerator()});\n

\n\n\n\n

For most purposes the default word frequency (or bag of words) feature generator is probably OK so using the command line tool is a good place to start. Here is the output from running the DoccatTrainer command line tool:

\n\n

> bin/opennlp DoccatTrainer -model models/en-newscat.bin -lang en \\\n                            -data sample_category_training_text.txt \\\n                            -encoding UTF-8\nIndexing events using cutoff of 5\n\n\tComputing event counts...  done. 4 events\n\tIndexing...  done.\nSorting and merging events... done. Reduced 4 events to 4.\nDone indexing.\nIncorporating indexed data for training...  \ndone.\n\tNumber of Event Tokens: 4\n\t    Number of Outcomes: 4\n\t  Number of Predicates: 153\n...done.\nComputing model parameters ...\nPerforming 100 iterations.\n  1:  ... loglikelihood=-5.545177444479562\t0.25\n  2:  ... loglikelihood=-4.730232204542369\t0.5\n  3:  ... loglikelihood=-4.232192673282495\t0.75\n   ...\n 98:  ... loglikelihood=-0.23411803835475453\t1.0\n 99:  ... loglikelihood=-0.23150121909902377\t1.0\n100:  ... loglikelihood=-0.22894028845170055\t1.0\nWriting document categorizer model ... done (0.041s)\n\nWrote document categorizer model to\npath: /Users/mark/power-java/opennlp/models/en-newscat.bin\n

\n\n\n\n

We will use our new trained model file en-newscat.bin in the next section.

\n\n

Please note that in this simple example I used very little data, just a few hundred words for each training category. I have used the OpenNLP maximum entropy library on various projects, mostly to good effect, but I used many thousands of words for each category. The more data, the better.

\n\n

Using Our New Trained Classification Model

\n\n

The code that uses the model we trained in the last section is short enough to list in its entirety:

\n\n

 1 package com.markwatson.opennlp;\n 2 \n 3 import opennlp.tools.doccat.DoccatModel;\n 4 import opennlp.tools.doccat.DocumentCategorizerME;\n 5 \n 6 import java.io.*;\n 7 import java.util.*;\n 8 \n 9 /**\n10  * This program uses the maximum entropy model we trained\n11  * using the instructions in the chapter on OpenNLP in the book.\n12  */\n13 \n14 public class NewsClassifier {\n15 \n16   public static List<Pair<String,Double>> allRankedCategories(String text) {\n17     DocumentCategorizerME aCategorizer = new DocumentCategorizerME(docCatModel);\n18     double[] outcomes = aCategorizer.categorize(text);\n19     List<Pair<String,Double>> results = new ArrayList<Pair<String, Double>>();\n20     for (int i=0; i<outcomes.length; i++) {\n21       results.add(new Pair<String, Double>(aCategorizer.getCategory(i), outcomes\\\n22 [i]));\n23     }\n24     return results;\n25   }\n26 \n27   public static String bestCategory(String text) {\n28     DocumentCategorizerME aCategorizer = new DocumentCategorizerME(docCatModel);\n29     double[] outcomes = aCategorizer.categorize(text);\n30     return aCategorizer.getBestCategory(outcomes);\n31   }\n32 \n33   public static Pair<String,Float> classifyWithScore(String text) {\n34     DocumentCategorizerME classifier = new DocumentCategorizerME(docCatModel);\n35     double [] scores = classifier.categorize(text);\n36     int num_categories = classifier.getNumberOfCategories();\n37     if (num_categories > 0) {\n38       String bestString =  classifier.getBestCategory(scores);\n39       for (int i=0; i<num_categories; i++) {\n40         if (classifier.getCategory(i).equals(bestString)) {\n41           return new Pair<String,Float>(bestString, (float)scores[i]);\n42         }\n43       }\n44     }\n45     return new Pair<String,Float>(\"<no category>\", 0f);\n46   }\n47   \n48   static DoccatModel docCatModel = null;\n49 \n50   static {\n51 \n52     try {\n53       InputStream modelIn = new FileInputStream(\"models/en-newscat.bin\");\n54       docCatModel = new DoccatModel(modelIn);\n55     } catch (IOException e) {\n56       e.printStackTrace();\n57     }\n58   }\n59 }\n

\n\n\n\n

In lines 33 through 42 we initialize the static data for an instance of the class DoccatModel that loads the model file created in the last section.

\n\n

A new instance of the class DocumentCategorizerME is created in line 28 each time we want to classify input text. I called the one argument constructor for this class that uses the default feature detector. An alternative constructor is:

\n\n

1 public DocumentCategorizerME(DoccatModel model,\n2                              FeatureGenerator... featureGenerators)\n

\n\n\n\n

The default feature generator is BagOfWordsFeatureGenerator which just uses word frequencies for classification. This is reasonable for smaller training sets as we used in the last section but when I have a large amount of training data available I prefer to combine BagOfWordsFeatureGenerator with NGramFeatureGenerator. You would use the constructor call:

\n\n

1 public DocumentCategorizerME(DoccatModel model,\n2            new FeatureGenerator[]{new BagOfWordsFeatureGenerator(),\n3                                   new NGramFeatureGenerator()});\n

\n\n\n\n

The following listings show interspersed both example code snippets for for using the NewsClassifier class followed by the output printed by each code snippet:

\n\n

    String sentence = \"Apple Computer, Microsoft, and Google are in the tech sec\\\ntor. Each is very profitable.\";\n    System.out.println(\"\\nNew test sentence:\\n\\n\" + sentence + \"\\n\");\n    String [] sentences = NLP.sentenceSplitter(sentence);\n    System.out.println(\"Sentences found: \" + Arrays.toString(sentences));\n\nNew test sentence:\n\nApple Computer, Microsoft, and Google are in the tech sector.\nEach is very profitable.\n\nSentences found: [Apple Computer, Microsoft, and Google are in the tech sector.,\n                  Each is very profitable.]\n

\n\n\n\n

    sentence = \"Apple Computer, Microsoft, and Google are in the tech sector.\";\n    Set<String> names = NLP.companyNames(sentence);\n    System.out.println(\"Company names found: \" + names);\n\nCompany names found: [Apple Computer, Microsoft]\n\n    sentence = \"Apple Computer is located in Cupertino, California and Microsoft\\\n is located in Seattle, Washington. He went to Oregon\";\n    System.out.println(\"\\nNew test sentence:\\n\\n\" + sentence + \"\\n\");\n    Set<String> names1 = NLP.locationNames(sentence);\n    System.out.println(\"Location names found: \" + names1);\n\nNew test sentence:\n\nApple Computer is located in Cupertino, California and Microsoft is located in S\\\neattle, Washington. He went to Oregon\n\nLocation names found: [Oregon, Cupertino, Seattle, California, Washington]\n

\n\n\n\n

    sentence = \"President Bill Clinton left office.\";\n    System.out.println(\"\\nNew test sentence:\\n\\n\" + sentence + \"\\n\");\n    Set<String> names2 = NLP.personNames(sentence);\n    System.out.println(\"Person names found: \" + names2);\n\nNew test sentence:\n\nPresident Bill Clinton left office.\n\nPerson names found: [Bill Clinton]\n\n    sentence = \"The White House is often mentioned in the news about U.S. foreig\\\nn policy. Members of Congress and the President are worried about the next elect\\\nion and may pander to voters by promising tax breaks. Diplomacy with Iran, Iraq,\\\n and North Korea is non existent in spite of a worry about nuclear weapons. A un\\\ni-polar world refers to the hegemony of one country, often a militaristic empire\\\n. War started with a single military strike. The voting public wants peace not w\\\nar. Democrats and Republicans argue about policy.\";\n    System.out.println(\"\\nNew test sentence:\\n\\n\" + sentence + \"\\n\");\n    List<Pair<String, Double>> results = NewsClassifier.allRankedCategories(sent\\\nence);\n    System.out.println(\"All ranked categories found: \" + results);\n\nNew test sentence:\n\nThe White House is often mentioned in the news about  U.S. foreign policy. Membe\\\nrs of Congress and the President are worried about the next election and may pan\\\nder to voters by promising tax breaks. Diplomacy with Iran, Iraq, and North Kore\\\na is non existent in spite of a worry about nuclear weapons. A uni-polar world r\\\nefers to the hegemony of one country, often a militaristic empire. War started w\\\nith a single military strike. The voting public wants peace not war. Democrats a\\\nnd Republicans argue about policy.\n\nAll ranked categories found: [[COMPUTERS, 0.07257665117660535], [ECONOMY, 0.2355\\\n9821969425127], [HEALTH, 0.29907267186308945], [POLITICS, 0.39275245726605396]]\n

\n\n\n\n

    sentence = \"The food affects colon cancer and ulcerative colitis. There is s\\\nome evidence that sex helps keep us healthy. Eating antioxidant rich foods may p\\\nrevent desease. Smoking may raise estrogen levels in men and lead to heart failu\\\nre.\";\n    System.out.println(\"\\nNew test sentence:\\n\\n\" + sentence + \"\\n\");\n    String results2 = NewsClassifier.bestCategory(sentence);\n    System.out.println(\"Best category found found (HEALTH): \" + results2);\n    List<Pair<String, Double>> results3 = NewsClassifier.allRankedCategories(sen\\\ntence);\n    System.out.println(\"All ranked categories found (HEALTH): \" + results3);\n    System.out.println(\"Best category with score: \" + NewsClassifier.classifyWit\\\nhScore(sentence));\n\nNew test sentence:\n\nThe food affects colon cancer and ulcerative colitis. There is some evidence tha\\\nt sex helps keep us healthy. Eating antioxidant rich foods may prevent desease. \\\nSmoking may raise estrogen levels in men and lead to heart failure.\n\nBest category found found (HEALTH): HEALTH\nAll ranked categories found (HEALTH): [[COMPUTERS, 0.1578880260493374], [ECONOMY\\\n, 0.2163099374638936], [HEALTH, 0.35546925520388845], [POLITICS, 0.2703327812828\\\n8035]]\nBest category with score: [HEALTH, 0.35546926]\n

\n\n\n\n

Using the OpenNLP Parsing Model

\n\n

We will use the parsing model that is included in the OpenNLP distribution to parse English language input text. You are unlikely to use a statistical parsing model in your work but I think you will enjoy the material in the section. If you are more interested in practical techniques then skip to the next chapter that covers more machine learning techniques.

\n\n

The following example code listing is long (68 lines) but I will explain the interesting parts after the listing:

\n\n

 1 package com.markwatson.opennlp;\n 2 \n 3 import opennlp.tools.cmdline.parser.ParserTool;\n 4 import opennlp.tools.parser.Parse;\n 5 import opennlp.tools.parser.ParserModel;\n 6 import opennlp.tools.parser.chunking.Parser;\n 7 \n 8 import java.io.FileInputStream;\n 9 import java.io.IOException;\n10 import java.io.InputStream;\n11 \n12 /**\n13  * Experiments with the chunking parser model\n14  */\n15 public class ChunkingParserDemo {\n16   public Parse[] parse(String text) {\n17     Parser parser = new Parser(parserModel);\n18     return ParserTool.parseLine(text, parser, 5);\n19   }\n20 \n21   public void prettyPrint(Parse p) {\n22     StringBuffer sb = new StringBuffer();\n23     p.show(sb);\n24     String s = sb.toString() + \" \";\n25     int depth = 0;\n26     for (int i = 0, size = s.length() - 1; i < size; i++) {\n27       if (s.charAt(i) == ' ' && s.charAt(i + 1) == '(') {\n28         System.out.print(\"\\n\");\n29         for (int j = 0; j < depth; j++) System.out.print(\"  \");\n30       } else if (s.charAt(i) == '(') System.out.print(s.charAt(i));\n31       else if (s.charAt(i) != ')' || s.charAt(i + 1) == ')')\n32         System.out.print(s.charAt(i));\n33       else {\n34         System.out.print(s.charAt(i));\n35         for (int j = 0; j < depth; j++) System.out.print(\"  \");\n36       }\n37       if (s.charAt(i) == '(') depth++;\n38       if (s.charAt(i) == ')') depth--;\n39     }\n40     System.out.println();\n41   }\n42 \n43   static ParserModel parserModel = null;\n44 \n45   static {\n46 \n47     try {\n48       InputStream modelIn = new FileInputStream(\"models/en-parser-chunking.bin\");\n49       parserModel = new ParserModel(modelIn);\n50     } catch (IOException e) {\n51       e.printStackTrace();\n52     }\n53   }\n54 \n55   public static void main(String[] args) {\n56     ChunkingParserDemo cpd = new ChunkingParserDemo();\n57     Parse[] possibleParses =\n58       cpd.parse(\"John Smith went to the store on his way home from work \");\n59     for (Parse p : possibleParses) {\n60       System.out.println(\"parse:\\n\");\n61       p.show();\n62       //p.showCodeTree();\n63       System.out.println(\"\\npretty printed parse:\\n\");\n64       cpd.prettyPrint(p);\n65       System.out.println(\"\\n\");\n66     }\n67   }\n68 }\n

\n\n\n\n

The OpenNLP parsing model is read from a file in lines 45 through 53. The static variable parserModel (instance of class ParserModel) if created in line 49 and used in lines 17 and 18 to parse input text. It is instructive to look at the intermediate calculation results. The value for variable parser defined in line 17 has a value of:

\n\n\n

\"Data — Data in the parser model as seen in IntelliJ

\n\n\n

Note that the parser returned 5 results because we specified this number in line 18. For a long sentence the parser generates a very large number of possible parses for the sentence and returns, in order of probability of being correct, the number of resutls we requested.

\n\n

The OpenNLP chunking parser code prints out results in a flat list, one result on a line. This is difficult to read which is why I wrote the method prettyPrint (lines 21 through 41) to print the parse results indented. Here is the output from the last code example (the first parse shown is all on one line but line wraps in the following listing):

\n\n

 1 parse:\n 2 \n 3 (TOP (S (NP (NNP John) (NNP Smith)) (VP (VBD went) (PP (TO to) (NP (DT the) (NN \\\n 4 store))) (PP (IN on) (NP (NP (NP (PRP$ his) (NN way)) (ADVP (NN home))) (PP (IN \\\n 5 from) (NP (NN work))))))))\n 6 \n 7 pretty printed parse:\n 8 \n 9 (TOP\n10   (S\n11     (NP\n12       (NNP John)        \n13       (NNP Smith))      \n14     (VP\n15       (VBD went)        \n16       (PP\n17         (TO to)          \n18         (NP\n19           (DT the)            \n20           (NN store)))        \n21       (PP\n22         (IN on)          \n23         (NP\n24           (NP\n25             (NP\n26               (PRP$ his)                \n27               (NN way))              \n28             (ADVP\n29               (NN home)))            \n30           (PP\n31             (IN from)              \n32             (NP\n33               (NN work))))))))\n

\n\n\n\n

In the 1980s I spent much time on syntax level parsing. While the example in this section is a statistical parsing model I don’t find these models very relevant to my own work but I wanted to include a probalistic parsing example for completeness in this chapter.

\n\n

OpenNLP is a great resource for Java programmers and its Apache 2 license is “business friendly.” If you can use software with a GPL license then please also look at the Stanford NLP libraries.

\n\n\n

Machine Learning Using Apache Spark

\n\n

I have used Java for Machine Learning (ML) for nearly twenty years using my own code and also great open source projects like Wika, Mahout, and Mallet. For the material in this chapter I will use a relatively new open source machine learning library MLlib that is included in the Apache Spark project. Spark is similar to Hadoop in providing functionality for using multiple servers for processing massively large data sets. Spark preferentially deals with data in memory and can provide near real time analysis of large data sets.

\n\n

We saw a specific use case of machine learning in the previous chapter on OpenNLP: building maximum entropy classification models. This chapter will introduce another good ML library. Machine learning is part of the fields of data science and artificial intelligence.

\n\n

There are several steps involved in building applications with ML:

\n\n

Understand what problems your organization has and what data is available to solve these problems. In other words, figure out where you should you spend your effort. You should also understand early in the process how it is that you will evaluate the usefulness and quality of the results.
Collecting data. This step might involve collecting numeric time series data from instruments; for example, to train regression models or collecting text for natural language processing ML projects as we saw in the last chapter.
Cleaning data. This might involve detecting and removing errors in data from faulty sensors or incomplete collection of text data from the web. In some cases data might be annotated (or labelled) in this step.
Integrating data from different sources and organizing it for experiments. Data that has been collected and cleaned might end up being used for multiple projects so it makes sense to organize it for ease of use. This might involve storing data on disk or cloud storage like S3 with meta data stored in a database. Meta data could include date of collection, data source, steps taken to clean the data, and references to models already build using this data.
\nThe main topic of this chapter: analysing data and creating models with machine learning (ML). This will often be experimental with many different machine learning algorithms used for quick and dirty experiments. These fast experiments should inform you on which approach works best for this data set. Once you decide which ML algorithm to use you can spend the time to refine features in data used by learning algorithms and to tune model parameters. It is usually best, however, to first quickly try a few approaches before settling on one approach and then investing a lot of time in building a model.
Interpreting and using the results of models. Often models will be used to process other data sets to automatically label data (as we do in this chapter with text data from Wikipedia) or making predictions from new sources of time series numerical data.
Deploy training models as embedded parts of a larger system. This is an optional step - sometimes it is sufficient to use a model to understand a problem well enough to know how to solve it. Usually, however, a model will be used repeatadly, often automatically, as part of normal data processing and decision workflows. I will take some care in this chapter to indicate how to reuse models embedded in your Java applications.

\n\n

In the examples in this book I have supplied test data with the example programs. Please keep in mind that although I have listed these steps for data mining and ML model building as a sequence of steps, in practice this is a very iterative process. Data collection and cleaning might be reworked after seeing preliminary results of building models using different algorithms. We might iteratively tune model parameters and also improve models by adding or removing the features in data that are used to build models.

\n\n

What do I mean by “features” of data? For text data features can be: words in text, collections of adjacent words in text (ngrams), structure of text (e.g., taking advantage of HTML or other markup), punctuation, etc. For numeric data features might be minimum and maximum of data points in a sequence, features in frequency space (calculated by Fast Forier Transfroms (FFTs), sometimes as Power Spectral Density (PSD) which is a FFT with phase information in the data discarded), etc. Cleaning data means different things for different types of data. In cleaning text data we might segment text into sentences, perform word stemming (i.e., map words like “banking” or “banks” to their stem of root “bank”), sometimes we might want to convert all text to lower case, etc. For numerical data we might have to scale data attributes to a specific numeric range - we will do this later in this chapter when using the University of Wisconsin cancer data sets.

\n\n

The previous seven steps that I outline are my view of the process of discovering information sources, processing the data, and producing actionable knowledge from data. Other acronyms that you are likely to run across is Knowledge Discovery in Data (KDD) and the CCC Big Data Pipeline which both use steps similar to those that I have just outlined.

\n\n

There are, broadly speaking, four types of ML models:

\n\n

Regression - predicts a numerical value. Inputs are numeric values.
Classification - predicts yes or no. Inputs are numeric values.
Clustering - partition a set of data items into disjoint sets. Data items might be text documents, numerical data sets from sensor readings, sets of structured data that might include text and numbers. Inputs must be converted to numeric values.
Recommendation - decision to recommend an object based on a user’s previous actions and/or by matching a user to other similar users to use other users’ actions.

\n\n

You will see later when dealing with text that we need to convert words to numbers (each word is assigned a unique numeric ID). A document is represented by a sparse vector that has a non-empty element for each unique word index. Further, we can divide machine learning problems into two general types:

\n\n

Supervised learning - input features used for training are labeled with the desired prediction. The example of supervised learning we will look at is building a regresion model using cancer symptom data to predict the malignancy vs. benign given a new input feature set.
Unsupervised learning - input features are not labeled. Examples of unsupervised learning that we will look at are: K-Means similar Wikipedia document clustering and using word2vec to find related words in documents.

\n\n

My goal in this chapter is to introduce you to practical techniques using Spark and machine learning algorithms. After you have worked through the material I recommend reading through the Spark documentation to see a wider variety of available models to use. Also consider taking Andrew Ng’s excellent Machine Learning class.

\n\n

Scaling Machine Learning

\n\n

While the ability to scale data mining and ML operations for very large data sets is sometimes important, for development and for working through the examples in this chapter you do not need to set up a cluster of servers to run Spark. We will use Spark in developer mode and you can run the examples on your laptop. It would be best if your laptop has at least 2 gigabytes of RAM to run the ML examples. The important thing to remember is that the techniques that you learn in this chapter can be scaled to very large data sets as needed without modifying your application source code.

\n\n

The Apache Spark project originated at the University of California at Berkeley. Spark has client support for Java, Scala, and Python. The examples in this chapter will all be in Java. If you are a Scala developer, after working through through the examples you might want to take a look at the Scala Spark documentation since Spark is written in Scala and Scala has the best client support. The Python Spark and MLlib APIs are also very nice to use so if you are a Python developer you should check out this Python support.

\n\n

Spark is an evolutionary step forward from using map reduce systems like Hadoop or Google’s Map Reduce. In following through the examples in this chapter you will likely think that you could accomplish the same effect with less code and runtime overhead. For dealing with small amounts of data that is certainly true but, just like using Hadoop, a little more verbosity in the code gives you the ability to scale your applications over large data sets. We put up with extra complexity for scalability. Spark has utility classes for reading and transforming data that make the example programs simpler than writing code yourself for importing data.

\n\n

I suggest that you download the source code for the Spark 2.0.0 distribution from the Apache Spark download page since there are many example programs that you can use for ideas and for reference. The Java client examples in the Spark machine learning library MLlib can be found in the directory:

\n\n

spark-2.0.0/examples/src/main/java/org/apache/spark/examples/mllib\n

\n\n\n\n

If you download the binary distribution, then the examples are in:

\n\n

spark-2.0.0-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib\n

\n\n\n\n

Where the directory spark-2.0.0 was created when you downloaded Spark source code distribution version 2.0.0.

\n\n

These examples use provided data files that provide numerical data, each line specifies a local Spark vector data type. The first K-Means clustering example in this chapter is derived from a MLlib example included with the Spark distribution. The second K-Means example that clusters Wikipedia articles is quite a bit different because we need to write new code to convert text into numeric feature vectors. I provide sample text that I manually selected from Wikipedia articles.

\n\n

We will start this chapter by showing you how to set up Spark and MLlib on your laptop and then look at a simple “Hello World” introductory type program for generating word counts from input text. We will then implement logistic regression and also take a brief look at the JavaKMeans example program from the Spark MLlib examples and use it to understand the requirement for reducing data to a set of numeric feature vectors. We will then develop examples using Wikipedia data and this will require writing custom converters for plain text to numeric feature vectors.

\n\n

Setting Up Spark On Your Laptop

\n\n

Assuming that you have cloned the github project for this book, the Spark examples are in the sub-directory machine_learning_spark. The maven pom.xml file contains the Spark dependencies. You can install these dependencies using:

\n\n

mvn clean install\n

\n\n\n\n

There are four examples in this chapter. If you want to skip ahead and run the code before reading the rest of the chapter, run each maven test separately:

\n\n

mvn test -Dtest=HelloTest\nmvn test -Dtest=KMeansTest\nmvn test -Dtest=WikipediaKMeansTest\nmvn test -Dtest=LogisticRegressionTest\nmvn test -Dtest=SvmClassifierTest\n

\n\n\n\n

Spark has all of Hadoop as a dependency because Spark is able to interact with Hadoop and Hadoop file systems. It will take a short while to download all of these dependencies but that only needs to be done one time. To augment the material in this chapter you can refer to the Spark Programming Guide. We will also be using material in the Spark Machine Learning Library (MLlib) Guide.

\n\n

The following figure shows the project for this chapter in the Community Edition of IntelliJ:

\n\n\n

Hello Spark - a Word Count Example

\n\n

The word count example is frequently used to introduce map reduce and I will also use it here for a “hello world” type simple Spark example. In the following listing, lines 3 through 6 import the required classes. In lines 3 and 4 “RDD” stands for Resilient Distributed Datasets (RDD). Spark was written in Scala so we will sometimes see Java wrapper classes that are recognizable with the prefix “Java.” Each RDD is a collection of objects of some type that can be distributed across multiple servers in a Spark cluster. RDDs are fault tolerant and there is sufficient information in a Spark cluster to recreate any lost data due to a server or software failure.

\n\n

 1 package com.markwatson.machine_learning;\n 2 \n 3 import org.apache.spark.api.java.JavaPairRDD;\n 4 import org.apache.spark.api.java.JavaRDD;\n 5 import org.apache.spark.api.java.JavaSparkContext;\n 6 import scala.Tuple2;\n 7 \n 8 import java.util.Arrays;\n 9 import java.util.List;\n10 import java.util.Map;\n11 \n12 public class HelloSpark {\n13 \n14   static private List<String> tokenize(String s) {\n15     return Arrays.asList(s.replaceAll(\"\\\\.\", \" \\\\. \").replaceAll(\",\", \" , \")\n16         .replaceAll(\";\", \" ; \").split(\" \"));\n17   }\n18 \n19   static public void main(String[] args) {\n20     JavaSparkContext sc = new JavaSparkContext(\"local\", \"Hello Spark\");\n21 \n22     JavaRDD<String> lines = sc.textFile(\"data/test1.txt\");\n23     JavaRDD<String> tokens = lines.flatMap(line -> tokenize(line));\n24     JavaPairRDD<String, Integer> counts =\n25         tokens.mapToPair(token ->\n26 \t\t\t new Tuple2<String, Integer>(token.toLowerCase(), 1))\n27               .reduceByKey((count1, count2) -> count1 + count2);\n28     Map countMap = counts.collectAsMap();\n29     System.out.println(countMap);\n30     List<Tuple2<String, Integer>> collection = counts.collect();\n31     System.out.println(collection);\n32   }\n33 }\n

\n\n\n\n

Lines 14 through 17 define the simple tokenizer method that takes English text and maps it to a list of strings. I broke this out into a separate method as a reminder that punctuation needs to be handled and breaking it in to a separate method makes it easier for you to experiment with custom tokenization that might be needed for your particular text input files. One possibility is using the OpenNLP tokenizer model that we used in the last chapter. For example, we would want the string “tree.” to be tokenized as [“tree”, “.”] and not [“tree.”]. This method is used in line 23 and is called for each line in the input text file.

\n\n

In this example in line 22 we are reading a local text file “data/test1.txt” but in general the argument to JavaSparkContext.textFile() can be a URI specifying a file on a local Hadoop file system (HDFS), an Amazon S3 object, of a Cassandra file system. Pay some attention to the type of the variable lines defined in line 22: the type JavaRDD<String> specifies an RDD collection where the distributed objects are strings.

\n\n

The code in lines 29 through 32 is really not very complicated if we parse it out a bit at a time. The type of counts is a RDD distributed collection where each object in the collection is a pair of string and integer values. The code in lines 25 through 27 takes each token in the original text file and produces a pair (or Tuple2) values consisting of the token string and the integer 1. As an example, an array of three tokens like:

\n\n

1     [\"the\", \"dog\", \"ran\"]\n

\n\n\n\n

would be converted to an array of three pairs like:

\n\n

1     [[\"the\", 1], [\"dog\", 1], [\"ran\", 1]]\n

\n\n\n\n

The statement in line 28 is uses the method reduceByKey which partitions the array of pairs by unique key value (the key being the first element in each pair) and sums the values for each matching key. The rest of this example shows two ways to extract data from an distributed RDD collection back to a Java process as local data. In line 29 we are pulling the collection of word counts into a Java Map<String,Integer> collection and in line 31 we are doing the same thing but pulling data into a list of Scala tuples. Here is what the output of the program looks like:

\n\n

1     {dog=2, most=1, of=1, down=1, rabbit=1, slept=1, .=2, the=5, street=1,\n2      chased=1, day=1}\n3     [(most,1), (down,1), (.,2), (street,1), (day,1), (chased,1), (dog,2),\n4      (rabbit,1), (of,1), (slept,1), (the,5)]\n

\n\n\n\n

Introducing the Spark MLlib Machine Learning Library

\n\n

As I update this Chapter in the spring of 2016, the Spark MLlib library is at version 2.0. I will update the git repository for this book periodically and I will try to keep the code examples compatible with the current stable versions of Spark and MLlib.

\n\n

The examples in the following sections show how to apply the MLlib library to problems of logistic regression, using the K-Means algoirthm to cluster similar documents, and using the word2vec algorithm to find strong associations between words in documents.

\n\n

MLlib Logistic Regression Example Using University of Wisconsin Cancer Database

\n\n

We will use supervised learning in this section to build a regresion model that maps a numeric feature vector of symptoms to a numeric value where larger values close to one indicate malignancy and smaller values closer to zero indicate benign.

\n\n

MLlib supports three types of linear models: Support Vector Machine (SVM), logistic regression (models for membership in one or more classes or categories), and linear regression (models that model outputs for a class as a floating point number in the range [0..1]). The Spark MLlib documentation (http://spark.apache.org/docs/latest/mllib-linear-methods.html) on these three linear models is very good and their sample programs are similar so it is easy enough to switch the type of linear model that you use.

\n\n

We will use logistic regression for the example in this section. Andrew Ng, in his excellent Machine Learning class said that the first model he tries when starting a new problem is a linear model. We will repeat this example in the next section using SVM instead of logistic regression.

\n\n

If you read my book Build Intelligent Systems with Javascript the training and test dataset used in this section will look familiar: University of Wisconsin breast cancer data set.

\n\n

From the University of Wisconsin web page, here are the attributes with their allowed ranges in this data set:

\n\n

- 0 Sample code number            integer\n- 1 Clump Thickness               1 - 10\n- 2 Uniformity of Cell Size       1 - 10\n- 3 Uniformity of Cell Shape      1 - 10\n- 4 Marginal Adhesion             1 - 10\n- 5 Single Epithelial Cell Size   1 - 10\n- 6 Bare Nuclei                   1 - 10\n- 7 Bland Chromatin               1 - 10\n- 8 Normal Nucleoli               1 - 10\n- 9 Mitoses                       1 - 10\n- 10 Class (2 for benign, 4 for malignant)\n

\n\n\n\n

The file data/university_of_wisconson_data_.txt contains raw data. We will use this sample data set with two changes: we clean it by removing the sample code column and the next 9 values are scaled to the range [0..1] and the class is mapped to 0 (benign) or 1 (malignant). So, our input features are scaled (values between [0 and 1] numerical values indicating clump thickness, uniformity of cell size, etc. The label that specifies what the desired prediction is the class (2 for for benign, 4 for malignant which we scale to 0 or 1)). In the following listing data is read from the training file and scaled in lines 27 through 44. Here I am using some utility methods provided by MLlib. In line 28 I am using the method JavaSparkContext.textFile to read the text file into what is effectively a list of strings (one string per input line). Lines 29 through 44 is creating a list of instances of the class LabeledPoint which contains a training label with associated numeric feature data. The container class JavaRDD acts as a list of elements that can be automatically managed across a Spark server cluster. When in development mode with a local JavaSparkContext everything is running in a single JVM so an instance of class JavaRDD behaves a lot like a simple in-memory list.

\n\n

 1 package com.markwatson.machine_learning;\n 2 \n 3 import org.apache.spark.api.java.JavaDoubleRDD;\n 4 import org.apache.spark.api.java.JavaRDD;\n 5 import org.apache.spark.api.java.JavaSparkContext;\n 6 import org.apache.spark.api.java.function.Function;\n 7 import org.apache.spark.mllib.linalg.Vectors;\n 8 import org.apache.spark.mllib.regression.LabeledPoint;\n 9 import org.apache.spark.mllib.regression.LinearRegressionModel;\n10 import org.apache.spark.mllib.regression.LinearRegressionWithSGD;\n11 import scala.Tuple2;\n12 \n13 public class LogisticRegression {\n14 \n15   public static double testModel(LinearRegressionModel model,\n16                                  double [] features) {\n17     org.apache.spark.mllib.linalg.Vector inputs = Vectors.dense(features);\n18     return model.predict(inputs);\n19   }\n20 \n21   public static void main(String[] args) {\n22     JavaSparkContext sc =\n23 \t  new JavaSparkContext(\"local\",\n24 \t                       \"University of Wisconson Cancer Data\");\n25 \n26     // Load and parse the data\n27     String path = \"data/university_of_wisconson_data_.txt\";\n28     JavaRDD<String> data = sc.textFile(path);\n29     JavaRDD<LabeledPoint> parsedData = data.map(\n30         new Function<String, LabeledPoint>() {\n31           public LabeledPoint call(String line) {\n32             String[] features = line.split(\",\");\n33             double label = 0;\n34             double[] v = new double[features.length - 2];\n35             for (int i = 0; i < features.length - 2; i++)\n36               v[i] = Double.parseDouble(features[i + 1]) * 0.09;\n37             if (features[10].equals(\"2\"))\n38               label = 0; // benign\n39             else\n40               label = 1; // malignant\n41             return new LabeledPoint(label, Vectors.dense(v));\n42           }\n43         }\n44     );\n45     // Split initial RDD into two data sets with 70% training data\n46     // and 30% testing data (13L is a random seed):\n47     JavaRDD<LabeledPoint>[] splits =\n48 \t  parsedData.randomSplit(new double[]{0.7, 0.3}, 13L);\n49     JavaRDD<LabeledPoint> training = splits[0].cache();\n50     JavaRDD<LabeledPoint> testing = splits[1];\n51     training.cache();\n52 \n53     // Building the model\n54     int numIterations = 100;\n55     final LinearRegressionModel model =\n56         LinearRegressionWithSGD.train(JavaRDD.toRDD(training),\n57 \t\t                              numIterations);\n58 \n59     // Evaluate model on training examples and compute training error\n60     JavaRDD<Tuple2<Double, Double>> valuesAndPreds = testing.map(\n61         new Function<LabeledPoint, Tuple2<Double, Double>>() {\n62           public Tuple2<Double, Double> call(LabeledPoint point) {\n63             double prediction = model.predict(point.features());\n64             return new Tuple2<Double, Double>(prediction, point.label());\n65           }\n66         }\n67     );\n68     double MSE = new JavaDoubleRDD(valuesAndPreds.map(\n69         new Function<Tuple2<Double, Double>, Object>() {\n70           public Object call(Tuple2<Double, Double> pair) {\n71             return Math.pow(pair._1() - pair._2(), 2.0);\n72           }\n73         }\n74     ).rdd()).mean();\n75     System.out.println(\"training Mean Squared Error = \" + MSE);\n76 \n77     // Save and load model and test:\n78     model.save(sc.sc(), \"generated_models\");\n79     LinearRegressionModel loaded_model =\n80 \t  LinearRegressionModel.load(sc.sc(), \"generated_models\");\n81     double[] malignant_test_data_1 =\n82 \t  {0.81, 0.6, 0.92, 0.8, 0.55, 0.83, 0.88, 0.71, 0.81};\n83     System.err.println(\"Should be malignant: \" + \n84 \t                    testModel(loaded_model, malignant_test_data_1));\n85     double[] benign_test_data_1 =\n86 \t  {0.55, 0.25, 0.34, 0.31, 0.29, 0.016, 0.51, 0.01, 0.05};\n87     System.err.println(\"Should be benign (close to 0.0): \" +\n88                        testModel(loaded_model, benign_test_data_1));\n89   }\n90 }\n

\n\n\n\n

Once the data is read and converted to numeric features the next thing we do is to randomly split the data in the input data file into two separate sets: one for training and one for testing our model after training. This data splitting is done in lines 47 and 48. Line 49 copies the training data and requests to the current Spark context to try to keep this data cached in memory. In line 50 we copy the testing data but don’t specify that the test data needs to be cached in memory.

\n\n

There are two sections of code left to discuss. In lines 53 through 75 I am training the model, saving the model to disk, and testing the model while it is still in memory. In lines 77 through 89 I am reloading the saved model from disk and showing how to use it with two new numeric test vectors. This last bit of code is meant as a demonstration of embedding a model in Java code and using it.

\n\n

The following three lines of code are the output of this example program. The first line of output indicates that the error for training the model is low. The last two lines show the output from reloading the model from disk and using it to evaluate two new numeric feature vectors:

\n\n

Test Data Mean Squared Error = 0.050140726106582885\nShould be malignant (close to 1.0): 1.1093117757012294\nShould be benign (close to 0.0): 0.29575663511929096\n

\n\n\n\n

This example shows a common pattern: reading training data from disk, scaling it, and building a model that is saved to disk for later use. This program can almost be reused as-is for new types of training data. You should just have to modify lines 27 through 44.

\n\n

MLlib SVM Classification Example Using University of Wisconsin Cancer Database

\n\n

The example in this section is very similar to that in the last section. The only differences are using the MLlib class SvmClassifier instead of the class LogisticRegression that we used in the last section and setting the number of training iterations to 500. We are using the same University of Wisconsin cancer data set in this section - refer to the previous section for a discusion of this data set.

\n\n

Since the code for this section is very similar to the last example I will not list it. Please refer to the source file SvmClassifier.java and the test file SvmClassifier.java in the github repository for this book. In the repository subdirectory machine_learning_spark you will need to remove the model files generated when running the example in the previous section before running this example:

\n\n

rm -r -f generated_models/*\nmvn test -Dtest=SvmClassifierTest\n

\n\n\n\n

The output for this example will look like:

\n\n

Test Data Mean Squared Error = 0.22950819672131167\nShould be malignant (close to 1.0): 1.0\nShould be benign (close to 0.0): 0.0\n

\n\n\n\n

Note that the test data mean square error is larger than the value of 0.050140726106582885 that we obtained using logistic regression in the last section.

\n\n

MLlib K-Means Example Program

\n\n

We will use unsupervised learning in this section, specifically the K-Means clustering algorithm. This clustering process partitions inputs into groups that share common features. Later in this chapter we will look at an example where our input will be some randomly chosen Wikipedia articles.

\n\n

K-Means analysis is useful for many types of data where you might want to cluster members of some set into different sub-groups. As an example, you might be a biologist studying fish. For each species of fish you might have attributes like length, weight, location, fresh/salt water, etc. You can use K-Means to divide fish specifies into similar sub-groups.

\n\n

If your job deals with marketing to people on social media, you might want to cluster people (based on what attributes you have for users of social media) into sub-groups. If you have sparse data on some users, you might be able to infer missing data on other users.

\n\n

The simple example in this section uses numeric feature vectors. Keep in mind, however our goal of processing Wikipedia text in a later section. In the next section we will see how to convert text to feature vectors, and finally, in the section after that we will cluster Wikipedia articles.

\n\n

Implementing K-Means is simple but using the MLlib implementation has the advantage of being able to process very large data sets. The first step in calculating K-Means is choosing the number of desired clusters NC. The calculate NC cluster centers randomly. We then perform an inner loop until either cluster centers stop changing or a specified number of iteractions is done:

\n\n

For each data point, assign it to the cluster center closest to it.
for each cluster center, move the center to the average location of the data points in that cluster

\n\n

In practice you will repeat this clustering process many times and use the cluster centers with the minimum distortion. The distortion is the sum of the squares of the distances from each data point in a cluster to the final cluster center.

\n\n

So before we look at the Wikipedia example (clustering Wikipedia articles) later in this chapter, I want to first review the JavaKMeansExample.java example from the Spark MLlib examples directory to see how to perform K-Means clustering on sample data (from the Spark source code distribution):

\n\n

spark-2.0.0/examples/src/main/java/org/apache/spark/examples/mllib/\n

\n\n\n\n

This is a very simple example that already uses numeric data. I copied the JavaKMeansExample.java file (changing the package name) to the github repository for his book.

\n\n

The most important thing in this section that you need to learn is the required types of input data: vectors of double size floating point numbers. Each sample of data needs to be converted to a numeric feature vector. For some applications this is easy. As a hypothetical example, you might have a set of features that measure weather in your town: low temperature, high temperature, average temperature, percentage of sunshine during the day, and wind speed. All of these features are numeric and can be used as a feature vector as-is. You might have 365 samples for a given year and would like to cluster the data to see which days are similar to each other.

\n\n

What if in this hypothetical example one of the features was a string representing the month? One obvious way to convert the data would be to map “January” to 1, “February” to 2, etc. We will see in the next section when we cluster Wikipedia articles that converting data to a numeric feature vector is not always so easy.

\n\n

We will use the tiny sample data set provided with the MLlib KMeans example:

\n\n

0.0 0.0 0.0\n0.1 0.1 0.1\n0.2 0.2 0.2\n9.0 9.0 9.0\n9.1 9.1 9.1\n9.2 9.2 9.2\n

\n\n\n\n

Here we have 6 samples and each sample has 3 features. From inspection, it looks like there are two clusters of data: the first three rows are in one cluster and the last three rows are in a second cluster. The order that feature vectors are processed in is not important; the samples in this file are organized so you can easily see the two clusters. When we run this sample program asking for two clusters we get:

\n\n

Cluster centers:\n [9.1,9.1,9.1]\n [0.1,0.1,0.1]\n

\n\n\n\n

We will go through the code in the sample program since it is the same general pattern we use throughout the rest of this chapter. Here is a slightly modified version of JavaKMeans.java with a few small changes from the version in the MLLib examples directory:

\n\n

 1 public final class JavaKMeans {\n 2 \n 3   private static class ParsePoint implements Function<String, Vector> {\n 4     private static final Pattern SPACE = Pattern.compile(\" \");\n 5 \n 6     @Override\n 7     public Vector call(String line) {\n 8       String[] tok = SPACE.split(line);\n 9       double[] point = new double[tok.length];\n10       for (int i = 0; i < tok.length; ++i) {\n11         point[i] = Double.parseDouble(tok[i]);\n12       }\n13       return Vectors.dense(point);\n14     }\n15   }\n16 \n17   public static void main(String[] args) {\n18 \n19     String inputFile = \"data/kmeans_data.txt\";\n20     int k = 2; // two clusters\n21     int iterations = 10;\n22     int runs = 1;\n23 \n24     JavaSparkContext sc = new JavaSparkContext(\"local\", \"JavaKMeans\");\n25     JavaRDD<String> lines = sc.textFile(inputFile);\n26 \n27     JavaRDD<Vector> points = lines.map(new ParsePoint());\n28 \n29     KMeansModel model = KMeans.train(points.rdd(), k, iterations, runs,\n30 \t                                 KMeans.K_MEANS_PARALLEL());\n31 \n32     System.out.println(\"Cluster centers:\");\n33     for (Vector center : model.clusterCenters()) {\n34       System.out.println(\" \" + center);\n35     }\n36     double cost = model.computeCost(points.rdd());\n37     System.out.println(\"Cost: \" + cost);\n38 \n39     sc.stop();\n40   }\n41 }\n

\n\n\n\n

The inner class ParsePoint is used to generate a single numeric feature vector given input data. In this simple case the input data is a string like “9.1 9.1 9.1” that needs to be tokenized and each token converted to a double floating point number. This inner class in the Wikipedia example will be much more complex.

\n\n

In line 24 we are creating a Spark execution context. In all of the examples in this chapter we use a “local” context which means that Spark will run inside the same JVM as our example code. In production the Spark context could access a Spark cluster utilizing multiple servers.

\n\n

Lines 25 and 27 introduce the use of the class JavaRDD which is the Java version of the Scala Ricj Data Definition class used internally to implement Spark. This data class can live inside a single JVM when we use a local Spark context or be spread over many servers when using a Spark server cluster. In line 25 we are reading the input strings and in line 27 we are using the inner class ParsePoint to convert the input text lines to numeric feature vectors.

\n\n

Lines 29 and 30 use the static method KMeans.train to create a trained clustering model. Lines 33 through 35 print out the clusers (listing of the output was shown earlier in this section).

\n\n

This simple example along with the “Hello Spark” example in the last section has shown you how to run Spark on your laptop in developers mode. In the next section we will see how to convert input text into a numeric feature vector.

\n\n

Converting Text to Numeric Feature Vectors

\n\n

In this section I develop a utility class TextToSparseVector that converts a set of text documents into sparse numeric feature vectors. Spark provides dense vector classes (like an array) and sparse vector classes that contain index and value pairs. For text feature vectors, In this example we will have a unique feature ID for each unique word stem (or word root) in all of the combined input documents.

\n\n

We start by reading in all of the input text and forming a Map where the keys are word stems and the values are unique IDs. Any given index document will be represented with a psarse vector. For each word, the word’s stem is represented by a unique ID. The vector element at the integer ID value is set. A short document with 100 unique words would have 100 elements of the sparse vector set.

\n\n

Listing of the class TextToSparseVector:

\n\n

  1 package com.markwatson.machine_learning;\n  2 \n  3 import java.nio.file.Files;\n  4 import java.nio.file.Paths;\n  5 import java.util.*;\n  6 import java.util.stream.Stream;\n  7 import org.apache.spark.mllib.linalg.Vector;\n  8 import org.apache.spark.mllib.linalg.Vectors;\n  9 \n 10 /**\n 11  * Copyright Mark Watson 2015. Apache 2 license.\n 12  */\n 13 public class TextToSparseVector {\n 14 \n 15   // The following value might have to be increased\n 16   // for large input texts:\n 17   static private int MAX_WORDS = 60000;\n 18 \n 19   static private Set<String> noiseWords = new HashSet<>();\n 20   static {\n 21     try {\n 22       Stream<String> lines =\n 23 \t    Files.lines(Paths.get(\"data\", \"stopwords.txt\"));\n 24       lines.forEach(s ->\n 25           noiseWords.add(s)\n 26       );\n 27       lines.close();\n 28     } catch (Exception e) { System.err.println(e); }\n 29   }\n 30 \n 31   private Map<String,Integer> wordMap = new HashMap<>();\n 32   private int startingWordIndex = 0;\n 33   private Map<Integer,String> reverseMap = null;\n 34 \n 35   public Vector tokensToSparseVector(String[] tokens) {\n 36     List<Integer> indices = new ArrayList();\n 37     for (String token : tokens) {\n 38       String stem = Stemmer.stemWord(token);\n 39       if(! noiseWords.contains(stem) && validWord((stem))) {\n 40         if (! wordMap.containsKey(stem)) {\n 41           wordMap.put(stem, startingWordIndex++);\n 42         }\n 43         indices.add(wordMap.get(stem));\n 44       }\n 45     }\n 46     int[] ind = new int[MAX_WORDS];\n 47 \n 48     double [] vals = new double[MAX_WORDS];\n 49     for (int i=0, len=indices.size(); i<len; i++) {\n 50       int index = indices.get(i);\n 51       ind[i] = index;\n 52       vals[i] = 1d;\n 53     }\n 54     Vector ret = Vectors.sparse(MAX_WORDS, ind, vals);\n 55     return ret;\n 56   }\n 57 \n 58   public boolean validWord(String token) {\n 59     if (token.length() < 3)  return false;\n 60     char[] chars = token.toCharArray();\n 61     for (char c : chars) {\n 62       if(!Character.isLetter(c)) return false;\n 63     }\n 64     return true;\n 65   }\n 66 \n 67   private static int NUM_BEST_WORDS = 20;\n 68 \n 69   public String [] bestWords(double [] cluster) {\n 70     int [] best = maxKIndex(cluster, NUM_BEST_WORDS);\n 71     String [] ret = new String[NUM_BEST_WORDS];\n 72     if (null == reverseMap) {\n 73       reverseMap = new HashMap<>();\n 74       for(Map.Entry<String,Integer> entry : wordMap.entrySet()){\n 75         reverseMap.put(entry.getValue(), entry.getKey());\n 76       }\n 77     }\n 78     for (int i=0; i<NUM_BEST_WORDS; i++)\n 79 \t  ret[i] = reverseMap.get(best[i]);\n 80     return ret;\n 81   }\n 82   // following method found on Stack Overflow\n 83   // and was written by user3879337. Thanks!\n 84   private  int[] maxKIndex(double[] array, int top_k) {\n 85     double[] max = new double[top_k];\n 86     int[] maxIndex = new int[top_k];\n 87     Arrays.fill(max, Double.NEGATIVE_INFINITY);\n 88     Arrays.fill(maxIndex, -1);\n 89 \n 90     top: for(int i = 0; i < array.length; i++) {\n 91       for(int j = 0; j < top_k; j++) {\n 92         if(array[i] > max[j]) {\n 93           for(int x = top_k - 1; x > j; x--) {\n 94             maxIndex[x] = maxIndex[x-1]; max[x] = max[x-1];\n 95           }\n 96           maxIndex[j] = i; max[j] = array[i];\n 97           continue top;\n 98         }\n 99       }\n100     }\n101     return maxIndex;\n102   }\n103 }\n

\n\n\n\n

The public method String [] bestWords(double [] cluster) is used for display purposes: it will be interesting to see the words in a document which provided the most evidence for its inclusion in a cluster. This Java class will be used in the next section to cluster similar Wikipedia articles.

\n\n

Using K-Means to Cluster Wikipedia Articles

\n\n

The Wikipedia article training files contain one article per line with the article title appearing at the beginning of the line. I have already performed some data cleaning: capturing HTML, stripping HTML tags, yielding plain text for the articles and then organizing the data one article per line in the input text file. I am providing two input files: one containing 41 articles and one containing 2001 articles.

\n\n

One challenge we face is converting the text of an article into a numeric feature vector. This was easy to do in an earlier section using cancer data already in numeric form; I just had to discard one data attribute and scale the remaining data atrributes.

\n\n

In addition to converting input data into numeric feature vectors we also need to decide how many clusters we want the K-Means algorithm to partitian our data into.

\n\n

Given the class TextToSparseVector developed in the last section, the code to cluster Wikipedia articles is fairly straightforward. In the following listing of the class WikipediaKMeans

\n\n

 1 // This example is derived from the Spark MLlib examples\n 2 \n 3 package com.markwatson.machine_learning;\n 4 \n 5 import java.io.IOException;\n 6 import java.nio.file.Files;\n 7 import java.nio.file.Paths;\n 8 import java.util.Arrays;\n 9 import java.util.regex.Pattern;\n10 import java.util.stream.Stream;\n11 \n12 import org.apache.spark.SparkConf;\n13 import org.apache.spark.api.java.JavaRDD;\n14 import org.apache.spark.api.java.JavaSparkContext;\n15 import org.apache.spark.api.java.function.Function;\n16 \n17 import org.apache.spark.mllib.clustering.KMeans;\n18 import org.apache.spark.mllib.clustering.KMeansModel;\n19 import org.apache.spark.mllib.linalg.Vector;\n20 import org.apache.spark.mllib.linalg.Vectors;\n21 \n22 public class WikipediaKMeans {\n23 \n24   // there are two example files in ./data/: one with\n25   // 41 articles and one with 2001\n26   //private final static String input_file = \"wikipedia_41_lines.txt\";\n27   private final static String input_file = \"wikipedia_2001_lines.txt\";\n28 \n29   private static TextToSparseVector sparseVectorGenerator =\n30    new TextToSparseVector();\n31   private static final Pattern SPACE = Pattern.compile(\" \");\n32 \n33   private static class ParsePoint implements Function<String, Vector> {\n34 \n35     @Override\n36     public Vector call(String line) {\n37       String[] tok = SPACE.split(line.toLowerCase());\n38       return sparseVectorGenerator.tokensToSparseVector(tok);\n39     }\n40   }\n41 \n42   public static void main(String[] args) throws IOException {\n43 \n44     int number_of_clusters = 8;\n45     int iterations = 100;\n46     int runs = 1;\n47 \n48     JavaSparkContext sc = new JavaSparkContext(\"local\", \"WikipediaKMeans\");\n49     JavaRDD<String> lines = sc.textFile(\"data/\" + input_file);\n50     JavaRDD<Vector> points = lines.map(new ParsePoint());\n51     KMeansModel model = KMeans.train(points.rdd(), number_of_clusters,\n52 \t                                 iterations, runs,\n53                                      KMeans.K_MEANS_PARALLEL());\n54 \n55     System.out.println(\"Cluster centers:\");\n56     for (Vector center : model.clusterCenters()) {\n57       System.out.println(\"\\n \" + center);\n58       String [] bestWords = sparseVectorGenerator.bestWords(center.toArray());\n59       System.out.println(\" bestWords: \" + Arrays.asList(bestWords));\n60     }\n61     double cost = model.computeCost(points.rdd());\n62     System.out.println(\"Cost: \" + cost);\n63 \n64     // re-read each \"document* (single line in input file\n65     // and predict which cluster it belongs to:\n66     Stream<String> lines2 = Files.lines(Paths.get(\"data\", input_file));\n67     lines2.forEach(s ->\n68         {\n69           String[] parts = s.split(\"\\t\");\n70           String[] tok = SPACE.split(parts[1].toLowerCase());\n71           Vector v = sparseVectorGenerator.tokensToSparseVector(tok);\n72           int cluster_index = model.predict(v);\n73           System.out.println(\"Title: \" + parts[0] + \", cluster index: \" +\n74 \t\t                     cluster_index);\n75         }\n76     );\n77     lines2.close();\n78 \n79     sc.stop();\n80   }\n81 }\n

\n\n\n\n

The class WikipediaKMeans is fairly simple because most of the work is done converting text to feature vectors using class TextToSparseVector that we saw in the last seciton. The Spark setup code is also similar to that seen in our tiny HelloSpark example.

\n\n

The parsepoint method defined in lines 33 to 40 uses the method tokensToSparseVector (defined in class TextToSparseVector) to convert the text in one line of the input file (remember that each line contains text for an entire Wikipedia article) to a sparse feature vector. There are two important parameters that you will want to experiment with. These parameters are defined on lines 44 and 45 and set the desired number of clusters and the number of iterations you want to run to form clusters. I suggest leaving iterations set initially to 100 as it is in the code and then vary number_of_clusters. If you are using this code in one of your own projects with a different data set then you might also want to experiment with running more iterations. If you cluster a very large number of text documents then you may have to increase the constant value MAX_WORDS set in line 9 in the class TextToSparseVector.

\n\n

In line 48 we are creating a new Spark context that is local and will run in the same JVM as your application code. Lines 49 and 50 read in the text data from Wikipedia and create a JavaRDD of vectors pass to the K-Means clustering code that is called in lines 51 to 53.

\n\n

Here are a few cluster indices for the 2001 input Wikipedia articles that are printed out by cluster index. When you run the sample program this list of documents assigned to eight different clusters is a lot of output. In the following I am listing just a small bit of the output for the first four clusters. If you look carefully at the generated clusters you will notice that Cluster seems not to be so useful since there are really two or three topics represented in the cluster. This might indicate re-running the clustering with a larger number of requested clusters (i.e., increasing the value of number_of_clusters). On the other hand, Cluster 1 seems very good, almost all articles are dealing with sports.

\n\n

DOCUMENTS IN CLUSTER INDEX 0\n\n   Article title: Academy Award for Best Art Direction\n   Article title: Agricultural science\n   Article title: Alien\n   Article title: Astronomer\n   Article title: Apollo\n   Article title: Anatomy\n   Article title: Demographics of Angola\n   Article title: Transport in Angola\n   Article title: Angolan Armed Forces\n   Article title: Arctic Circle\n   Article title: List of anthropologists\n   Article title: Asociaci贸n Alumni\n   Article title: Aramaic alphabet\n   Article title: Applied ethics\n   Article title: August\n   Article title: Politics of Antigua and Barbuda\n   Article title: Royal Antigua and Barbuda Defence Force\n   Article title: Demographics of Armenia\n   Article title: Geography of American Samoa\n   Article title: America's National Game\n   Article title: Augustin-Jean Fresnel\n   Article title: Ashmore and Cartier Islands\n   Article title: Extreme poverty\n   Article title: Geography of Antarctica\n   Article title: Transport in Antarctica\n   Article title: Algernon Swinburne\n   Article title: Alfred Lawson\n\nDOCUMENTS IN CLUSTER INDEX 1\n\n   Article title: Albert Spalding\n   Article title: Arizona Cardinals\n   Article title: Atlanta Falcons\n   Article title: Arizona Diamondbacks\n   Article title: Atlanta Braves\n   Article title: Arsenal F.C.\n   Article title: AZ (football club)\n   Article title: American Football League\n   Article title: Board game\n   Article title: Baseball statistics\n   Article title: Base on balls\n   Article title: Hit by pitch\n   Article title: Stolen base\n   Article title: Baseball\n   Article title: History of baseball in the United States\n   Article title: Major League Baseball Most Valuable Player Award\n   Article title: Major League Baseball Rookie of the Year Award\n   Article title: National League Championship Series\n   Article title: American League Championship Series\n   Article title: National League Division Series\n   Article title: 2001 World Series\n   Article title: 1903 World Series\n   Article title: Basketball\n   Article title: National Baseball Hall of Fame and Museum\n   Article title: Bill Walsh (American football coach)\n   Article title: Babe Ruth\n   Article title: Baltimore Ravens\n   Article title: Buffalo Bills\n   Article title: Backgammon\n   Article title: Boston Red Sox\n   Article title: Baltimore Orioles\n   Article title: Barry Bonds\n   Article title: Bob Costas\n\nDOCUMENTS IN CLUSTER INDEX 2\n\n   Article title: International Atomic Time\n   Article title: Altruism\n   Article title: ASCII\n   Article title: Arithmetic mean\n   Article title: Algae\n   Article title: Analysis of variance\n   Article title: Assistive technology\n   Article title: Acid\n   Article title: American National Standards Institute\n   Article title: Atomic number\n   Article title: Algorithm\n   Article title: Axiom of choice\n   Article title: Algorithms for calculating variance\n   Article title: Analysis\n   Article title: Amplitude modulation\n   Article title: Automorphism\n   Article title: Artificial intelligence\n   Article title: Astrometry\n   Article title: Alloy\n   Article title: Angle\n   Article title: Acoustics\n   Article title: Atomic physics\n   Article title: Atomic orbital\n   Article title: Amino acid\n   Article title: Area\n   Article title: Astronomical unit\n   Article title: Kolmogorov complexity\n   Article title: An Enquiry Concerning Human Understanding\n   Article title: Adenosine triphosphate\n   Article title: Antibacterial\n\nDOCUMENTS IN CLUSTER INDEX 3\n\n   Article title: Academy Award\n   Article title: Animation\n   Article title: Andrei Tarkovsky\n   Article title: Alfred Hitchcock\n   Article title: American Film Institute\n   Article title: Akira Kurosawa\n   Article title: ABBA\n   Article title: Afro Celt Sound System\n   Article title: The Alan Parsons Project\n   Article title: Dutch hip hop\n   Article title: Anyone Can Whistle\n   Article title: A Funny Thing Happened on the Way to the Forum\n   Article title: Albert Brooks\n   Article title: Arlo Guthrie\n   Article title: The Birth of a Nation\n   Article title: Blade Runner\n   Article title: Blazing Saddles\n   Article title: Brigitte Bardot\n   Article title: Blue Velvet (film)\n   Article title: Bing Crosby\n

\n\n\n\n

This example program can be reused in your applications with few changes. You will probably want to manually assign meaningful labels to each cluster index and store the label with each document. For example, the cluster at index 1 would probably be labeled as “sports.”

\n\n

Using SVM for Text Classification

\n\n

  1 package com.markwatson.machine_learning; \n  2 \n  3 import org.apache.spark.api.java.JavaDoubleRDD;\n  4 import org.apache.spark.api.java.JavaRDD;\n  5 import org.apache.spark.api.java.JavaSparkContext;\n  6 import org.apache.spark.api.java.function.Function;\n  7 import org.apache.spark.mllib.classification.SVMModel;\n  8 import org.apache.spark.mllib.classification.SVMWithSGD;\n  9 import org.apache.spark.mllib.linalg.Vector;\n 10 import org.apache.spark.mllib.regression.LabeledPoint;\n 11 import scala.Tuple2;\n 12 \n 13 import java.io.IOException;\n 14 import java.nio.file.Files;\n 15 import java.nio.file.Paths;\n 16 import java.util.*;\n 17 import java.util.regex.Pattern;\n 18 import java.util.stream.Stream;\n 19 \n 20 /**\n 21  * Created by markw on 11/25/15.\n 22  */\n 23 public class SvmTextClassifier {\n 24   private final static String input_file = \"test_classification.txt\";\n 25 \n 26   private static TextToSparseVector sparseVectorGenerator = new TextToSparseVect\\\n 27 or();\n 28   private static final Pattern SPACE = Pattern.compile(\" \");\n 29 \n 30   static private String [] tokenizeAndRemoveNoiseWords(String s) {\n 31     return s.toLowerCase().replaceAll(\"\\\\.\", \" \\\\. \").replaceAll(\",\", \" , \")\n 32         .replaceAll(\"\\\\?\", \" ? \").replaceAll(\"\\n\", \" \")\n 33         .replaceAll(\";\", \" ; \").split(\" \");\n 34   }\n 35 \n 36   static private Map<String, Integer> label_to_index = new HashMap<>();\n 37   static private int label_index_count = 0;\n 38   static private Map<String, String> map_to_print_original_text = new HashMap<>(\\\n 39 );\n 40 \n 41   private static class ParsePoint implements Function<String, LabeledPoint> {\n 42 \n 43     @Override\n 44     public LabeledPoint call(String line) {\n 45       String [] data_split = line.split(\"\\t\");\n 46       String label = data_split[0];\n 47       Integer label_index = label_to_index.get(label);\n 48       if (null == label_index) {\n 49         label_index = label_index_count;\n 50         label_to_index.put(label, label_index_count++);\n 51       }\n 52       Vector tok = sparseVectorGenerator.tokensToSparseVector(\n 53                           tokenizeAndRemoveNoiseWords(data_split[1]));\n 54       // Save original text, indexed by compressed features, for later\n 55       // display (for debug ony):\n 56       map_to_print_original_text.put(tok.compressed().toString(), line);\n 57       return new LabeledPoint(label_index, tok);\n 58     }\n 59   }\n 60 \n 61   public static void main(String[] args) throws IOException {\n 62 \n 63     JavaSparkContext sc = new JavaSparkContext(\"local\", \"WikipediaKMeans\");\n 64 \n 65     JavaRDD<String> lines = sc.textFile(\"data/\" + input_file);\n 66 \n 67     JavaRDD<LabeledPoint> points = lines.map(new ParsePoint());\n 68 \n 69     // Split initial RDD into two with 70% training data and 30% testing\n 70     // data (13L is a random seed):\n 71     JavaRDD<LabeledPoint>[] splits =\n 72        points.randomSplit(new double[]{0.7, 0.3}, 13L);\n 73     JavaRDD<LabeledPoint> training = splits[0].cache();\n 74     JavaRDD<LabeledPoint> testing = splits[1];\n 75     training.cache();\n 76 \n 77     // Building the model\n 78     int numIterations = 500;\n 79     final SVMModel model =\n 80         SVMWithSGD.train(JavaRDD.toRDD(training), numIterations);\n 81     model.clearThreshold();\n 82     // Evaluate model on testing examples and compute training error\n 83     JavaRDD<Tuple2<Double, Double>> valuesAndPreds = testing.map(\n 84         new Function<LabeledPoint, Tuple2<Double, Double>>() {\n 85           public Tuple2<Double, Double> call(LabeledPoint point) {\n 86             double prediction = model.predict(point.features());\n 87             System.out.println(\" ++ prediction: \" + prediction + \" original: \" +\n 88               map_to_print_original_text.get(\n 89                     point.features().compressed().toString()));\n 90             return new Tuple2<Double, Double>(prediction, point.label());\n 91           }\n 92         }\n 93     );\n 94 \n 95     double MSE = new JavaDoubleRDD(valuesAndPreds.map(\n 96         new Function<Tuple2<Double, Double>, Object>() {\n 97           public Object call(Tuple2<Double, Double> pair) {\n 98             return Math.pow(pair._1() - pair._2(), 2.0);\n 99           }\n100         }\n101     ).rdd()).mean();\n102     System.out.println(\"Test Data Mean Squared Error = \" + MSE);\n103 \n104     sc.stop();\n105   }\n106 \n107   static private Set<String> noiseWords = new HashSet<>();\n108   static {\n109     try {\n110       Stream<String> lines = Files.lines(Paths.get(\"data\", \"stopwords.txt\"));\n111       lines.forEach(s -> noiseWords.add(s));\n112       lines.close();\n113     } catch (Exception e) { System.err.println(e); }\n114   }\n115 \n116 }\n

\n\n\n\n

This code calculates prediction values less than zero for the EDUCATION class and greater than zero for the HEALTH class:

\n\n

 ++ prediction: -0.11864189612108378 original: EDUCATION\tThe university admitted\\\n more students this year and dropout rate is lessening.\n ++ prediction: -0.5304494104173201 original: EDUCATION\tThe students turned in t\\\nheir homework at school before summer break.\n ++ prediction: -0.3935231701688143 original: EDUCATION\tThe school suspended fou\\\nr students for cheating.\n ++ prediction: 0.7546558085203632 original: HEALTH\tThe cold outbreak was bad bu\\\nt not an epidemic.\n ++ prediction: 1.4008731503216094 original: HEALTH\tThe doctor and the nurse adv\\\nised be to get rest because of my cold.\n ++ prediction: -0.25659692992030847 original: EDUCATION\tThe school board and te\\\nachers met with the city council.\n ++ prediction: -0.5304494104173201 original: EDUCATION\tThe student lost her hom\\\nework.\n ++ prediction: 1.4324107108481499 original: HEALTH\tThe hospital has 215 patient\\\ns, 10 doctors, and 22 nurses.\n ++ prediction: 0.47977453447263274 original: HEALTH\tThe cold and flu season sta\\\nrted this month.\n\nTest Data Mean Squared Error = 0.1640042373582832\n

\n\n\n\n

Using word2vec To Find Similar Words In Documents

\n\n

Google released their open source word2vec library as Apache 2 licensed open source. The Deeplearning4j project and Spark’s MLlib both contain implementations. As I write this chapter the Deeplearning4j version has more flexibility but we will use the MLlib version here. We cover Deeplearning4j in a later chapter. We will use the sample data file from the Deeplearning4j project in this section.

\n\n

The word2vec library can identify which other words in text are strongly related to any other word in the text. If you run word2vec on a large sample of English language text you will find the the word woman is closely associated with the word man, the word child to the word mother, the word road to the word car, etc.

\n\n

The following listing shows the class Word2VecRelatedWords that is fairly simple because all we need to do is tokenize text, discard noise (or stop) words, and convert the input text to the correct Spark data types for processing by the MLlib class *Word2VecModel**.

\n\n

 1 package com.markwatson.machine_learning;\n 2 \n 3 import org.apache.spark.api.java.JavaRDD;\n 4 import org.apache.spark.api.java.JavaSparkContext;\n 5 import org.apache.spark.mllib.feature.Word2Vec;\n 6 import org.apache.spark.mllib.feature.Word2VecModel;\n 7 \n 8 import java.io.IOException;\n 9 import java.nio.file.Files;\n10 import java.nio.file.Paths;\n11 import java.util.*;\n12 import java.util.stream.Collectors;\n13 import java.util.stream.Stream;\n14 \n15 import scala.Tuple2;\n16 \n17 public class Word2VecRelatedWords {\n18   // Use the example data file form the deeplearning4j.org word2vec example:\n19   private final static String input_file = \"data/raw_sentences.txt\";\n20 \n21   static private List<String> tokenizeAndRemoveNoiseWords(String s) {\n22     return Arrays.asList(s.toLowerCase().replaceAll(\"\\\\.\", \" \\\\. \")\n23         .replaceAll(\",\", \" , \").replaceAll(\"\\\\?\", \" ? \")\n24         .replaceAll(\"\\n\", \" \").replaceAll(\";\", \" ; \").split(\" \"))\n25         .stream().filter((w) ->\n26 \t\t     ! noiseWords.contains(w)).collect(Collectors.toList());\n27   }\n28 \n29   public static void main(String[] args) throws IOException {\n30     JavaSparkContext sc = new JavaSparkContext(\"local\", \"WikipediaKMeans\");\n31 \n32     String sentence = new String(Files.readAllBytes(Paths.get(input_file)));\n33     List<String> words = tokenizeAndRemoveNoiseWords(sentence);\n34     List<List<String>> localWords = Arrays.asList(words, words);\n35     Word2Vec word2vec = new Word2Vec().setVectorSize(10).setSeed(113L);\n36     JavaRDD<List<String>> rdd_word_list = sc.parallelize(localWords);\n37     Word2VecModel model = word2vec.fit(rdd_word_list);\n38     \n39     Tuple2<String, Object>[] synonyms = model.findSynonyms(\"day\", 15);\n40     for (Object obj : synonyms)\n41 \t  System.err.println(\"-- words associated with day: \" + obj);\n42 \n43     synonyms = model.findSynonyms(\"children\", 15);\n44     for (Object obj : synonyms)\n45 \t  System.err.println(\"-- words associated with children: \" + obj);\n46 \n47     synonyms = model.findSynonyms(\"people\", 15);\n48     for (Object obj : synonyms)\n49 \t  System.err.println(\"-- words associated with people: \" + obj);\n50 \n51     synonyms = model.findSynonyms(\"three\", 15);\n52     for (Object obj : synonyms)\n53 \t  System.err.println(\"-- words associated with three: \" + obj);\n54 \n55     synonyms = model.findSynonyms(\"man\", 15);\n56     for (Object obj : synonyms)\n57 \t  System.err.println(\"-- words associated with man: \" + obj);\n58 \n59     synonyms = model.findSynonyms(\"women\", 15);\n60     for (Object obj : synonyms)\n61 \t  System.err.println(\"-- words associated with women: \" + obj);\n62 \n63     sc.stop();\n64   }\n65 \n66   static private Set<String> noiseWords = new HashSet<>();\n67   static {\n68     try {\n69       Stream<String> lines = Files.lines(Paths.get(\"data\", \"stopwords.txt\"));\n70       lines.forEach(s -> noiseWords.add(s));\n71       lines.close();\n72     } catch (Exception e) { System.err.println(e); }\n73   }\n74 }\n

\n\n\n\n

The output from this example program is:

\n\n

words associated with 'day': (season,0.952233990962071)\nwords associated with 'day': (night,0.9511060013892698)\nwords associated with 'day': (big,0.8728547447056546)\nwords associated with 'day': (good,0.8576788222594224)\nwords associated with 'day': (today,0.8396659969645013)\nwords associated with 'day': (end,0.8293329248658234)\nwords associated with 'day': (life,0.8249991404480943)\nwords associated with 'day': (game,0.80432033480323)\nwords associated with 'day': (program,0.792725056069907)\nwords associated with 'day': (business,0.7913084744641095)\nwords associated with 'day': (political,0.7818051986127171)\nwords associated with 'day': (market,0.7764758770314986)\nwords associated with 'day': (second,0.7737555854409787)\nwords associated with 'day': (year,0.7429513803623038)\nwords associated with 'day': (set,0.7294508559210614)\nwords associated with 'children': (million,1.0903815875201748)\nwords associated with 'children': (women,1.059845154178747)\nwords associated with 'children': (days,0.9776504379129254)\nwords associated with 'children': (two,0.9646151248422365)\nwords associated with 'children': (years,0.936748689685324)\nwords associated with 'children': (members,0.9017954013921553)\nwords associated with 'children': (house,0.8657203835921035)\nwords associated with 'children': (old,0.8631901736913122)\nwords associated with 'children': (left,0.86281306293074)\nwords associated with 'children': (four,0.8613461580704683)\nwords associated with 'children': (three,0.858547609404486)\nwords associated with 'children': (week,0.8522042667181716)\nwords associated with 'children': (five,0.8179405841924944)\nwords associated with 'children': (home,0.8095612388770681)\nwords associated with 'children': (case,0.7461659839438249)\nwords associated with 'people': (know,0.7792464691200726)\nwords associated with 'people': (want,0.734389563281012)\nwords associated with 'people': (work,0.6829544862339761)\nwords associated with 'people': (make,0.6645301218487332)\nwords associated with 'people': (say,0.6153710744965498)\nwords associated with 'people': (public,0.6054330328462636)\nwords associated with 'people': (ms,0.5842404516570064)\nwords associated with 'people': (just,0.5791629468565187)\nwords associated with 'people': (white,0.5636639753271303)\nwords associated with 'people': (home,0.5586558330551199)\nwords associated with 'people': (little,0.5401336635483608)\nwords associated with 'people': (think,0.5176197541759774)\nwords associated with 'people': (american,0.5098579866944077)\nwords associated with 'people': (women,0.5046623970160208)\nwords associated with 'people': (?,0.47238591701531557)\nwords associated with 'three': (two,2.0172783800362466)\nwords associated with 'three': (five,1.9963427800055753)\nwords associated with 'three': (four,1.9890870749494771)\nwords associated with 'three': (years,1.836102840486217)\nwords associated with 'three': (million,1.7966031095487769)\nwords associated with 'three': (days,1.7456308099866056)\nwords associated with 'three': (ago,1.7438148167574663)\nwords associated with 'three': (old,1.6704126685943632)\nwords associated with 'three': (week,1.6393314973787558)\nwords associated with 'three': (times,1.6236709983968334)\nwords associated with 'three': (children,1.4605285096333689)\nwords associated with 'three': (left,1.3927866198115773)\nwords associated with 'three': (members,1.2757167489324481)\nwords associated with 'three': (companies,1.106585238117151)\nwords associated with 'three': (year,1.0806081055554506)\nwords associated with 'man': (john,0.6841310695193488)\nwords associated with 'man': (case,0.6795045575483528)\nwords associated with 'man': (program,0.6652340138422144)\nwords associated with 'man': (dr,0.659963781737535)\nwords associated with 'man': (market,0.6552229884848991)\nwords associated with 'man': (ms,0.6488172627464075)\nwords associated with 'man': (yesterday,0.640285291527994)\nwords associated with 'man': (director,0.6400444581814078)\nwords associated with 'man': (street,0.6395737967168845)\nwords associated with 'man': (mr,0.6181822944209798)\nwords associated with 'man': (court,0.6113708937966306)\nwords associated with 'man': (political,0.5994838292027088)\nwords associated with 'man': (life,0.5873452829470825)\nwords associated with 'man': (office,0.5714323032795753)\nwords associated with 'women': (children,0.7336026558437857)\nwords associated with 'women': (home,0.6950468303126367)\nwords associated with 'women': (war,0.65634414969338)\nwords associated with 'women': (money,0.6548505665001131)\nwords associated with 'women': (say,0.6490701416147411)\nwords associated with 'women': (like,0.6322006183913979)\nwords associated with 'women': (public,0.6058074917094844)\nwords associated with 'women': (court,0.6057305040658708)\nwords associated with 'women': (want,0.5977223947592321)\nwords associated with 'women': (music,0.5939362450235905)\nwords associated with 'women': (street,0.5876902247693967)\nwords associated with 'women': (house,0.5866848059316422)\nwords associated with 'women': (country,0.5724038019422217)\nwords associated with 'women': (case,0.57149905752473)\nwords associated with 'women': (family,0.5609861524585015)\n

\n\n\n\n

While word2vec results are just the result of statistical processing and the code does not understand the semantic of the English language, the results are still impressive.

\n\n

Chapter Wrap Up

\n\n

In the previous chapter we used machine learning for natural language processing using the OpenNLP project. In the examples in this chapter we used the Spark MLlib library for for logistic regression, document clustering, and determining which words are closely associated with each other.

\n\n

You can go far using OpenNLP and Spark MLlib in your projects. There are a few things to keep in mind, however. One of the most difficult aspects of machine learning is finding and preparing training data. This process will be very dependant on what kind of project you are working on and what sources of data you have.

\n\n

One of the most powerful machine learning techniques is artificial neural networks. I have a long history of using neural networks (I was on a DARPA neural network advisory panel and wrote the first version of the commercial ANSim neural network product in the late 1980s). I decided not to cover simple neural network models in this book because I have already written about neural networks in one chapter of my book Practical Artificial Intelligence Programming in Java. “Deep learning” neural networks are also very effective (if difficult to use) for some types of machine learning problems (e.g., image recognition and speech recognition) and we will use the Deeplearning4j project in a later chapter.

\n\n

In the next chapter we cover another type of machine learning: anomaly detection.

\n\n\n

Anomaly Detection Machine Learning Example

\n\n

Anomaly detection models are used in one very specific class of use cases: when you have many negative (non-anomaly) examples and relatively few positive (anomaly) examples. For training we will ignore positive examples, create a model of “how things should be”, and hopefully be able to detect anomalies different from the original negative examples.

\n\n

If you have a large training set of both negative and positive examples then do not use anomaly detection models.

\n\n

Motivation for Anomaly Detection

\n\n

There are two other examples in this book using the University of Wisconsin cancer data. These other examples are supervised learning. Anomaly detection as we do it in this chapter is, more or less, unsupervised learning.

\n\n

When should we use anomaly detection? You should use supervised learning algorithms like neural networks and logistic classification when there are roughly equal number of available negative and positive examples in the training data. The University of Wisconsin cancer data set is fairly evenly split between negative and positive examples.

\n\n

Anomaly detection should be used when you have many negative (“normal”) examples and relatively few positive (“anomaly”) examples. For the example in this chapter we will simulate scarcity of positive (“anomaly”) results by preparing the data using the Wisconsin cancer data as follows:

\n\n

We will split the data into training (60%), cross validation (20%) and testing (20%).
For the training data, we will discard all but two positive (“anomaly”) examples. We do this to simulate the real world test case where some positive examples are likely to end up in the training data in spite of the fact that we would prefer the training data to only contain negative (“normal”) examples.
We will use the cross validation data to find a good value for the epsilon meta parameter.
After we find a good epsilon value, we will calculate the F1 measurement for the model.

\n\n

Math Primer for Anomaly Detection

\n\n

We are trying to model “normal” behavior and we do this by taking each feature and fitting a Gaussian (bell curve) distribution to each feature. The learned parameters for a Gaussian distribution are the mean of the data (where the bell shaped curve is centered) and the variance. You might be more familiar with the term standard deviation, \$\\sigma\$. Variance is defined as \$ \\sigma ^2\$.

\n\n

We will need to calculate the probability of a value x given the mean and variance of a probability distribution: \$P(x : \\mu, \\sigma ^2)\$ where \$\\mu\$ is the mean and \$ \\sigma ^2\$\n is the squared variance:

\n\n\n$$\nP(x : \\mu, \\sigma ^2) = \\frac{1}{{\\sigma \\sqrt {2\\pi } }}e^{{{ - \\left( {x - \\mu } \\right)^2 } \\mathord{\\left/ {\\vphantom {{ - \\left( {x - \\mu } \\right)^2 } {2\\sigma ^2 }}} \\right. \\kern-\\nulldelimiterspace} {2\\sigma ^2 }}}\n$$\n\n

where \$x_i\$ are the samples and we can calculate the squared variance as:

\n\n\n$$\n\\sigma^2 = \\frac{\\displaystyle\\sum_{i=1}^{m}(x_i - \\mu)^2} {m}\n$$\n\n

We calculate the parameters of \$\\mu\$ and \$ \\sigma ^2\$ for each feature. A bell shaped distribution in two dimensions is easy to visualize as is an inverted bowl shape in three dimentions. What if we have many features? Well, the math works and don’t worry about not being able to picture it in your mind.

\n\n

AnomalyDetection Utility Class

\n\n

The class AnomalyDetection developed in this section is fairly general purpose. It processes a set of training examples and for each feature calculates \$\\mu\$ and \$ \\sigma ^2\$. We are also training for a third parameter: an epsilon “cutoff” value: if for a given input vector if \$P(x : \\mu, \\sigma ^2)\$ evaluates to a value greater than epsilon then the input vector is “normal”, less than epsilon implies that the input vector is an “anomaly.” The math for calulating these three features from training data is fairly easy but the code is not: we need to organize the training data and search for a value of epsilon that minimizes the error for a cross validaton data set.

\n\n

To be clear: we separate the input examples into three separate sets of training, cross validation, and testing data. We use the training data to set the model parameters, use the cross validation data to learn an epsilon value, and finally use the testing data to get precision, recall, and F1 scores that indicate how well the model detects anomalies in data not used for training and cross validation.

\n\n

I present the example program as one long listing, with more code explanation after the listing. Please note the long loop over each input training example starting at line 28 and ending on line 74. The code in lines 25 through 44 processes the input training data sample into three disjoint sets of training, cross validation, and testing data. Then the code in lines 45 through 63 copies these three sets of data to Java arrays.

\n\n

The code in lines 65 through 73 calculates, for a training example, the value of \$\\mu\$ (the varible mu in the code).

\n\n

Please note in the code example that I prepend class variables used in methods with “this.” even when it is not required. I do this for legibility and is a personal style.

\n\n

  1 package com.markwatson.anomaly_detection;\n  2 \n  3 import java.util.ArrayList;\n  4 import java.util.List;\n  5 \n  6 /**\n  7  * Created by markw on 10/7/15.\n  8  */\n  9 public class AnomalyDetection {\n 10 \n 11   public AnomalyDetection() { }\n 12 \n 13   /**\n 14    * AnomalyDetection is a general purpose class for building anomaly detection\n 15    * models. You should use this type of mdel when you have mostly negative\n 16    * training examples with relatively few positive examples and you need a model\n 17    * that detects postive (anomaly) inputs.\n 18    *\n 19    * @param num_features\n 20    * @param num_training_examples\n 21    * @param training_examples [num_training_examples][num_features]\n 22    */\n 23   public AnomalyDetection(int num_features, int num_training_examples,\n 24                           double [][] training_examples) {\n 25     List<double[]> training = new ArrayList<>();\n 26     List<double []> cross_validation = new ArrayList<>();\n 27     List<double []> testing = new ArrayList<>();\n 28     int outcome_index = num_features - 1; // index of target outcome\n 29     for (int i=0; i<num_training_examples; i++) {\n 30       if (Math.random() < 0.6) { // 60% training\n 31         // Only keep normal (negative) examples in training, except\n 32         // remember the reason for using this algorithm is that it\n 33         // works with many more negative examples than positive\n 34         // examples, and that the algorithm works even with some positive\n 35         // examples mixed into the training set. The random test is to\n 36         // allow about 10% positive examples to get into the training set:\n 37         if (training_examples[i][outcome_index] < 0.5 || Math.random() < 0.1)\n 38           training.add(training_examples[i]);\n 39       } else if (Math.random() < 0.7) {\n 40           cross_validation.add(training_examples[i]);\n 41       } else {\n 42         testing.add(training_examples[i]);\n 43       }\n 44     }\n 45     this.num_training_examples = training.size();\n 46     this.num_cross_validation_examples = cross_validation.size();\n 47     this.num_testing_examples = testing.size();\n 48 \n 49     this.training_examples = new double[this.num_training_examples][];\n 50     for (int k=0; k<this.num_training_examples; k++) {\n 51       this.training_examples[k] = training.get(k);\n 52     }\n 53 \n 54     this.cross_validation_examples = new double[num_cross_validation_examples][];\n 55     for (int k=0; k<num_cross_validation_examples; k++) {\n 56       this.cross_validation_examples[k] = cross_validation.get(k);\n 57     }\n 58 \n 59     this.testing_examples = new double[num_testing_examples][];\n 60     for (int k=0; k<num_testing_examples; k++) {\n 61       // dimensions of [num_training_examples][num_features]:\n 62       this.testing_examples[k] = testing.get(k);\n 63     }\n 64 \n 65     this.mu = new double[num_features];\n 66     this.sigma_squared = new double[num_features];\n 67     this.num_features = num_features;\n 68     for (int nf = 0; nf < this.num_features; nf++) { //\n 69       double sum = 0;\n 70       for (int nt = 0; nt < this.num_training_examples; nt++)\n 71         sum += training_examples[nt][nf];\n 72       this.mu[nf] = sum / this.num_training_examples;\n 73     }\n 74   }\n 75 \n 76   public double [] muValues()     { return mu; }\n 77   public double [] sigmaSquared() { return sigma_squared; }\n 78   public double bestEpsilon()     { return best_epsilon; }\n 79 \n 80   /**\n 81    *\n 82    * Train the model using a range of epsilon values. Epsilon is a\n 83    * hyper parameter that we want to find a value that\n 84    * minimizes the error count.\n 85    */\n 86   public void train() {\n 87     double best_error_count = 1e10;\n 88     for (int epsilon_loop=0; epsilon_loop<40; epsilon_loop++) {\n 89       double epsilon = 0.05 + 0.01 * epsilon_loop;\n 90       double error_count = train(epsilon);\n 91       if (error_count < best_error_count) {\n 92         best_error_count = error_count;\n 93         best_epsilon = epsilon;\n 94       }\n 95     }\n 96     System.out.println(\"\\n**** Best epsilon value = \" + best_epsilon );\n 97 \n 98     // retrain for the best epsilon setting the maximum likelyhood parameters\n 99     // which are now in epsilon, mu[] and sigma_squared[]:\n100     train_helper(best_epsilon);\n101 \n102     // finally, we are ready to see how the model performs with test data:\n103     test(best_epsilon);\n104   }\n105 \n106   /**\n107    * calculate probability p(x; mu, sigma_squared)\n108    *\n109    * @param x - example vector\n110    * @return\n111    */\n112   private double p(double [] x) {\n113     double sum = 0;\n114     // use (num_features - 1) to skip target output:\n115     for (int nf=0; nf<num_features - 1; nf++) {\n116       sum += (1.0 / (SQRT_2_PI * sigma_squared[nf])) *\n117              Math.exp(- (x[nf] - mu[nf]) * (x[nf] - mu[nf]));\n118     }\n119     return sum / num_features;\n120   }\n121 \n122   /**\n123    * returns 1 if input vector is an anoomaly\n124    *\n125    * @param x - example vector\n126    * @return\n127    */\n128   public boolean isAnamoly(double [] x) {\n129     double sum = 0;\n130     // use (num_features - 1) to skip target output:\n131     for (int nf=0; nf<num_features - 1; nf++) {\n132       sum += (1.0 / (SQRT_2_PI * sigma_squared[nf])) *\n133              Math.exp(- (x[nf] - mu[nf]) * (x[nf] - mu[nf]));\n134     }\n135     return (sum / num_features) < best_epsilon;\n136   }\n137 \n138   private double train_helper_(double epsilon) {\n139     // use (num_features - 1) to skip target output:\n140     for (int nf = 0; nf < this.num_features - 1; nf++) {\n141       double sum = 0;\n142       for (int nt=0; nt<this.num_training_examples; nt++) {\n143         sum += (this.training_examples[nt][nf] - mu[nf]) *\n144                (this.training_examples[nt][nf] - mu[nf]);\n145       }\n146       sigma_squared[nf] = (1.0 / num_features) * sum;\n147     }\n148     double cross_validation_error_count = 0;\n149     for (int nt=0; nt<this.num_cross_validation_examples; nt++) {\n150       double[] x = this.cross_validation_examples[nt];\n151       double calculated_value = p(x);\n152       if (x[9] > 0.5) { // target training output is ANOMALY\n153         // if the calculated value is greater than epsilon\n154         // then this x vector is not an anomaly (e.g., it\n155         // is a normal sample):\n156         if (calculated_value > epsilon) cross_validation_error_count += 1;\n157       } else { // target training output is NORMAL\n158         if (calculated_value < epsilon) cross_validation_error_count += 1;\n159       }\n160     }\n161     System.out.println(\"   cross_validation_error_count = \" +\n162                        cross_validation_error_count +\n163                        \" for epsilon = \" + epsilon);\n164     return cross_validation_error_count;\n165   }\n166 \n167   private double test(double epsilon) {\n168     double num_false_positives = 0, num_false_negatives = 0;\n169     double num_true_positives = 0, num_true_negatives = 0;\n170     for (int nt=0; nt<this.num_testing_examples; nt++) {\n171       double[] x = this.testing_examples[nt];\n172       double calculated_value = p(x);\n173       if (x[9] > 0.5) { // target training output is ANOMALY\n174         if (calculated_value > epsilon) num_false_negatives++;\n175         else                            num_true_positives++;\n176       } else { // target training output is NORMAL\n177         if (calculated_value < epsilon) num_false_positives++;\n178         else                            num_true_negatives++;\n179       }\n180     }\n181     double precision = num_true_positives /\n182                        (num_true_positives + num_false_positives);\n183     double recall = num_true_positives /\n184                     (num_true_positives + num_false_negatives);\n185     double F1 = 2 * precision * recall / (precision + recall);\n186 \n187     System.out.println(\"\\n\\n -- best epsilon = \" + this.best_epsilon);\n188     System.out.println(\" -- number of test examples = \" +\n189                        this.num_testing_examples);\n190     System.out.println(\" -- number of false positives = \" + num_false_positives);\n191     System.out.println(\" -- number of true positives = \" + num_true_positives);\n192     System.out.println(\" -- number of false negatives = \" + num_false_negatives);\n193     System.out.println(\" -- number of true negatives = \" + num_true_negatives);\n194     System.out.println(\" -- precision = \" + precision);\n195     System.out.println(\" -- recall = \" + recall);\n196     System.out.println(\" -- F1 = \" + F1);\n197     return F1;\n198   }\n199 \n200   double best_epsilon = 0.02;\n201   private double [] mu;  // [num_features]\n202   private double [] sigma_squared; // [num_features]\n203   private int num_features;\n204   private static double SQRT_2_PI = 2.50662827463;\n205 \n206 \n207   private double[][] training_examples; // [num_features][num_training_examples]\n208   // [num_features][num_training_examples]:\n209   private double[][] cross_validation_examples; \n210   private double[][] testing_examples; // [num_features][num_training_examples]\n211   private int num_training_examples;\n212   private int num_cross_validation_examples;\n213   private int num_testing_examples;\n214 \n215 }\n

\n\n\n\n

Once the training data and the values of \$\\mu\$ (the varible mu in the code) are defined for each feature we can define the method train in lines 86 through 104 that calculated the best epsilon “cutoff” value for the training data set using the method train_helper defined in lines 138 through 165. We use the “best” epsilon value by testing with the separate cross validation data set; we do this by calling the method test that is defined in lines 167 through 198.

\n\n

Example Using the University of Wisconsin Cancer Data

\n\n

The example in this section loads the University of Wisconsin data and uses the class AnomalyDetection developed in the last section to find anomalies, which for this example will be input vectors that represented malignancy in the original data.

\n\n

 1 package com.markwatson.anomaly_detection;\n 2 \n 3 import java.io.*;\n 4 import org.apache.commons.io.FileUtils;\n 5 \n 6 /**\n 7  * Train a deep belief network on the University of Wisconsin Cancer Data Set.\n 8  */\n 9 public class WisconsinAnomalyDetection {\n10 \n11   private static boolean PRINT_HISTOGRAMS = true;\n12 \n13   public static void main(String[] args) throws Exception {\n14 \n15     String [] lines =\n16       FileUtils.readFileToString(\n17          new File(\"data/cleaned_wisconsin_cancer_data.csv\")).split(\"\\n\");\n18     double [][] training_data = new double[lines.length][];\n19     int line_count = 0;\n20     for (String line : lines) {\n21       String [] sarr = line.split(\",\");\n22       double [] xs = new double[10];\n23       for (int i=0; i<10; i++) xs[i] = Double.parseDouble(sarr[i]);\n24       for (int i=0; i<9; i++) xs[i] *= 0.1;\n25       xs[9] = (xs[9] - 2) * 0.5; // make target output be [0,1] instead of [2,4]\n26       training_data[line_count++] = xs;\n27     }\n28 \n29     if (PRINT_HISTOGRAMS) {\n30       PrintHistogram.historam(\"Clump Thickness\", training_data, 0, 0.0, 1.0, 10);\n31       PrintHistogram.historam(\"Uniformity of Cell Size\", training_data,\n32                               1, 0.0, 1.0, 10);\n33       PrintHistogram.historam(\"Uniformity of Cell Shape\", training_data,\n34                               2, 0.0, 1.0, 10);\n35       PrintHistogram.historam(\"Marginal Adhesion\", training_data,\n36                               3, 0.0, 1.0, 10);\n37       PrintHistogram.historam(\"Single Epithelial Cell Size\", training_data,\n38                               4, 0.0, 1.0, 10);\n39       PrintHistogram.historam(\"Bare Nuclei\", training_data, 5, 0.0, 1.0, 10);\n40       PrintHistogram.historam(\"Bland Chromatin\", training_data, 6, 0.0, 1.0, 10);\n41       PrintHistogram.historam(\"Normal Nucleoli\", training_data, 7, 0.0, 1.0, 10);\n42       PrintHistogram.historam(\"Mitoses\", training_data, 8, 0.0, 1.0, 10);\n43     }\n44     \n45     AnomalyDetection detector = new AnomalyDetection(10, line_count - 1,\n46                                                      training_data);\n47 \n48     // the train method will print training results like\n49     // precision, recall, and F1:\n50     detector.train();\n51 \n52     // get best model parameters:\n53     double best_epsilon = detector.bestEpsilon();\n54     double [] mean_values = detector.muValues();\n55     double [] sigma_squared = detector.sigmaSquared();\n56 \n57     // to use this model, us the method AnomalyDetection.isAnamoly(double []):\n58 \n59     double [] test_malignant = new double[] {0.5,1,1,0.8,0.5,0.5,0.7,1,0.1};\n60     double [] test_benign = new double[] {0.5,0.4,0.5,0.1,0.8,0.1,0.3,0.6,0.1};\n61     boolean malignant_result = detector.isAnamoly(test_malignant);\n62     boolean benign_result = detector.isAnamoly(test_benign);\n63     System.out.println(\"\\n\\nUsing the trained model:\");\n64     System.out.println(\"malignant result = \" + malignant_result +\n65                        \", benign result = \" + benign_result);\n66   }\n67 }\n

\n\n\n\n

Data used by an anomaly detecton model should have (roughly) a Gaussian (bell curve shape) distribution. What form does the cancer data have? Unfortunately, each of the data features seems to either have a greater density at the lower range of feature values or large density at the extremes of the data feature ranges. This will cause our model to not perform as well as we would like. Here are the inputs displayed as five-bin histograms:

\n\n

Clump Thickness\n0\t177\n1\t174\n2\t154\n3\t63\n4\t80\n\nUniformity of Cell Size\n0\t393\n1\t86\n2\t54\n3\t43\n4\t72\n\nUniformity of Cell Shape\n0\t380\n1\t92\n2\t58\n3\t54\n4\t64\n\nMarginal Adhesion\n0\t427\n1\t85\n2\t42\n3\t37\n4\t57\n\nSingle Epithelial Cell Size\n0\t394\n1\t117\n2\t76\n3\t28\n4\t33\n\nBare Nuclei\n0\t409\n1\t44\n2\t33\n3\t27\n4\t135\n\nBland Chromatin\n0\t298\n1\t182\n2\t41\n3\t97\n4\t30\n\nNormal Nucleoli\n0\t442\n1\t56\n2\t39\n3\t37\n4\t74\n\nMitoses\n0\t567\n1\t42\n2\t8\n3\t17\n4\t14\n

\n\n\n\n

I won’t do it in this example, but the feature “Bare Nuclei” should be removed because it is not even close to being a bell-shaped distribution. Another thing that you can do (recommended by Andrew Ng in his Coursera Machine Learning class) is to take the log of data and otherwise transform it to something that looks more like a Gaussian distribution. In the class WisconsinAnomalyDetection, you could for example, transform the data using something like:

\n\n

1   // make the data look more like a Gaussian (bell curve shaped) distribution:\n2   double min = 1.e6, max = -1.e6;\n3   for (int i=0; i<9; i++) {\n4     xs[i] = Math.log(xs[i] + 1.2);\n5     if (xs[i] < min) min = xs[i];\n6     if (xs[i] > max) max = xs[i];\n7   }\n8   for (int i=0; i<9; i++) xs[i] = (xs[i] - min) / (max - min);\n

\n\n\n\n

The constant 1.2 in line 4 is a tuning parameter that I got by trial and error by iterating on adjusting the factor and looking at the data histograms.

\n\n

In a real application you would drop features that you can not transform to something like a Gaussian distribution.

\n\n

Here are the results of running the code as it is in the github repository for this book:

\n\n

 -- best epsilon = 0.28\n -- number of test examples = 63\n -- number of false positives = 0.0\n -- number of true positives = 11.0\n -- number of false negatives = 8.0\n -- number of true negatives = 44.0\n -- precision = 1.0\n -- recall = 0.5789473684210527\n -- F1 = 0.7333333333333334\n\n\nUsing the trained model:\nmalignant result = true, benign result = false\n

\n\n\n\n

How do we evaluate these results? The precision value of 1.0 means that there were no false positives. False positives are predictions of a true result when it should have been false. The value 0.578 for recall means that of all the samples that should have been classified as positive, we only predicted about 57.8% of them. The F1 score is calculated as two times the product of precision and recall, divided by the sum of precision plus recall.

\n\n\n

Deep Learning Using Deeplearning4j

\n\n

The Deeplearning4j.org Java library supports several neural network algorithms including support for Deep Learning (DL). We will look at an example of DL implementing Deep-Belief networks using the same University of Wisconsin cancer database that we used in the chapters on machine learning with Spark and on anomaly detection. Deep learning refers to neural networks with many layers, possibly with weights connecting neurons in non-adjacent layers which makes it possible to model temporal and spacial patterns in data.

\n\n

I am going to assume that you have some knowledge of simple backpropagation neural networks. If you are unfamiliar with neural networks you might want to pause and do a web search for “neural networks backpropagation tutorial” or read the neural network chapter in my book Practical Artificial Intelligence Programming With Java, 4th edition.

\n\n

The difficulty in training many layer networks with backpropagation is that the delta errors in the layers near the input layer (and far from the output layer) get very small and training can take a very long time even on modern processors and GPUs. Geoffrey Hinton and his colleagues created a new technique for pretraining weights. In 2012 I took a Coursera course taught by Hinton and some colleagues titled ‘Neural Networks for Machine Learning’ and the material may still be available online when you read this book.

\n\n

For Java developers, Deeplearning4j is a great starting point for experimenting with deep learning. If you also use Python then there are good tutorials and other learning assets at deeplearning.net.

\n\n

Deep Belief Networks

\n\n

Deep Belief Networks (DBN) are a type of deep neural network containing multiple hidden neuron layers where there are no connections between neurons inside any specific hidden layer. Each hidden layer learns features based on training data an the values of weights from the previous hidden layer. By previous layer I refer to the connected layer that is closer to the input layer. A DBN can learn more abstract features, with more abstraction in the hidden layers “further” from the input layer.

\n\n

DBNs are first trained a layer at a time. Initially a set of training inputs is used to train the weights between the input and the first hidden layer of neurons. Technically, as we preliminarily train each succeeding pair of layers we are training a restricted Boltzmann machine (RBM) to learn a new set of features. It is enough for you to know at this point that RBMs are two layers, input and output, that are completely connected (all neurons in the first layer have a weighted connection to each neuron in the next layer) and there are no inner-layer connections.

\n\n

As we progressively train a DBN, the output layer for one RBM becomes the input layer for the next neuron layer pair for preliminary training.

\n\n

Once the hidden layers are all preliminarily trained then backpropagation learning is used to retrain the entire network but now delta errors are calculated by comparing the forward pass outputs with the training outputs and back-propagating errors to update weights in layers proceeding back towards the first hidden layer.

\n\n

The important thing to understand is that backpropagation tends to not work well for networks with many hidden layers because the back propagated errors get smaller as we process backwards towards the input neurons - this would cause network training to be very slow. By precomputing weights using RBM pairs, we are closer to a set of weights to minimize errors over the training set (and the test set).

\n\n

Deep Belief Example

\n\n

The following screen show shows an IntelliJ project (you can use the free community orprofessional version for the examples in this book) for the example in this chapter:

\n\n\n

The Deeplearning4j library uses user-written Java classes to import training and testing data into a form that the Deeplearning4j library can use. The following listing shows the implementation of the class WisconsinDataSetIterator that iterates through the University of Wisconsin cancer data set:

\n\n

 1 package com.markwatson.deeplearning;\n 2 \n 3 import org.deeplearning4j.datasets.iterator.BaseDatasetIterator;\n 4 \n 5 import java.io.FileNotFoundException;\n 6 \n 7 \n 8 public class WisconsinDataSetIterator extends BaseDatasetIterator {\n 9   private static final long serialVersionUID = -2023454995728682368L;\n10 \n11   public WisconsinDataSetIterator(int batch, int numExamples)\n12          throws FileNotFoundException {\n13     super(batch, numExamples, new WisconsinDataFetcher());\n14   }\n15 }\n

\n\n\n\n

The WisconsinDataSetIterator constructor calls its super class with an instance of the class WisconsinDataFetcher (defined in the next listing) to read the Comma Separated Values (CSV) spreadsheet data from the file data/cleaned_wisconsin_cancer_data.csv:

\n\n

 1 package com.markwatson.deeplearning;\n 2 \n 3 import org.deeplearning4j.datasets.fetchers.CSVDataFetcher;\n 4 \n 5 import java.io.FileInputStream;\n 6 import java.io.FileNotFoundException;\n 7 \n 8 /**\n 9  * Created by markw on 10/5/15.\n10  */\n11 public class WisconsinDataFetcher extends CSVDataFetcher {\n12 \n13   public WisconsinDataFetcher() throws FileNotFoundException {\n14     super(new FileInputStream(\"data/cleaned_wisconsin_cancer_data.csv\"), 9);\n15   }\n16   @Override\n17   public void fetch(int i) {\n18     super.fetch(i);\n19   }\n20 \n21 }\n

\n\n\n\n

In line 14, the last argument 9 defines which column in the input CSV file contains the label data. This value is zero indexed so if you look at the input file data/cleaned_wisconsin_cancer_date.csv this will be the last column. Values of 4 in the last column idicate malignant and values of 2 indicate not malignant.

\n\n

The following listing shows the definition of the class DeepBeliefNetworkWisconsinData that reads the University of Wisconsin cancer data set using the code in the last two listings, randmly selects part of it to use for training and for testing, creates a DBN, and tests it. The value of the variable numHidden set in line 51 refers to the number of neurons in each hidden layer. Setting numberOfLayers to 3 in line 52 indicates that we will just use a single hidden layer since this value (3) also counts the input and output layers.

\n\n

The network is configured and constructed in lines 83 through 104. If we increased the number of hidden units (something that you might do for more complex problems) then you would repeat lines 88 through 97 to add a new hidden layer, and you would change the layer indices (first argument) as appropriate in calls to the chained method .layer().

\n\n

  1 package com.markwatson.deeplearning;\n  2 \n  3 import org.deeplearning4j.datasets.iterator.DataSetIterator;\n  4 import org.deeplearning4j.eval.Evaluation;\n  5 import org.deeplearning4j.nn.conf.MultiLayerConfiguration;\n  6 import org.deeplearning4j.nn.conf.NeuralNetConfiguration;\n  7 import org.deeplearning4j.nn.conf.layers.OutputLayer;\n  8 import org.deeplearning4j.nn.conf.layers.RBM;\n  9 import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;\n 10 import org.deeplearning4j.nn.weights.WeightInit;\n 11 import org.nd4j.linalg.api.ndarray.INDArray;\n 12 import org.nd4j.linalg.dataset.DataSet;\n 13 import org.nd4j.linalg.dataset.SplitTestAndTrain;\n 14 import org.nd4j.linalg.factory.Nd4j;\n 15 import org.nd4j.linalg.lossfunctions.LossFunctions;\n 16 import org.slf4j.Logger;\n 17 import org.slf4j.LoggerFactory;\n 18 \n 19 import java.io.*;\n 20 import java.nio.file.Files;\n 21 import java.nio.file.Paths;\n 22 import java.util.Random;\n 23 \n 24 import org.apache.commons.io.FileUtils;\n 25 \n 26 /**\n 27  * Train a deep belief network on the University of Wisconsin Cancer Data Set.\n 28  */\n 29 public class DeepBeliefNetworkWisconsinData {\n 30 \n 31   private static Logger log =\n 32     LoggerFactory.getLogger(DeepBeliefNetworkWisconsinData.class);\n 33 \n 34   public static void main(String[] args) throws Exception {\n 35 \n 36     final int numInput = 9;\n 37     int outputNum = 2;\n 38     /**\n 39      * F1 scores as a function of the number of hidden layer\n 40      * units hyper-parameter:\n 41      *\n 42      *    numHidden   F1\n 43      *    ---------   --\n 44      *            2   0.50734\n 45      *            3   0.87283 (value to use - best result / smallest network)\n 46      *           13   0.87283\n 47      *          100   0.55987 (over fitting)\n 48      *\n 49      *   Other hyper parameters held constant: batchSize = 648\n 50      */\n 51     int numHidden = 3;\n 52     int numberOfLayers = 3; // input, hidden, output\n 53     int numSamples = 648;\n 54 \n 55     /**\n 56      * F1 scores as a function of the number of batch size:\n 57      *\n 58      *    batchSize   F1\n 59      *    ---------   --\n 60      *           30   0.50000\n 61      *           64   0.78787\n 62      *          323   0.67123\n 63      *          648   0.87283 (best to process all training vectors in one batch)\n 64      *\n 65      *   Other hyper parameters held constant: numHidden = 3\n 66      */\n 67     int batchSize = 648; // equal to number of samples\n 68 \n 69     int iterations = 100;\n 70     int fractionOfDataForTraining = (int) (batchSize * 0.7);\n 71     int seed = 33117;\n 72 \n 73     DataSetIterator iter = new WisconsinDataSetIterator(batchSize, numSamples);\n 74     DataSet next = iter.next();\n 75     next.normalizeZeroMeanZeroUnitVariance();\n 76 \n 77     SplitTestAndTrain splitDataSets =\n 78 \t  next.splitTestAndTrain(fractionOfDataForTraining, new Random(seed));\n 79     DataSet train = splitDataSets.getTrain();\n 80     DataSet test = splitDataSets.getTest();\n 81 \n 82     MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()\n 83         .seed(seed) //use the same random seed\n 84         .iterations(iterations)\n 85         .l1(1e-1).regularization(true).l2(2e-4)\n 86         .list(numberOfLayers - 1) // don't count the input layer\n 87         .layer(0,\n 88             new RBM.Builder(RBM.HiddenUnit.RECTIFIED, RBM.VisibleUnit.GAUSSIAN)\n 89                 .nIn(numInput)\n 90                 .nOut(numHidden)\n 91                 // set variance of random initial weights based on\n 92                 // input and output layer size:\n 93                 .weightInit(WeightInit.XAVIER)\n 94                 .dropOut(0.25)\n 95                 .build()\n 96         )\n 97         .layer(1, new OutputLayer.Builder(LossFunctions.LossFunction.MCXENT)\n 98             .nIn(numHidden)\n 99             .nOut(outputNum)\n100             .activation(\"softmax\")\n101             .build()\n102         )\n103         .build();\n104     MultiLayerNetwork model = new MultiLayerNetwork(conf);\n105     model.init();\n106     model.fit(train);\n107 \n108     log.info(\"\\nEvaluating model:\\n\");\n109     Evaluation eval = new Evaluation(outputNum);\n110     INDArray output = model.output(test.getFeatureMatrix());\n111 \n112     for (int i = 0; i < output.rows(); i++) {\n113       String target = test.getLabels().getRow(i).toString();\n114       String predicted = output.getRow(i).toString();\n115       log.info(\"target: \" + target + \" predicted: \" + predicted);\n116     }\n117 \n118     eval.eval(test.getLabels(), output);\n119     log.info(eval.stats());\n120 \n121     /**\n122      * Save the model for reuse:\n123      */\n124     OutputStream fos = Files.newOutputStream(Paths.get(\"saved-model.bin\"));\n125     DataOutputStream dos = new DataOutputStream(fos);\n126     Nd4j.write(model.params(), dos);\n127     dos.flush();\n128     dos.close();\n129     FileUtils.writeStringToFile(\n130        new File(\"conf.json\"),\n131                 model.getLayerWiseConfigurations().toJson());\n132 \n133     /**\n134      * Load saved model and test again:\n135      */\n136     log.info(\"\\nLoad saved model from disk:\\n\");\n137     MultiLayerConfiguration confFromJson =\n138 \t  MultiLayerConfiguration.fromJson(\n139                      FileUtils.readFileToString(new File(\"conf.json\")));\n140     DataInputStream dis = new DataInputStream(\n141                                 new FileInputStream(\"saved-model.bin\"));\n142     INDArray newParams = Nd4j.read(dis);\n143     dis.close();\n144     MultiLayerNetwork savedModel = new MultiLayerNetwork(confFromJson);\n145     savedModel.init();\n146     savedModel.setParameters(newParams);\n147 \n148     log.info(\"\\nEvaluating model loaded from disk:\\n\");\n149     Evaluation eval2 = new Evaluation(outputNum);\n150     INDArray output2 = savedModel.output(test.getFeatureMatrix());\n151 \n152     for (int i = 0; i < output2.rows(); i++) {\n153       String target = test.getLabels().getRow(i).toString();\n154       String predicted = output.getRow(i).toString();\n155       log.info(\"target: \" + target + \" predicted: \" + predicted);\n156     }\n157 \n158     eval2.eval(test.getLabels(), output2);\n159     log.info(eval2.stats());\n160   }\n161 }\n

\n\n\n\n

In line 71 we set fractionOfDataForTraining to 0.7 which means that we will use 70% of the available data for training and 30% for testing. It is very important to not use training data for testing because performance on recognizing training data should always be good assuming that you have enough memory capacity in a network (i.e., enough hidden units and enough neurons in each hidden layer). In lines 78 through 81 we divide our data into training and testing disjoint sets.

\n\n

In line 86 we are setting three meta learning parameters: learning rate for the first set of weights between the input and hidden layer to be 1e-1, setting the model to use regularization, and setting the learning rate for the hidden to output weights to be 2e-4.

\n\n

In line 95 we are setting the dropout factor, here saying that we will randomly not use 25% of the neurons for any given training example. Along with regularization, using dropout helps prevent overfitting. Overfitting occurs when a neural netowrk fails to generalize and learns noise in training data. The goal is to learn important features that affect the utility of a trained model for processing new data. We don’t want to learn random noise in the data as important features. Another way to prevent overfitting is to use the smallest possible number of neurons in hidden labels and still perform well on the independent test data.

\n\n

After training the model in lines 105 through 107, we test the model (lines 110 through 120) on the separate test data. The program output when I ran the model is:

\n\n

Evaluating model:\n\n14:59:46.872 [main] INFO  c.m.d.DeepBeliefNetworkWisconsinData -\n                                  target: [ 1.00, 0.00] predicted: [ 0.76, 0.24]\n14:59:46.873 [main] INFO  c.m.d.DeepBeliefNetworkWisconsinData -\n                                  target: [ 1.00, 0.00] predicted: [ 0.65, 0.35]\n\nActual Class 0 was predicted with Predicted 0 with count 151 times\n\nActual Class 1 was predicted with Predicted 0 with count 44 times\n\n==========================Scores========================================\n Accuracy:  0.7744\n Precision: 0.7744\n Recall:    1\n F1 Score:  0.8728323699421966\n===========================================================================\n

\n\n\n\n

The F1 score is calculated as twice precision times recall, all divided by precision + recall. We would like F1 to be as close to 1.0 as possible and it is common to spend a fair amount of time experimenting with meta learning paramters to increase F1.

\n\n

It is also fairly common to try to learn good values of meta learning paramters also. We won’t do this here but the process involves splitting the data into three disjoint sets: training, validation, and testing. The meta parameters are varied, training is performed, and using the validation data the best set of meta parameters is selected. Finally, we test the netowrk as defined my meta parameters and learned weights for those meta parameters with the separate test data to see what the effective F1 score is.

\n\n

Deep Learning Wrapup

\n\n

I first used complex neural network topologies in the late 1980s for phoneme (speech) recognition, specifically using time delay neural networks and I gave a talk about it at IEEE First Annual International Conference on Neural Networks San Diego, California June 21-24, 1987. Back then, neural networks were not really considered to be a great technology for this application but in the present time Google, Microsoft, and other companies are using deep (many layered) neural networks for speech and image recognition. Exciting work is also being done in the field of natural language processing. I just provided a small example in this chapter that you can experiment with easily. I wanted to introduce you to Deeplearning4j because I think it is probably the easiest way for Java developers to get started working with many layered neural networks and I refer you to the project documentation.

\n\n\n

Web Scraping Examples

\n\n

Except for the first chapter on network programming techniques, this chapter, and the final chapter on what I call Knowledge Management-Lite, this book is primarily about machine learning in one form or another. As a practical matter, much of the data that many people use for machine learning either comes from the web or from internal data sources. This short chapter provides some guidance and examples for getting text data from the web.

\n\n

In my work I usually use the Ruby scripting language for web scraping and information gathering (as I wrote about in my APress book Scripting Intelligence Web 3.0 Information Gathering and Processing) but there is also good support for using Java for web scraping and since this is a book on modern Java development, we will use Java in this chapter.

\n\n

Before we start a technical discusion about “web scraping” I want to point out to you that much of the information on the web is copyright and the first thing that you should do is to read the terms of service for web sites to insure that your use of “scraped” or “spidered” data conforms with the wishes of the persons or organizations who own the content and pay to run scraped web sites.

\n\n

Motivation for Web Scraping

\n\n

As we will see in the next chapter on linked data there is a huge amount of structured data available on the web via web services, semantic web/linked data markup, and APIs. That said, you will frequently find data that is useful to pull raw text from web sites but this text is usually fairly unstructured and in a messy (and frequently changing) format as web pages meant for human consumption and not meant to be ingested by software agents. In this chapter we will cover useful “web scraping” techniques. You will see that there is often a fair amount of work in dealing with different web design styles and layouts. To make things even more inconvenient you might find that your software information gathering agents will often break because of changes in web sites.

\n\n

I tend to use one of three general techniques for scraping web sites. Only the first two will be covered in this chapter:

\n\n

Use an HTML parsing library that strips all HTML markup and Javascript from a page and returns a “pure text” block of text. Here the text in navigation menus, headers, etc. will be interspersed with what we might usually think of a “content” from a web site.
Exploit HTML DOM (Document Omject MOdel) formatting information on web sites to pick out headers, page titles, navigation menues, and large blocks of content text.
Use a tool like (Selenium)[http://docs.seleniumhq.org/] to programatically control a web browser so your software agents can login to site and otherwise perform navigation. In other words your software agents can simulate a human using a web browser.

\n\n

I seldom need to use tools like Selenium but as the saying goes “when you need them, you need them.” For simple sites I favor extracting all text as a single block and use DOM processing as needed.

\n\n

I am not going to cover the use of Selenium and the Java Selenium Web-Driver APIs in this chapter because, as I mentioned, I tend to not use it frequently and I think that you are unlikely to need to do so either. I refer you to the Selenium documentation if the first two approaches in the last list do not work for your application. Selenium is primarily intended for building automating testing of complex web applications, so my occasional use in web spidering is not the common use case.

\n\n

I assume that you have some experience with HTML and DOM. For reference, the following figure shows a small part of the DOM for a page on one of my web sites:

\n\n\n

\"Using — Using the Chrome Developer Tools

\n\n\n

This screen shot shows the Chrome web browser developer tools, specifically viewing the page’s DOM. Since a DOM is a tree data structure it is useful to be able to collapse or to expand sub-trees in the DOM. In this figure, the HTML BODY element contains two top level DIV elements. The first DIV that contains the navigation menu for my site is collapes. The second DIV contains an H2 heading and various nested DIV and P (paragraph) elements. I show this fragment of my web pagenot as an example of clean HTML coding but rather as an example of how messy and nested web page elements can be.

\n\n

Using the jsoup Library

\n\n

We will use the MIT licensed library jsoup. One reason I selected jsoup for the examples in this chapter out of many fine libraries that provide similar functionality is the particularly nice documentation, especially The jsoup Cookbook which I urge you to bookmark as a general reference. In this chapter I will concentrate on just the most frequent web scraping use cases that I use in my own work.

\n\n

The following bit of code uses jsoup to get the text inside all P (paragraph) elements that are direct children of any DIV element. On line 14 we use the jsoup library to fetch my home web page:

\n\n

 1 package com.markwatson.web_scraping;\n 2 \n 3 import org.jsoup.*;\n 4 import org.jsoup.nodes.Document;\n 5 import org.jsoup.nodes.Element;\n 6 import org.jsoup.select.Elements;\n 7 \n 8 /**\n 9  * Examples of using jsoup\n10  */\n11 public class MySitesExamples {\n12 \n13   public static void main(String[] args) throws Exception {\n14     Document doc = Jsoup.connect(\"http://www.markwatson.com\").get();\n15     Elements newsHeadlines = doc.select(\"div p\");\n16     for (Element element : newsHeadlines) {\n17       System.out.println(\" next element text: \" + element.text());\n18     }\n19   }\n20 }\n

\n\n\n\n

In line 15 I am selecting the pattern that returns all P elements that are direct children of any DIV element and in lines 16 to 18 print the text inside these P elements.

\n\n

For training data for machine learning it is useful to just grab all text on a web page and assume that common phrases dealing with web navigaion, etc. will be dropped from learned models because they occur in many different training examples for different classifications. The following code snippet shows how to fetch the plain text from an entire web page:

\n\n

    Document doc = Jsoup.connect(\"http://www.markwatson.com\").get();\n    String all_page_text = doc.text();\n    System.out.println(\"All text on web page:\\n\" + all_page_text);\n

\n\n\n\n

All text on web page:\nMark Watson: Consultant specializing in machine learning and artificial intellig\\\nence Toggle navigation Mark Watson Home page Consulting Free mentoring Blog Book\\\ns Open Source Fun Consultant specializing in machine learning, artificial intell\\\nigence, cognitive computing, and web engineering...\n

\n\n\n\n

The 2gram (i.e., two words in sequence) “Toggle navigation” in the last listing has nothing to do with the real content in my site and is an artifact of using the Bootstrap CSS and Javascript tools. Often “noise” like this is simply ignored by machine learning models if it appears on many different sites but beware that this might be a problem and you might need to precisiely fetch text from specific DOM elements. Similarly, notice that this last listing picks up the plain text from the navigation menus.

\n\n

The following code snippet finds HTML anchor elements and prints the data associated with these elements:

\n\n

    Document doc = Jsoup.connect(\"http://www.markwatson.com\").get();\n    Elements anchors = doc.select(\"a[href]\");\n    for (Element anchor : anchors) {\n      String uri = anchor.attr(\"href\");\n      System.out.println(\" next anchor uri: \" + uri);\n      System.out.println(\" next anchor text: \" + anchor.text());\n    }\n

\n\n\n\n

 1  next anchor uri: #\n 2  next anchor text: Mark Watson\n 3  next anchor uri: /\n 4  next anchor text: Home page\n 5  next anchor uri: /consulting/\n 6  next anchor text: Consulting\n 7  next anchor uri: /mentoring/\n 8  next anchor text: Free mentoring\n 9  next anchor uri: http://blog.markwatson.com\n10  next anchor text: Blog\n11  next anchor uri: /books/\n12  next anchor text: Books\n13  next anchor uri: /opensource/\n14  next anchor text: Open Source\n15  next anchor uri: /fun/\n16  next anchor text: Fun\n17  next anchor uri: http://www.cognition.tech\n18  next anchor text: www.cognition.tech\n19  next anchor uri: https://github.com/mark-watson\n20  next anchor text: GitHub\n21  next anchor uri: https://plus.google.com/117612439870300277560\n22  next anchor text: Google+\n23  next anchor uri: https://twitter.com/mark_l_watson\n24  next anchor text: Twitter\n25  next anchor uri: http://www.freebase.com/m/0b6_g82\n26  next anchor text: Freebase\n27  next anchor uri: https://www.wikidata.org/wiki/Q18670263\n28  next anchor text: WikiData\n29  next anchor uri: https://leanpub.com/aijavascript\n30  next anchor text: Build Intelligent Systems with JavaScript\n31  next anchor uri: https://leanpub.com/lovinglisp\n32  next anchor text: Loving Common Lisp, or the Savvy Programmer's Secret Weapon\n33  next anchor uri: https://leanpub.com/javaai\n34  next anchor text: Practical Artificial Intelligence Programming With Java\n35  next anchor uri: http://markwatson.com/index.rdf\n36  next anchor text: XML RDF\n37  next anchor uri: http://markwatson.com/index.ttl\n38  next anchor text: Turtle RDF\n39  next anchor uri: https://www.wikidata.org/wiki/Q18670263\n40  next anchor text: WikiData\n

\n\n\n\n

Notice that there are diffent types of URIs like #, relative, and absolute. Any characters following a # character do not affect the routing of which web page is shown (or which API is called) but the characters after the # character are available for use in specifying anchor positions on a web page or extra parameters for API calls. Relative APIs like /consulting/ (as seen in line 5) are understood to be relative to the base URI of the web site.

\n\n

I often require that URIs be absolute URIs (i.e., starts with a protocol like “http:” or “https:”) and the following code snippet selects just absolute URI anchors:

\n\n

1     Elements absolute_uri_anchors = doc.select(\"a[href]\");\n2     for (Element anchor : absolute_uri_anchors) {\n3       String uri = anchor.attr(\"abs:href\");\n4       System.out.println(\" next anchor absolute uri: \" + uri);\n5       System.out.println(\" next anchor absolute text: \" + anchor.text());\n6     }\n

\n\n\n\n

In line 3 I am specifying the attribute as “abs:href” to be more selective:

\n\n

 next anchor absolute text: Mark Watson\n next anchor absolute uri: http://www.markwatson.com/\n next anchor absolute text: Home page\n next anchor absolute uri: http://www.markwatson.com/consulting/\n next anchor absolute text: Consulting\n next anchor absolute uri: http://www.markwatson.com/mentoring/\n next anchor absolute text: Free mentoring\n next anchor absolute uri: http://blog.markwatson.com\n next anchor absolute text: Blog\n ...\n

\n\n\n\n

Wrap Up

\n\n

I just showed you quick reference to the most common use cases for my work. What I didn’t show you was an example for organizing spidered information for reuse.

\n\n

I sometimes collect training data for machine learning by using web searches with query keywords tailored to find information in specific categories. I am not covering automating web search in this book but I would like to refer you to an open source wrapper that I wrote for Microsoft’s Bing Search APIs on github. As an example, just to give you an idea for experimentation, if you wanted to train a model to categorize text containing car descriptions into two classes: “US domestic cars” “foreign made cars” thsn you might use search queries like “cars Ford Chevy” and “cars Volvo Volkswagen Peugeot” to get example text for these two categories.

\n\n

If you use the Bing Search APIs for collecting training data, then you can for the top ranked search results use the techniques covered in this chapter to retreive the text from the original web pages. Then use one or more of the machine learning techniques covered in this book to build classification models. This is a good technique and some people might even consider it a super power.

\n\n

For machine learning, I sometimes collect text in file names that indicate the classification of the text in each file. I often also collect data in a NOSQL datastore like MongoDB or CouchDB, or use a relational database like Postgres to store training data for future reuse.

\n\n\n

Linked Data

\n\n

This chapter introduces you to techniques for information architecture and development using linked data and semantic web technologies. This chapter is paired with the next chapter on knowledge management so please read these two chapters in order.

\n\n

Note: I published “Practical Semantic Web and Linked Data Applications: Java, JRuby, Scala, and Clojure Edition” and you can get a free PDF of this book here. While the technical details of that book are still relevant, I am now more opinionated about which linked data and semantic web technologies are most useful so this chapter is more focused. With some improvements, I use the Java client code for DBPedia search and SPARQL queries from my previous book. I also include new versions that return query results as JSON. I am going to assume that you have fetched a free copy of my book “Practical Semantic Web and Linked Data Applications” and use chapters 6 and 7 for an introduction to the Resource Definition Language (RDF and RDFS) and chapter 8 for a tutorial on the SPARQL query language. It is okay to read this background material later if you want to dive right into the examples in this chapter. I do provide a quick introduction to SPARQL and RDF i the next section, but please do also get a free copy of my semantic web book for a longer treatment. We use the standard query language SPARQL to retrieve information from RDF data sources in much the same way that we use SQL to retrieve information from relational databases.

\n\n

In this chapter we will look at consuming linked data using SPARQL queries against public RDF data stores and searching DBPedia. Using the code from the earlier chapter “Natural Language Processing Using OpenNLP” we will also develop an example application that annotates text by tagging entities with unique DBPedia URIs for the entities (we will use this example also in the next chapter in a code example for knowledge Management).

\n\n

Why use linked data?

\n\n

Before we jump into technical details I want to give you some motivation for why you might use linked data and semantic web technologies to organize and access information. Suppose that your company sells products to customers and your task is to augment the relational databases used for ordering, shipping, and customer data with a new system that needs to capture relationships between products, customers, and information sources on the web that might help both your customers and customer support staff increase the utility of your products to your customers.

\n\n

Customers, products, and information sources are entities that will sometimes be linked with properties or relationships with other entities. This new data will be unstructured and you do not know ahead of time all of the relationships and what new types of entities that the new system will use. If you come from a relational database background using an sort of graph database might be uncomfortable at first. As motivation consider that Google uses its Knowledge Graph (I customized the Knowledge Graph when I worked at Google) for internal applications, to add value to search results, and to power Google Now. Facebook uses its Open Graph to store information about uses, user posts, and relationships between users and posted material. These graph data stores are key assets for Google and Facebook. Just because you don’t deal with huge amounts of data does not mean that the flexibility of graph data is not useful for your projects! We will use the DBPedia graph database in this chapter for the example programs. DBPedia (which Google used as core knowledge when developing their Knowledge Graph) is a rich information source for typed data for products, people, companies, places, etc. Connecting internal data in your organization to external data sources like DBPedia increases the utility of yor private data. You can make this data sharing relationship symmetrical by selectively publishing some of your internal data externally as linked data for the benefit of your customers and business partners.

\n\n

The key idea is to design the shape of your information (what entities do you deal with and what types of properties or relationships exist between different types of entities), ingest data into a graph database with a plan for keeping everything up to date, and then provide APIs or a query language so application programmers can access this linked data when developing new applications. In this chapter we will use semantic web standards with the SPARQL query language to access data but these ideas should apply to using other graph databases like Neo4J or non-relational data stores like MongoDB or CouchDB.

\n\n

Example Code

\n\n

The examples for this chapter are set up to run either from the command line or using IntelliJ.

\n\n

There are three examples in this chapter. If you want to try running the code before reading the rest of the chapter, run each maven test separately:

\n\n

1     mvn test -Dtest=AnnotationTest\n2     mvn test -Dtest=DBPediaTest\n3     mvn test -Dtest=SparqlTest\n

\n\n\n\n

The following figure shows the project for this chapter in the Community Edition of IntelliJ:

\n\n\n

Overview of RDF and SPARQL

\n\n

The Resource Description Framework (RDF) is used to store information\nas subject/predicate/object triple values. RDF data was originally encoded as XML and intended for automated\nprocessing. In this chapter we will use two simple to read formats\ncalled “N-Triples” and “N3.” RDF data consists of a set of triple values:

\n\n

subject
predicate
object

\n\n

As an example, suppose our application involved\nnews stories and we wanted a way to store and query meta data. We might do this by extracting semantic information from the text, and storing\nit in RDF. I will use this application domain for the examples in this\nchapter. We might use triples like:

\n\n

subject: a URL (or URI) of a news article
predicate: a relation like “containsPerson”
object: a value like “Bill Clinton”

\n\n

As previously mentioned, we will use either URIs or string literals as\nvalues for subjects and objects. We will always use URIs for the values\nof predicates. In any case URIs are usually preferred to string literals\nbecause they are unique. We will see an example of this preferred use\nbut first we need to learn the N-Triple and N3 RDF formats.

\n\n

Any part of a triple (subject, predicate, or object) is either a URI or\na string literal. URIs encode namespaces. For example, the\ncontainsPerson predicate in the last example could properly be written\nas:

\n\n

1 http://knowledgebooks.com/ontology/#containsPerson\n

\n\n\n\n

The first part of this URI is considered to be the namespace for (what\nwe will use as a predicate) “containsPerson.” When different RDF triples\nuse this same predicate, this is some assurance to us that all users of\nthis predicate subscribe to the same meaning. Furthermore, we will see\nin Section on RDFS that we can use RDFS to state equivalency between\nthis predicate (in the namespace http://knowledgebooks.com/ontology/)\nwith predicates represented by different URIs used in other data\nsources. In an “artificial intelligence” sense, software that we write\ndoes not understand a predicate like “containsPerson” in the way that a\nhuman reader can by combining understood common meanings for the words\n“contains” and “person” but for many interesting and useful types of\napplications that is fine as long as the predicate is used consistently.\nWe will see shortly that we can define abbreviation prefixes for\nnamespaces which makes RDF and RDFS files shorter and easier to read.

\n\n

A statement in N-Triple format consists of three URIs (or string\nliterals – any combination) followed by a period to end the statement.\nWhile statements are often written one per line in a source file they\ncan be broken across lines; it is the ending period which marks the end\nof a statement. The standard file extension for N-Triple format files is\n*.nt and the standard format for N3 format files is *.n3.

\n\n

My preference is to use N-Triple format files as output from programs\nthat I write to save data as RDF. I often use Sesame to convert N-Triple\nfiles to N3 if I will be reading them or even hand editing them. You\nwill see why I prefer the N3 format when we look at an example:

\n\n

1 @prefix kb:  <http://knowledgebooks.com/ontology#> .\n2 <http://news.com/201234 /> kb:containsCountry \"China\"  .\n

\n\n\n\n

Here we see the use of an abbreviation prefix “kb:” for the namespace\nfor my company KnowledgeBooks.com ontologies. The first term in the RDF\nstatement (the subject) is the URI of a news article. The second term\n(the predicate) is “containsCountry” in the “kb:” namespace. The last\nitem in the statement (the object) is a string literal “China.” I would\ndescribe this RDF statement in English as, “The news article at URI\nhttp://news.com/201234 mentions the country China.”

\n\n

This was a very simple N3 example which we will expand to show\nadditional features of the N3 notation. As another example, suppose that\nthis news article also mentions the USA. Instead of adding a whole new\nstatement like this:

\n\n

1     @prefix kb:  <http://knowledgebooks.com/ontology#> .\n2     <http://news.com/201234 /> kb:containsCountry \"China\"  .\n3     <http://news.com/201234 /> kb:containsCountry \"USA\"  .\n

\n\n\n\n

we can combine them using N3 notation. N3 allows us to collapse multiple\nRDF statements that share the same subject and optionally the same\npredicate:

\n\n

1     @prefix kb:  <http://knowledgebooks.com/ontology#> .\n2     <http://news.com/201234 /> kb:containsCountry \"China\" ,\n3                                                   \"USA\" .\n

\n\n\n\n

We can also add in additional predicates that use the same subject:

\n\n

 1     @prefix kb:  <http://knowledgebooks.com/ontology#> .\n 2 \n 3     <http://news.com/201234 /> kb:containsCountry \"China\" ,\n 4                                                   \"USA\" .\n 5         kb:containsOrganization \"United Nations\" ;\n 6         kb:containsPerson \"Ban Ki-moon\" , \"Gordon Brown\" ,\n 7                           \"Hu Jintao\" , \"George W. Bush\" ,\n 8                           \"Pervez Musharraf\" ,\n 9                           \"Vladimir Putin\" , \n10                           \"Mahmoud Ahmadinejad\" .\n

\n\n\n\n

This single N3 statement represents ten individual RDF triples. Each\nsection defining triples with the same subject and predicate have\nobjects separated by commas and ending with a period. Please note that\nwhatever RDF storage system we use (we will be using Sesame) it makes no\ndifference if we load RDF as XML, N-Triple, of N3 format files:\ninternally subject, predicate, and object triples are stored in the same\nway and are used in the same way.

\n\n

I promised you that the data in RDF data stores was easy to extend. As\nan example, let us assume that we have written software that is able to\nread online news articles and create RDF data that captures some of the\nsemantics in the articles. If we extend our program to also recognize\ndates when the articles are published, we can simply reprocess articles\nand for each article add a triple to our RDF data store using a form\nlike:

\n\n

1     <http://news.com/201234 /> kb:datePublished\n2                                \"2008-05-11\" .\n

\n\n\n\n

Furthermore, if we do not have dates for all news articles that is often\nacceptable depending on the application.

\n\n

SPARQL is a query language used to query RDF data stores. While SPARQL\nmay initially look like SQL, there are some important\ndifferences like support for RDFS and OWL inferencing. We will cover the basics of SPARQL in this section.

\n\n

We will use the following sample RDF data for the discusion in this section:

\n\n

 1     @prefix kb:  <http://knowledgebooks.com/ontology#> .\n 2     @prefix rdfs:  <http://www.w3.org/2000/01/rdf-schema#> .\n 3 \n 4     kb:containsCity rdfs:subPropertyOf kb:containsPlace .\n 5 \n 6     kb:containsCountry rdfs:subPropertyOf kb:containsPlace .\n 7 \n 8     kb:containsState rdfs:subPropertyOf kb:containsPlace .\n 9 \n10     <http://yahoo.com/20080616/usa_flooding_dc_16 />\n11         kb:containsCity \"Burlington\" , \"Denver\" ,\n12                         \"St. Paul\" ,\" Chicago\" ,\n13                         \"Quincy\" , \"CHICAGO\" ,\n14                         \"Iowa City\" ;\n15         kb:containsRegion \"U.S. Midwest\" , \"Midwest\" ;\n16         kb:containsCountry \"United States\" , \"Japan\" ;\n17         kb:containsState \"Minnesota\" , \"Illinois\" , \n18                          \"Mississippi\" , \"Iowa\" ;\n19         kb:containsOrganization \"National Guard\" ,\n20                          \"U.S. Department of Agriculture\" ,\n21                          \"White House\" ,\n22                          \"Chicago Board of Trade\" ,\n23                          \"Department of Transportation\" ;\n24         kb:containsPerson \"Dena Gray-Fisher\" ,\n25                           \"Donald Miller\" ,\n26                           \"Glenn Hollander\" ,\n27                           \"Rich Feltes\" ,\n28                           \"George W. Bush\" ;\n29         kb:containsIndustryTerm \"food inflation\" , \"food\" ,\n30                                 \"finance ministers\" ,\n31                                 \"oil\" .\n32 \n33     <http://yahoo.com/78325/ts_nm/usa_politics_dc_2 />\n34         kb:containsCity \"Washington\" , \"Baghdad\" ,\n35                         \"Arlington\" , \"Flint\" ;\n36         kb:containsCountry \"United States\" ,\n37                            \"Afghanistan\" ,\n38                            \"Iraq\" ;\n39         kb:containsState \"Illinois\" , \"Virginia\" ,\n40                          \"Arizona\" , \"Michigan\" ;\n41         kb:containsOrganization \"White House\" ,\n42                                 \"Obama administration\" ,\n43                                 \"Iraqi government\" ;\n44         kb:containsPerson \"David Petraeus\" ,\n45                           \"John McCain\" ,\n46                           \"Hoshiyar Zebari\" ,\n47                           \"Barack Obama\" ,\n48                           \"George W. Bush\" ,\n49                           \"Carly Fiorina\" ;\n50         kb:containsIndustryTerm \"oil prices\" .\n51 \n52     <http://yahoo.com/10944/ts_nm/worldleaders_dc_1 />\n53         kb:containsCity \"WASHINGTON\" ;\n54         kb:containsCountry \"United States\" , \"Pakistan\" ,\n55                            \"Islamic Republic of Iran\" ;\n56         kb:containsState \"Maryland\" ;\n57         kb:containsOrganization \"University of Maryland\" ,\n58                                 \"United Nations\" ;\n59         kb:containsPerson \"Ban Ki-moon\" , \"Gordon Brown\" ,\n60                           \"Hu Jintao\" , \"George W. Bush\" ,\n61                           \"Pervez Musharraf\" ,\n62                           \"Vladimir Putin\" ,\n63                           \"Steven Kull\" ,\n64                           \"Mahmoud Ahmadinejad\" .\n65 \n66     <http://yahoo.com/10622/global_economy_dc_4 />\n67         kb:containsCity \"Sao Paulo\" , \"Kuala Lumpur\" ;\n68         kb:containsRegion \"Midwest\" ;\n69         kb:containsCountry \"United States\" , \"Britain\" ,\n70                            \"Saudi Arabia\" , \"Spain\" ,\n71                            \"Italy\" , India\" , \n72                            \"\"France\" , \"Canada\" ,\n73                            \"Russia\" , \"Germany\" , \"China\" ,\n74                            \"Japan\" , \"South Korea\" ;\n75         kb:containsOrganization \"Federal Reserve Bank\" ,\n76                                 \"European Union\" ,\n77                                 \"European Central Bank\" ,\n78                                 \"European Commission\" ;\n79         kb:containsPerson \"Lee Myung-bak\" , \"Rajat Nag\" ,\n80                           \"Luiz Inacio Lula da Silva\" ,\n81                           \"Jeffrey Lacker\" ;\n82         kb:containsCompany \"Development Bank Managing\" ,\n83                            \"Reuters\" ,\n84                            \"Richmond Federal Reserve Bank\" ;\n85         kb:containsIndustryTerm \"central bank\" , \"food\" ,\n86                                 \"energy costs\" ,\n87                                 \"finance ministers\" ,\n88                                 \"crude oil prices\" ,\n89                                 \"oil prices\" ,\n90                                 \"oil shock\" ,\n91                                 \"food prices\" ,\n92                                 \"Finance ministers\" ,\n93                                 \"Oil prices\" , \"oil\" .\n

\n\n\n\n

In the following examples, we will look at queries but not the results.\nWe will start with a simple SPARQL query for subjects (news article\nURLs) and objects (matching countries) with the value for the predicate\nequal to $containsCountry$:

\n\n

1     SELECT ?subject ?object\n2       WHERE {\n3         ?subject\n4         http://knowledgebooks.com/ontology#containsCountry>\n5         ?object .\n6       }\n

\n\n\n\n

Variables in queries start with a question mark character and can have\nany names. We can make this query easier and reduce the chance of\nmisspelling errors by using a namespace prefix:

\n\n

1     PREFIX kb:  <http://knowledgebooks.com/ontology#>\n2     SELECT ?subject ?object\n3       WHERE {\n4           ?subject kb:containsCountry ?object .\n5       }\n

\n\n\n\n

We could have filtered on any other predicate, for instance\n$containsPlace$. Here is another example using a match against a string\nliteral to find all articles exactly matching the text “Maryland.” The\nfollowing queries were copied from Java source files and were embedded\nas string literals so you will see quotation marks backslash escaped in\nthese examples. If you were entering these queries into a query form you\nwould not escape the quotation marks.

\n\n

1     PREFIX kb: <http://knowledgebooks.com/ontology#>\n2        SELECT ?subject\n3        WHERE { ?subject kb:containsState \\\"Maryland\\\" . }\n

\n\n\n\n

We can also match partial string literals against regular expressions:

\n\n

1     PREFIX kb:  \n2     SELECT ?subject ?object\n3        WHERE {\n4          ?subject\n5          kb:containsOrganization\n6          ?object FILTER regex(?object, \\\"University\\\") .\n7        }\n

\n\n\n\n

Prior to this last example query we only requested that the query return\nvalues for subject and predicate for triples that matched the query.\nHowever, we might want to return all triples whose subject (in this case\na news article URI) is in one of the matched triples. Note that there\nare two matching triples, each terminated with a period:

\n\n

 1     PREFIX kb: <http://knowledgebooks.com/ontology#>\n 2     SELECT ?subject ?a_predicate ?an_object\n 3        WHERE {\n 4         ?subject\n 5         kb:containsOrganization\n 6         ?object FILTER regex(?object, \\\"University\\\") .\n 7 \n 8         ?subject ?a_predicate ?an_object .\n 9        }\n10        DISTINCT\n11        ORDER BY ?a_predicate  ?an_object\n12        LIMIT 10\n13        OFFSET 5\n

\n\n\n\n

When WHERE clauses contain more than one triple pattern to match, this\nis equivalent to a Boolean “and” operation. The DISTINCT clause removes\nduplicate results. The ORDER BY clause sorts the output in alphabetical\norder: in this case first by predicate (containsCity, containsCountry,\netc.) and then by object. The LIMIT modifier limits the number of\nresults returned and the OFFSET modifier sets the number of matching\nresults to skip.

\n\n

We are finished with our quick tutorial on using the SELECT query form.\nThere are three other query forms that I am not covering in this\nchapter:

\n\n

CONSTRUCT – returns a new RDF graph of query results
ASK – returns Boolean true or false indicating if a query matches\nany triples
DESCRIBE – returns a new RDF graph containing matched resources

\n\n

SPARQL Query Client

\n\n

There are many available public RDF data sources. Some examples of publicly available SPARQL endpoints are:

\n\n

\nDBPedia which encodes the structured information in Wikipedia info boxes as RDF data. The DBPedia data is also available as a free download for hosting on your own SPARQL endpoint.
\nLinkDb Which contains human genome data
\nLibrary of Congress There is currently no SPARQL endpoint provided but subject matter Sparql data is available for download for use on your own SPARQL endpoint and there is a search faclity that can return RDF data results.
BBC Programs and Music
\nFactForge Combines RDF data from a many sources including DBPedia, Geonames, Wordnet, and Freebase. The SPARQL query page has many great examples to get you started. For example, look at the SPARQL for the query and results for “Show the distance from London of airports located at most 50 miles away from it.”

\n\n

There are many public SPARQL endpoints on the web but I need to warn you that they are not always available. The raw RDF data for most public endpoints is usually available so if you are building a system relying on a public data set you should consider hosting a copy of the data on one of your own servers. I usually use one of the following RDF repositories and SPARQL query interfaces: Star Dog, Virtuoso, and Sesame. That said, there are many other fine RDF repository server options, both commercial and open source. After reading this chapter, you may want to read a fairly recent blog article I wrote for importing and using the OpenCYC world knowledge base in to a local repository.

\n\n

In the next two sections you will learn how to make SPARQL queries against RDF data stores on the web, getting results both as JSON and parsed out to Java data values. A major theme in the next chapter is combining data in the “cloud” with local data.

\n\n

JSON Client

\n\n

 1 package com.markwatson.linkeddata;\n 2 \n 3 import org.apache.http.HttpResponse;\n 4 import org.apache.http.client.methods.HttpGet;\n 5 import org.apache.http.impl.client.DefaultHttpClient;\n 6 \n 7 import java.io.BufferedReader;\n 8 import java.io.InputStreamReader;\n 9 import java.net.URLEncoder;\n10 \n11 public class SparqlClientJson {\n12   public static String query(String endpoint_URL, String sparql)\n13                        throws Exception {\n14     StringBuffer sb = new StringBuffer();\n15     try {\n16       org.apache.http.client.HttpClient httpClient =\n17 \t    new DefaultHttpClient();\n18       String req = URLEncoder.encode(sparql, \"utf-8\");\n19       HttpGet getRequest =\n20           new HttpGet(endpoint_URL + \"?query=\" + req);\n21       getRequest.addHeader(\"accept\", \"application/json\");\n22       HttpResponse response = httpClient.execute(getRequest);\n23       if (response.getStatusLine().getStatusCode() != 200)\n24 \t     return \"Server error\";\n25 \n26       BufferedReader br =\n27           new BufferedReader(\n28               new InputStreamReader(\n29 \t\t\t\t   (response.getEntity().getContent())));\n30       String line;\n31       while ((line = br.readLine()) != null) {\n32         sb.append(line);\n33       }\n34       httpClient.getConnectionManager().shutdown();\n35     } catch (Exception e) {\n36       e.printStackTrace();\n37     }\n38     return sb.toString();\n39   }\n40 }\n

\n\n\n\n

We will be using the SPARQL query for finding people who were born in Arizona:

\n\n

1 PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n2         PREFIX dbpedia2: <http://dbpedia.org/property/>\n3         PREFIX dbpedia: <http://dbpedia.org/>\n4         SELECT ?name ?person WHERE {\n5              ?person dbpedia2:birthPlace <http://dbpedia.org/resource/Arizona> .\n6              ?person foaf:name ?name .\n7         }\n8         LIMIT 10\n

\n\n\n\n

You can run the unit tests for the DBPedia lookups and SPARQL queries using:

\n\n

1 mvn test -Dtest=DBPediaTest\n

\n\n\n\n

Here is a small bit of the output produced when running the unit test for this class with this example SPARQL query:

\n\n

{\n  \"head\": {\n    \"link\": [],\n    \"vars\": [\n      \"name\",\n      \"person\"\n    ]\n  },\n  \"results\": {\n    \"distinct\": false,\n    \"ordered\": true,\n    \"bindings\": [\n      {\n        \"name\": {\n          \"type\": \"literal\",\n          \"xml:lang\": \"en\",\n          \"value\": \"Ryan David Jahn\"\n        },\n        \"person\": {\n          \"type\": \"uri\",\n          \"value\": \"http://dbpedia.org/resource/Ryan_David_Jahn\"\n        }\n      },\n      ...\n\t]\n\t...\n  },\n }\n

\n\n\n\n

This example uses the DBPedia SPARQL endpoint. There is another public DBPedia service for looking up names and we will look at it in the next section.

\n\n

DBPedia Entity Lookup

\n\n

Later in this chapter I will show you what the raw DBPedia data looks like. For this section we will access DBPdedia through a public search API. The following two sub-sections show example code for getting the DBPedia lookup results as Java data and as JSON data.

\n\n

Java Data Client

\n\n

In the DBPedia Lookup service that we use in this section \nI only list a bit of the example program for this section. This example is fairly long and I refer you to the source code.

\n\n

 1 public class DBpediaLookupClient extends DefaultHandler {\n 2   public DBpediaLookupClient(String query) throws Exception {\n 3     this.query = query;\n 4     HttpClient client = new HttpClient();\n 5 \n 6     String query2 = query.replaceAll(\" \", \"+\");\n 7     HttpMethod method =\n 8         new GetMethod(\n 9           \"http://lookup.dbpedia.org/api/search.asmx/KeywordSearch?QueryString=\"\\\n10  +\n11           query2);\n12     try {\n13       client.executeMethod(method);\n14       InputStream ins = method.getResponseBodyAsStream();\n15       SAXParserFactory factory = SAXParserFactory.newInstance();\n16       SAXParser sax = factory.newSAXParser();\n17       sax.parse(ins, this);\n18     } catch (HttpException he) {\n19       System.err.println(\"Http error connecting to lookup.dbpedia.org\");\n20     } catch (IOException ioe) {\n21       System.err.println(\"Unable to connect to lookup.dbpedia.org\");\n22     }\n23     method.releaseConnection();\n24   }\n25 \n26   public void printResults(List<Map<String, String>> results) {\n27     for (Map<String, String> result : results) {\n28       System.out.println(\"\\nNext result:\\n\");\n29       for (String key : result.keySet()) {\n30         System.out.println(\"  \" + key + \"\\t:\\t\" + result.get(key));\n31       }\n32     }\n33   }\n34 \n35   //  ... XML parsing code not shown ...\n36   \n37   public List<Map<String, String>> variableBindings() {\n38     return variableBindings;\n39   }\n40 }\n

\n\n\n\n

The DBPedia Lookup service returns XML response data by default. The DBpediaLookupClient class is fairly simple. It encodes a query string, calls the web service, and parses the XML payload that is returned from the service.

\n\n

You can run the unit tests for the DBPedia lookups and SPARQL queries using:

\n\n

1 mvn test -Dtest=DBPediaTest\n

\n\n\n\n

Sample output for looking up “London” showing values for map key “Description”, “Label”, and “URI” in the fetched data looks like:

\n\n

1   Description\t:\tThe City of London was a United Kingdom Parliamentary constituen\\\n2 cy. It was a constituency of the House of Commons of the Parliament of England t\\\n3 hen of the Parliament of Great Britain from 1707 to 1800 and of the Parliament o\\\n4 f the United Kingdom from 1801 to 1950. The City of London was a United Kingdom \\\n5 Parliamentary constituency. It was a constituency of the House of Commons of the\\\n6  Parliament of England then of the Parliament of Great Britain from 1707 to 1800\\\n7  and of the Parliament of the United Kingdom from 1801 to 1950.\n8   Label\t:\tUnited Kingdom Parliamentary constituencies established in 1298\n9   URI\t:\thttp://dbpedia.org/resource/City_of_London_(UK_Parliament_constituency)\n

\n\n\n\n

If you run this example program you will see that the description text prints all on one line; it line wraps in the about example.

\n\n

This example returns a list of maps as a result where each list item is a map which each key is a variable name and each map value is the value for the key. In the next section we will look at similar code that instead returns JSON data.

\n\n

JSON Client

\n\n

The example in the section is simpler than the one in the last section because we just need to return the payload from the lookup web service as-is (i.e., as JSON data encoded in a string).

\n\n

 1 package com.markwatson.linkeddata;\n 2 \n 3 import org.apache.http.HttpResponse;\n 4 import org.apache.http.client.HttpClient;\n 5 import org.apache.http.client.methods.HttpGet;\n 6 import org.apache.http.impl.client.DefaultHttpClient;\n 7 \n 8 import java.io.BufferedReader;\n 9 import java.io.InputStreamReader;\n10 \n11 /**\n12  * Copyright 2015 Mark Watson.  Apache 2 license.\n13  */\n14 public class DBpediaLookupClientJson {\n15   public static String lookup(String query) {\n16     StringBuffer sb = new StringBuffer();\n17     try {\n18       HttpClient httpClient = new DefaultHttpClient();\n19       String query2 = query.replaceAll(\" \", \"+\");\n20       HttpGet getRequest =\n21           new HttpGet(\n22 \t\t    \"http://lookup.dbpedia.org/api/search.asmx/KeywordSearch?QueryString=\"\n23               + query2);\n24       getRequest.addHeader(\"accept\", \"application/json\");\n25       HttpResponse response = httpClient.execute(getRequest);\n26 \t  \n27       if (response.getStatusLine().getStatusCode() != 200) return \"Server error\";\n28 \n29       BufferedReader br =\n30           new BufferedReader(\n31               new InputStreamReader((response.getEntity().getContent())));\n32       String line;\n33       while ((line = br.readLine()) != null) {\n34         sb.append(line);\n35       }\n36       httpClient.getConnectionManager().shutdown();\n37     } catch (Exception e) {\n38       e.printStackTrace();\n39     }\n40     return sb.toString();\n41   }\n42 }\n

\n\n\n\n

You can run the unit tests for the DBPedia lookups and SPARQL queries using:

\n\n

1 mvn test -Dtest=DBPediaTest\n

\n\n\n\n

The following example for the lookup for “Bill Clinton” shows sample JSON (as a string) content that I “pretty printed” using IntelliJ for readability:

\n\n

{\n  \"results\": [\n    {\n      \"uri\": \"http://dbpedia.org/resource/Bill_Clinton\",\n      \"label\": \"Bill Clinton\",\n      \"description\": \"William Jefferson \\\"Bill\\\" Clinton (born William Jefferson\\\n Blythe III; August 19, 1946) is an American politician who served as the 42nd P\\\nresident of the United States from 1993 to 2001. Inaugurated at age 46, he was t\\\nhe third-youngest president. He took office at the end of the Cold War, and was \\\nthe first president of the baby boomer generation. Clinton has been described as\\\n a New Democrat. Many of his policies have been attributed to a centrist Third W\\\nay philosophy of governance.\",\n      \"refCount\": 7034,\n      \"classes\": [\n        {\n          \"uri\": \"http://dbpedia.org/ontology/OfficeHolder\",\n          \"label\": \"office holder\"\n        },\n        {\n          \"uri\": \"http://dbpedia.org/ontology/Person\",\n          \"label\": \"person\"\n        },\n        ...\n      ],\n      \"categories\": [\n        {\n          \"uri\": \"http://dbpedia.org/resource/Category:Governors_of_Arkansas\",\n          \"label\": \"Governors of Arkansas\"\n        },\n        {\n          \"uri\": \"http://dbpedia.org/resource/Category:American_memoirists\",\n          \"label\": \"American memoirists\"\n        },\n        {\n          \"uri\": \"http://dbpedia.org/resource/Category:Spouses_of_United_States_\\\nCabinet_members\",\n          \"label\": \"Spouses of United States Cabinet members\"\n        },\n\t\t...\n      ],\n      \"templates\": [],\n      \"redirects\": []\n    },\n

\n\n\n\n

Annotate Text with DBPedia Entity URIs

\n\n

I have used DBPdedia, the semantic web/linked data version of Wikipedia, in several projects. One useful use case is using DBPedia URIs as a unique identifier of a unique real world entity. For example, there are many people named Bill Clinton but usually this name refers to a President of the USA. It is useful to annotate people’s names, company names, etc. in text with the DBPedia URI for that entity. We will do this in a quick and dirty way (but hopefully still useful!) in this section and then solve the same problem in a different way in the next section.

\n\n

In this section we simply look for exact matches in text for the descriptions for DBPedia URIs. In the next section we tokenize the descriptions and also tokenize the text to match entity names with URIs. I will also show you some samples of the raw DBPedia RDF data in the next section.

\n\n

For one of my own projects I created mappings of entity names to DBPedia URIs for nine entity classes. For the example in this section I use three of my entity class mappings: people, countries, and companies. The data for these entity classes is stored in text files in the subdirectory dbpedia_entities. Later in this chapter I will show you what the raw DBPedia dump data looks like.

\n\n

The example program in this section modified input text adding DBPedia URIs after entities recognized in the text.

\n\n

The annotation code in the following example is simple but not very efficient. See the source code for comments discussing this that I left out of the following listing (edited to fit the page width):

\n\n

 1 package com.markwatson.linkeddata;\n 2 \n 3 import java.io.BufferedReader;\n 4 import java.io.FileInputStream;\n 5 import java.io.InputStream;\n 6 import java.io.InputStreamReader;\n 7 import java.nio.charset.Charset;\n 8 import java.util.HashMap;\n 9 import java.util.Map;\n10 \n11 public class AnnotateEntities {\n12 \n13   /**\n14    * @param text - input text to annotate with DBPedia URIs for entities\n15    * @return original text with embedded annotations\n16    */\n17   static public String annotate(String text) {\n18     String s = text.replaceAll(\"\\\\.\", \" !! \").replaceAll(\",\", \" ,\").\n19 \t                replaceAll(\";\", \" ;\").replaceAll(\"  \", \" \");\n20     for (String entity : companyMap.keySet())\n21       if (s.indexOf(entity) > -1) s = s.replaceAll(entity, entity +\n22 \t\t   \" (\" + companyMap.get(entity) + \")\");\n23     for (String entity : countryMap.keySet())\n24       if (s.indexOf(entity) > -1) s = s.replaceAll(entity, entity +\n25 \t\t   \" (\" + countryMap.get(entity) + \")\");\n26     for (String entity : peopleMap.keySet())\n27       if (s.indexOf(entity) > -1) s = s.replaceAll(entity, entity +\n28 \t\t   \" (\" + peopleMap.get(entity) + \")\");\n29     return s.replaceAll(\" !!\", \".\");\n30   }\n31 \n32   static private Map<String, String> companyMap = new HashMap<>();\n33   static private Map<String, String> countryMap = new HashMap<>();\n34   static private Map<String, String> peopleMap = new HashMap<>();\n35   static private void loadMap(Map<String, String> map, String entityFilePath) {\n36     try {\n37       InputStream fis = new FileInputStream(entityFilePath);\n38       InputStreamReader isr =\n39         new InputStreamReader(fis, Charset.forName(\"UTF-8\"));\n40       BufferedReader br = new BufferedReader(isr);\n41       String line = null;\n42       while ((line = br.readLine()) != null) {\n43         int index = line.indexOf('\\t');\n44         if (index > -1) {\n45           String name = line.substring(0, index);\n46           String uri = line.substring((index + 1));\n47           map.put(name, uri);\n48         }\n49       }\n50     } catch (Exception ex) {\n51       ex.printStackTrace();\n52     }\n53   }\n54   static {\n55     loadMap(companyMap, \"dbpedia_entities/CompanyNamesDbPedia.txt\");\n56     loadMap(countryMap, \"dbpedia_entities/CountryNamesDbPedia.txt\");\n57     loadMap(peopleMap, \"dbpedia_entities/PeopleDbPedia.txt\");\n58   }\n59 }\n

\n\n\n\n

If you run the unit tests for this code the output looks like:

\n\n

 1 > mvn test -Dtest=AnnotationTest\n 2 [INFO] Compiling 5 source files to /Users/markw/BITBUCKET/power-java/linked_data\\\n 3 /target/classes\n 4 \n 5 -------------------------------------------------------\n 6  T E S T S\n 7 -------------------------------------------------------\n 8 Running com.markwatson.linkeddata.AnnotationTest\n 9 \n10 Annotated string: Bill Clinton (<http://dbpedia.org/resource/Bill_Clinton>) went\\\n11  to a charity event in Thailand (<http://dbpedia.org/resource/Thailand>). \n12 \n13 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.251 sec\n14 \n15 Results :\n16 \n17 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0\n

\n\n\n\n

Resolving Named Entities in Text to Wikipedia URIs

\n\n

We will solve the same problem in this section as we looked at in the last section: resolving named entities with DBPedia (Wikipedia) URIs but we will use an alternative approach.

\n\n

I some of my work I find it useful to resolve named entities (e.g., people, organizations, products, locations, etc.) in text to unique DBPedia URIs. The DBPedia semantic web data, which we will also discuss in the next chapter, is derived from the “info boxes” in Wikipedia articles. I find DBPedia to be an incredibly useful knowledge source. When I worked at Google with the Knowledge Graph we were using a core of knowledge from Wikipedia/DBPedia and Freebase with a lot of other information sources. DBPedia is already (reasonably) clean data and is ready to be used as-is in your applications.

\n\n

The idea behind the example in this chapter is simple: the RDF data dumps from DBPedia can be processed to capture an entity name and the corresponding DBPedia entity URI.

\n\n

The raw DPBedia files can be downloaded from Resolving Named Entities (latest version when I wrote this chapter) and as an example, the SKOS categories dataset looks like (reformatted to be a litle easier to read):

\n\n

 1 <http://dbpedia.org/resource/Category:World_War_II>\n 2     <http://www.w3.org/2004/02/skos/core#broader>\n 3     <http://dbpedia.org/resource/Category:Conflicts_in_1945> .\n 4 \t\n 5 <http://dbpedia.org/resource/Category:World_War_II>\n 6     <http://www.w3.org/2004/02/skos/core#broader>\n 7 \t<http://dbpedia.org/resource/Category:Global_conflicts> .\n 8 \t\n 9 <http://dbpedia.org/resource/Category:World_War_II>\n10     <http://www.w3.org/2004/02/skos/core#broader>\n11 \t<http://dbpedia.org/resource/Category:Modern_Europe> .\n12 \t\n13 <http://dbpedia.org/resource/Category:World_War_II>\n14     <http://www.w3.org/2004/02/skos/core#broader>\n15 \t<http://dbpedia.org/resource/Category:The_World_Wars> .\n

\n\n\n\n

SKOS (Simple Knowledge Organization System) defines hierarchies of categories that are useful for categorizing real world entities, actions, etc. Cateorgies are then used for RDF versions of Wikipedia article titles, abstracts, geographic data, etc. Here is a small sample of RDF that maps subject URIs to text labels:

\n\n

 1 <http://dbpedia.org/resource/Abraham_Lincoln>\n 2     <http://www.w3.org/2000/01/rdf-schema#label>\n 3 \t\"Abraham Lincoln\"@en .\n 4 \n 5 <http://dbpedia.org/resource/Alexander_the_Great>\n 6     <http://www.w3.org/2000/01/rdf-schema#label>\n 7 \t\"Alexander the Great\"@en .\n 8 \n 9 <http://dbpedia.org/resource/Ballroom_dancing>\n10     <http://www.w3.org/2000/01/rdf-schema#label>\n11 \t\"Ballroom dancing\"@en .\n

\n\n\n\n

As a final example of what the raw DBPedia RDF dump data looks like, here are some RDF statements that define abstracts for articles (text descriptions are shortened in the following listing to fit on one line):

\n\n

1 <http://dbpedia.org/resource/An_American_in_Paris>\n2    <http://www.w3.org/2000/01/rdf-schema#comment>\n3    \"An American in Paris is a jazz-influenced symphonic poem by ....\"@en .\n4 \n5 <http://dbpedia.org/resource/Alberta>\n6     <http://www.w3.org/2000/01/rdf-schema#comment>\n7 \t\"Alberta (is a western province of Canada. With a population of 3,645,257 ....\"\\\n8 @en .\n

\n\n\n\n

I have already pulled entity names and URIs for nine classes of entities and this data is available in the github repo for this book in the directory inked_data/dbpedia_entities.

\n\n

Java Named Entity Recognition Library

\n\n

We already saw the use of Named Entity Resolution (NER) in the chapter using OpenNLP. Here we do something different. For my own projects I have data that I have processed that maps entity names to DBPedia URIs and Clojure code to use this data. For the purposes of this book, I am releasing my data under the Apache 2 license so you can use it in your projects and I converted my Clojure NER library to Java.

\n\n

The following figure shows the project for this chapter in IntelliJ:

\n\n\n

The following code example uses my data for mapping nine classes of entities to DBPedia URIs:

\n\n

 1 package com.markwatson.ner_dbpedia;\n 2 \n 3 import java.io.IOException;\n 4 import java.nio.file.Files;\n 5 import java.nio.file.Paths;\n 6 import java.util.HashMap;\n 7 import java.util.Map;\n 8 import java.util.stream.Stream;\n 9 \n10 /**\n11  * Created by markw on 10/12/15.\n12  */\n13 public class NerMaps {\n14   private static Map<String, String> textFileToMap(String nerFileName) {\n15     Map<String, String> ret = new HashMap<String, String>();\n16     try {\n17       Stream<String> lines =\n18           Files.lines(Paths.get(\"dbpedia_as_text\", nerFileName));\n19       lines.forEach(line -> {\n20         String[] tokens = line.split(\"\\t\");\n21         if (tokens.length > 1) {\n22           ret.put(tokens[0], tokens[1]);\n23         }\n24       });\n25       lines.close();\n26     } catch (Exception ex) {\n27       ex.printStackTrace();\n28     }\n29     return ret;\n30   }\n31 \n32   static public final Map<String, String> broadcastNetworks =\n33     textFileToMap(\"BroadcastNetworkNamesDbPedia.txt\");\n34   static public final Map<String, String> cityNames =\n35     textFileToMap(\"CityNamesDbpedia.txt\");\n36   static public final Map<String, String> companyames =\n37     textFileToMap(\"CompanyNamesDbPedia.txt\");\n38   static public final Map<String, String> countryNames =\n39     textFileToMap(\"CountryNamesDbpedia.txt\");\n40   static public final Map<String, String> musicGroupNames =\n41     textFileToMap(\"MusicGroupNamesDbPedia.txt\");\n42   static public final Map<String, String> personNames =\n43     textFileToMap(\"PeopleDbPedia.txt\");\n44   static public final Map<String, String> politicalPartyNames =\n45     textFileToMap(\"PoliticalPartyNamesDbPedia.txt\");\n46   static public final Map<String, String> tradeUnionNames =\n47     textFileToMap(\"TradeUnionNamesDbPedia.txt\");\n48   static public final Map<String, String> universityNames =\n49     textFileToMap(\"UniversityNamesDbPedia.txt\");\n50 \n51   public static void main(String[] args) throws IOException {\n52     new NerMaps().textFileToMap(\"CityNamesDbpedia.txt\");\n53   }\n54 }\n

\n\n\n\n

The class TextToDbpediaUris uses the nine NER types defined in class NerMaps. In the following listing the code for all by the NER class Broadcast News Networks is not shown for brevity:

\n\n

 1 package com.markwatson.ner_dbpedia;\n 2 \n 3 public class TextToDbpediaUris {\n 4   private TextToDbpediaUris() {\n 5   }\n 6 \n 7   public TextToDbpediaUris(String text) {\n 8     String[] tokens = tokenize(text + \" . . .\");\n 9     String s = \"\";\n10     for (int i = 0, size = tokens.length - 2; i < size; i++) {\n11       String n2gram = tokens[i] + \" \" + tokens[i + 1];\n12       String n3gram = n2gram + \" \" + tokens[i + 2];\n13       // check for 3grams:\n14       if ((s = NerMaps.broadcastNetworks.get(n3gram)) != null) {\n15         log(\"broadcastNetwork\", i, i + 2, n3gram, s);\n16         i += 2;\n17         continue;\n18       }\n19 \n20       // check for 2grams:\n21       if ((s = NerMaps.broadcastNetworks.get(n2gram)) != null) {\n22         log(\"broadCastNetwork\", i, i + 1, n2gram, s);\n23         i += 1;\n24         continue;\n25       }\n26 \n27       // check for 1grams:\n28       if ((s = NerMaps.broadcastNetworks.get(tokens[i])) != null) {\n29         log(\"broadCastNetwork\", i, i + 1, tokens[i], s);\n30         continue;\n31       }\n32     }\n33   }\n34 \n35   /** In your applications you will want to subclass TextToDbpediaUris\n36    * and override <strong>log</strong> to do something with the\n37    * DBPedia entities that you have identified.\n38    * @param nerType\n39    * @param index1\n40    * @param index2\n41    * @param ngram\n42    * @param uri\n43    */\n44   void log(String nerType, int index1, int index2, String ngram, String uri) {\n45     System.out.println(nerType + \"\\t\" + index1 + \"\\t\" + index2 + \"\\t\" + ngram + \\\n46 \"\\t\" + uri);\n47   }\n48 \n49   private String[] tokenize(String s) {\n50     return s.replaceAll(\"\\\\.\", \" \\\\. \").replaceAll(\",\", \" , \")\n51         .replaceAll(\"\\\\?\", \" ? \").replaceAll(\"\\n\", \" \").replaceAll(\";\", \" ; \").s\\\n52 plit(\" \");\n53   }\n54 }\n

\n\n\n\n

Here is an example showing how to use this code and the output:

\n\n

    String s = \"PTL Satellite Network covered President Bill Clinton going to Gu\\\natemala and visiting the Cocoa Cola Company.\";\n    TextToDbpediaUris test = new TextToDbpediaUris(s);\n

\n\n\n\n

broadcastNetwork 0 2 PTL Satellite Network\t<http://dbpedia.org/resource/PTL_Sate\\\nllite_Network>\nperson   5  6 Bill Clinton <http://dbpedia.org/resource/Bill_Clinton>\ncountry  9 10 Guatemala    <http://dbpedia.org/resource/Guatemala>\ncompany 13 14 Coca Cola    <http://dbpedia.org/resource/Coca-Cola>\n

\n\n\n\n

The second and third values are word indices (first word in sequence and last word in sequence from the input tokenized text).

\n\n

There are many types of applications that can benefit from using “real world” information from sources like DBPedia.

\n\n

Combining Data from Public and Private Sources

\n\n

A powerful idea is combining public and private linked data; for example, combining the vast real world knowledge stored in DBPedia with an RDF data store containing private data specific to your company. There are two ways to do this:

\n\n

Use SPARQL queries to join data from multiple SPARQL endpoints
Assuming you have an RDF repository of private data, import external sources like DBPedia into your local data store

\n\n

It is beyond the scope of my coverage of SPARQL but data can be joined from multiple endpoints using the SERVICE keyword in a SPARQL WHERE clause. In a similar way, public sources like DBPedia can be imported into your local RDF repository: this both makes queries more efficient and makes your system more robust in the face of failures of third party services.

\n\n

One other data source that can be very useful to integrate into your applications is web search. I decided not to cover Java examples for calling public web search APIs in this book but I will refer you to my project on github that wraps the Microsoft Bing Search API. I mention this as an example pattern of using local data (notes and stored PDF documents) with public data APIs.

\n\n

We will talk more about combining different data sources in the next chapter.

\n\n

Wrap Up for Linked Data

\n\n

The semantic web and linked data are large topics and my goal in this chapter was to provide an easy to follow introduction to hopefully interest you in the subject and motivate you for further study. If you have not seen graph data before then RDF likely seems a little strange to you. There are more general graph databases like the commercial Neo4j commercial product (a free community edition is available) which provides an alternative to RDF data stores. If you would like to experiment with running your own RDF data store and SPARQL endpoint there are many fine open source and commercial products to choose from. If you read my web blog you have seen my experiments using the free edition of the Stardog RDF data store.

\n\n\n

Java Strategies for Working with Cloud Data: Knowledge Management-Lite

\n\n

Knowledge Management (KM) is the collection, organization and maintenance of knowledge. The last chapter introduced you to techniques for information architecture and development using linked data and semantic web technologies and is paired with this chapter so please read these two chapters in order.

\n\n

KM is a vast subject but I am interested in exploring one particular idea in this chapter: strategies for combining information stored in local data sources (relational and NoSQL data stores) with cloud data sources (e.g., Google Drive, Microsoft OneDrive, and Dropbox) to produce fused information that in turn may provide actionable knowledge. Think about your workflow: do you use Google Docs and Calendars? Microsoft Office 365 documents in the Azure cloud? I use both companies cloud services in my work. Sometimes cloud data just exists to backup local data on my laptops but usually I use cloud data for storing:

\n\n

very large data sets
data that needs to be available for a wide range of devices
data that is valuable and needs to be replicated

\n\n

I have been following the field of KM for many years but in the last few years I have been more taken with the idea of fusing together public information sources with private data. The flexibility of hybrid private data storage in the cloud and locally stored is a powerful pattern:

\n\n\n

In this chapter we use data sources of documents in the cloud and local relational databases.

\n\n

When I first planned on writing this chapter the example program that I wanted to write would access a user’s data in real time from either Google’s or Microsoft’s cloud services. I decided that the complexity of implementing OAuth2 to access Google and Microsoft services was too much for a book example because of the need for a custom web app to support an example application authenticating with these services.

\n\n

What I decided to do was export (using Google Takeout) a small set of test data from my Google account for my calendar, Google Drive, and GMail and use a sanitized version of these data for examples programs in this chapter. Google Drive documents export as Microsoft document formats (Microsoft Word docx files, Excel xlsx spreadsheet files) and in standard calendar ics format files and email in the standard mbox format. In other words, this example could also be applicable for data stored in Microsoft Office 365. While using canned data does not make a totally satisfactory demo it does allow me to concentrate on techniques for accessing the content of popular document types while ignoring the complexities of authenticating users to access their data on Google and Microsoft cloud services.

\n\n

If you want to experiment with the example programs in this chapter with your own data you can use www.google.com/settings/takeout/ to download your own data and replace my sanitized Google Takeout data - doing this will make the example programs more interesting and hopefully motivate you to experiment further with them.

\n\n

A real system using the techniques that we will cover could be implemented as a web app running on either Google Cloud Services or Microsoft Azure that would authenticate a user with their Google or Microsoft accounts, and have live access to the user’s cloud data.

\n\n

We will be using open source Java libraires to access the data in Microsoft Excel and Office formats (note that Google exports Google docs in these formats as does Microsoft’s Office 365).

\n\n

One excellant tool for building KM systems that I am not covering is web search. Automatically accessing a web-wide search index to provide system users with more information to make decisions is something to consider when designong KM and decision support systems. I decided not to cover Java examples for calling public web search APIs in this book but you can use my project on github that wraps the Microsoft Bing Search API. I once wrote a service using search APIs to annotate notes and stored documents. Using DBPedia (as we did in the last chapter) and Bing search was easy and allowed me to annotate text (with hyperlinks to web information sources) in near real time.

\n\n

Motivation for Knowledge Management

\n\n

Knowledge workers are different than previous kinds of workers who were trained to do a specific job repetitively. Knowledge workers tend to combine information from different sources to make decisions.

\n\n

When workers leave, if their knowledge is not codified then the organization loses value. This situation will get worse in the future as baby boomers retire in larger numbers. So, the broad goals of KM are to capture employee knowledge, to facilitate sharing knowledge and information between workers company wide, and provide tools to increase workers’ velocity in making good decisions that are based on solid information and previous organizational knowledge.

\n\n

For the purposes of this chapter we will discuss KM from the viewpoint of making maximum use of all available data sources, including shared cloud-based sources of information. We will do this through the implementation of Java code examples for accessing common document formats.

\n\n

Since KM is a hugely broad topic I am going to concentrate on a few relatively simple examples that will give you ideas for both capturing and using knowledge and in some cases also provide useful code from the examples in this book that you can use:

\n\n

Capture your email in a relational database and assign categories using some of the techniques we learned in the NLP and machine learning chapters.
Store web page URIs and content snippets for future use.
Java clients to process data in Google Drive, specifically reading Microsoft file formats that Google uses for exporting Google docs.
Annotate text in a document store by resolving entity names to unique DBPedia URIs.
Use Postgres as a local database to store discovered data that is discovered by the example software in this chapter.

\n\n

Microsoft Office 365, which I use, also has APIs for integrating your own programs into an Office 365 work flow. I decided to use Google Drive in this chapter because it is free to use but I also encourage you to experiment with the Office 365 APIs. You can take free Office 365 API classes offered online edX.org. We will use Google Drive as a data source for the example code developed in this chapter.

\n\n

As I mentioned earlier in this chapter in order to build a real system based on the ideas and example code in this chapter you will need to write a web applicaton that can authenticate users of your system to access data in Google Drive and/or in Microsoft OneDrive. In addition to real time access to shared data in the cloud, another important part of organizational KM systems that we are not discussing in this chapter is providing user and access roles. In the examples in this chapter we will ignore these organizational requirements and concentrate on the process of automatic discovery of useful metadata that helps to organize shared data.

\n\n

There are many KM products that cater to organizations or individuals. Depending on your requirements you might want to mix custom software with off the shelf software tools. Another important aspect of KM is the concept of master data which is creating a uniform data model and storage strategy for diverse (and often legacy) databases. This is partly what I worked on at Google. Even for the three simple examples we will develop in this chapter there are interesting issues involving master data that I will return to later.

\n\n

Most KM systems deal with internal data that is stored on local servers. While I do provide some very simple examples of using the Postgres relational database later in this chapter, I wanted to get you thinking about using multiple data sources and concentrating on data stored in commercial cloud services makes sense.

\n\n

Using Google Drive Cloud Takeout Service

\n\n

As I mentioned in the introduction to this chapter, while it is fairly easy to write a web app for Microsoft Azure that uses the Office 365 APIs an a web app for AppEngine (or Google Cloud) that uses the Google Drive data access APIs, building a web app with the auth2 support required to use these APIs is outside of the scope for an example program. So, I am cheating a little. Google has a “Google Takeout” service that everyone who uses Google services should periodically use to save all of their data from Google services.

\n\n

In this section and contained sub-sections we will use a “Google Takeout” dump as the data used by the example programs. The following subsections contain utilities for reading and parsing:

\n\n

Microsoft Excel which is what Google Takeout exports Google Drive spreadsheets as.
Microsoft Word docx files which is the format that Google Takeout uses to export Google Drive word processing documents.
iCal calendar files which is a standard format that Google Calendar data is exported as.
Email mbox files which is the format that GMail is exported as.

\n\n

These are all widely used file formats so I hope the utilities developed in the next four sub-sections will be generally useful to you. The following figure shows a screen grab of the IntelliJ Community Edition project browser showing just the data files that I exported using Google Takeout. I “sanitized” these files by changing names, email addresses, etc. to use for the example programs in this chapter.

\n\n\n

\"Sample — Sample Google Takeout files for this chapter

\n\n\n

We will be using the Apache Tika and iCal4j libraires to implement the code in the next three sub-sections. The following figure shows the IntelliJ source and test files used in this chapter:

\n\n\n

\"Java — Java source and test files for this chapter

\n\n\n

Processing Microsoft Excel Spreadsheet xlsx and Microsoft Word docx Files

\n\n

We will be using the Tika library that parses all sheets (i.e., separate pages) in Excel spreadsheet files. The output is a nicely formatted text file that is new line and tab character delimited. The following utility class parses this text output for more convenient use by parsing the plain text representation into Java data:

\n\n

 1 package com.markwatson.km;\n 2 \n 3 import java.util.*;\n 4 import java.util.regex.Pattern;\n 5 \n 6 public class ExcelData {\n 7   public List<List<List<String>>> sheetsAndRows = new ArrayList<>();\n 8   private List<List<String>> currentSheet = null;\n 9 \n10   public ExcelData(String contents) {\n11     Pattern.compile(\"\\n\")\n12         .splitAsStream(contents)\n13         .forEach((String line) -> handleContentsStream(line));\n14     if (currentSheet != null) sheetsAndRows.add(currentSheet); \n15   }\n16 \n17   private void handleContentsStream(String nextLine) {\n18     if (nextLine.startsWith(\"Sheet\")) {\n19       if (currentSheet != null) sheetsAndRows.add(currentSheet);\n20       currentSheet = new ArrayList<>();\n21     } else {\n22       String[] columns = nextLine.substring(1).split(\"\\t\");\n23       currentSheet.add(Arrays.asList(columns));\n24     }\n25   }\n26 \n27   public String toString() {\n28     StringBuilder sb = new StringBuilder(\"<ExcelData # sheets: \" +\n29               sheetsAndRows.size() + \"\\n\");\n30     for (List<List<String>> sheet : sheetsAndRows) {\n31       sb.append(\"  <Sheet\\n\");\n32       for (List<String> row : sheet) {\n33         sb.append(\"    <row \" + row + \">\\n\");\n34       }\n35       sb.append(\"  >\\n>\\n\");\n36     }\n37     return sb.toString();\n38   }\n39 }\n

\n\n\n\n

When a new instance of the class ExcelData is created you can access the individual sheets and the rows in each sheet from the public data in sheetsAndRows.

\n\n

The first index of sheetsAndRows provides access to individual sheets, the second index provided access to the rows in a selected sheet, and the third (inner) index provides access to the individual columns in a selected row. The public utility method toString provides an example for accessing this data. The class ExcellData is used in the example code in the next listing.

\n\n

The following utility class PoiMicrosoftFileReader has methods for processing Excel and Word files:

\n\n

 1 package com.markwatson.km;\n 2 \n 3 import java.io.*;\n 4 import org.apache.poi.openxml4j.exceptions.InvalidFormatException;\n 5 import org.apache.tika.Tika;\n 6 import org.apache.tika.exception.TikaException;\n 7 import org.apache.tika.metadata.Metadata;\n 8 import org.apache.tika.parser.ParseContext;\n 9 import org.apache.tika.parser.microsoft.ooxml.OOXMLParser;\n10 import org.apache.tika.sax.BodyContentHandler;\n11 import org.apache.xmlbeans.XmlException;\n12 import org.xml.sax.SAXException;\n13 \n14 public class PoiMicrosoftFileReader {\n15 \n16   static private boolean DEBUG_PRINT_META_DATA = false;\n17 \n18   public static String DocxToText(String docxFilePath)\n19       throws IOException, InvalidFormatException, XmlException,\n20              TikaException {\n21     String ret = \"\";\n22     FileInputStream fis = new FileInputStream(docxFilePath);\n23     Tika tika = new Tika();\n24     ret = tika.parseToString(fis);\n25     fis.close();\n26     return ret;\n27   }\n28 \n29   public static ExcelData readXlsx(String xlsxFilePath)\n30       throws IOException, InvalidFormatException, XmlException,\n31              TikaException, SAXException {\n32     BodyContentHandler bcHandler = new BodyContentHandler();\n33     Metadata metadata = new Metadata();\n34     FileInputStream inputStream = new FileInputStream(new File(xlsxFilePath));\n35     ParseContext pcontext = new ParseContext();\n36     OOXMLParser parser = new OOXMLParser();\n37     parser.parse(inputStream, bcHandler, metadata, pcontext);\n38     if (DEBUG_PRINT_META_DATA) {\n39       System.err.println(\"Metadata:\");\n40       for (String name : metadata.names())\n41         System.out.println(name + \"\\t:\\t\" + metadata.get(name));\n42     }\n43     ExcelData spreedsheet = new ExcelData(bcHandler.toString());\n44     return spreedsheet;\n45   }\n46 }\n

\n\n\n\n

The static public method DocxToText transforms a file path specifying a Microsoft Word docx file to the plain text in that file. The public static method readXlsx transforms a file path to an Excel xlsx file to an instance of the class ExcelData that we looked at earlier.

\n\n

The following output was generated from running the test programs GoogleDriveTest.testDocxToText and GoogleDriveTest.testExcelReader. I am using jUnit tests as a convenient way to provide examples of calling APIs in the example code. Here is the test code and its output:

\n\n

 1   String contents =\n 2     PoiMicrosoftFileReader.DocxToText(\n 3 \t      \"GoogleTakeout/Drive/booktest/Test document 1.docx\");\n 4   System.err.println(\"Contents of test Word docx file:\\n\\n\" +\n 5                      contents + \"\\n\");\n 6 \n 7   ExcelData spreadsheet =\n 8     PoiMicrosoftFileReader.readXlsx(\n 9 \t\t\"GoogleTakeout/Drive/booktest/Test spreadsheet 1.xlsx\");\n10   System.err.println(\"Contents of test spreadsheet:\\n\\n\" +\n11                      spreadsheet);\n

\n\n\n\n

 1 Contents of test Word docx file:\n 2 \n 3 This is a test document for Power Java book.\n 4 \n 5 \n 6 Contents of test spreadsheet:\n 7 \n 8 <ExcelData # sheets: 1\n 9   <Sheet\n10     <row [Name, Age]>\n11     <row [Brady, 12]>\n12     <row [Calvin, 17]>\n13   >\n14 >\n

\n\n\n\n

Processing iCal Calendar Files

\n\n

The iCal format is the standard way that Google, Apple, Microsoft, and other vendors export calendar data. The following code uses the net.fortuna.ical4j library that provides a rich API for dealing with iCal formatted files. In the follow example we are using a small part of this API.

\n\n

 1 package com.markwatson.km;\n 2 \n 3 // refenerence: https://github.com/ical4j/ical4j\n 4 \n 5 import net.fortuna.ical4j.data.CalendarBuilder;\n 6 import net.fortuna.ical4j.data.ParserException;\n 7 import net.fortuna.ical4j.model.Component;\n 8 import net.fortuna.ical4j.model.Property;\n 9 \n10 import java.io.FileInputStream;\n11 import java.io.IOException;\n12 import java.util.*;\n13 \n14 public class ReadCalendarFiles {\n15 \n16   public ReadCalendarFiles(String filePath)\n17              throws IOException, ParserException {\n18     Map<String, String> calendarEntry = null;\n19     FileInputStream fin = new FileInputStream(filePath);\n20     CalendarBuilder builder = new CalendarBuilder();\n21     net.fortuna.ical4j.model.Calendar calendar = builder.build(fin);\n22     for (Iterator i = calendar.getComponents().iterator();\n23          i.hasNext(); ) {\n24       Component component = (Component) i.next();\n25       if (component.getName().equalsIgnoreCase(\"VEVENT\")) {\n26         calendarEntry = new HashMap<>();\n27         for (Iterator j = component.getProperties().iterator();\n28              j.hasNext(); ) {\n29           net.fortuna.ical4j.model.Property property =\n30              (Property) j.next();\n31           calendarEntry.put(property.getName(),\n32                             property.getValue());\n33         }\n34         calendarEntries.add(calendarEntry);\n35       }\n36     }\n37   }\n38 \n39   public int size() { return calendarEntries.size(); }\n40   public Map<String, String> getCalendarEntry(int index) {\n41     return calendarEntries.get(index);\n42   }\n43 \n44   /**\n45    * List of calendar entries where each entry\n46    *  is a Map<String, String>\n47    *\n48    */\n49   private List<Map<String, String>> calendarEntries =\n50        new ArrayList<>();\n51 }\n

\n\n\n\n

The following output was generated from running the test program GoogleDriveTest.testCalendarFileReader. Here is the test code and its output:

\n\n

 1   ReadCalendarFiles calendar =\n 2     new ReadCalendarFiles(\"GoogleTakeout/Calendar/Mark Watson.ics\");\n 3   System.err.println(\"\\n\\nTest iCal calendar file has \" +\n 4                      calendar.size() + \" calendar entries.\");\n 5   for (int i=0, size=calendar.size(); i<size; i++) {\n 6     Map<String,String> entry = calendar.getCalendarEntry(i);\n 7     System.err.println(\"\\n\\tNext calendar entry:\");\n 8     System.err.println(\"\\n\\t\\tDTSAMP:\\t\" + entry.getOrDefault(\"DTSTAMP\", \"\"));\n 9     System.err.println(\"\\n\\t\\tSUMMARY:\\t\" + entry.getOrDefault(\"SUMMARY\", \"\"));\n10     System.err.println(\"\\n\\t\\tDESCRIPTION:\\t\" +\n11                        entry.getOrDefault(\"DESCRIPTION\", \"\"));\n12   }\n

\n\n\n\n

 1 Test iCal calendar file has 11 calendar entries.\n 2 \n 3 \tNext calendar entry:\n 4 \n 5 \t\tDTSAMP: 20150917T234033Z\n 6 \n 7 \t\tSUMMARY: Free Foobar Test  Meeting\n 8 \n 9 \t\tDESCRIPTION: Bob Smith has invited you to join a meeting\n10 \n11  ... ...\n

\n\n\n\n

The utility method toString for the class ReadCalendarFiles prints for each entry three attributes: a date stamp, a summary, and a description.

\n\n

Processing Email mbox Files

\n\n

In this section we use the Apace Tika library for parsing email mbox formatted files. There is a commented out printout on line 36 and you might want to uncomment this and look at the full message text to see all available meta data for each email as it is processed. You might want to use more of the meta data in your applications than I am using in the example code.

\n\n

  1 package com.markwatson.km;\n  2 \n  3 // reference: https://tika.apache.org/\n  4 \n  5 import org.apache.poi.openxml4j.exceptions.InvalidFormatException;\n  6 import org.apache.tika.exception.TikaException;\n  7 import org.apache.tika.metadata.Metadata;\n  8 import org.apache.tika.parser.ParseContext;\n  9 import org.apache.tika.parser.Parser;\n 10 import org.apache.tika.parser.mbox.MboxParser;\n 11 import org.apache.tika.parser.microsoft.ooxml.OOXMLParser;\n 12 import org.apache.tika.sax.BodyContentHandler;\n 13 import org.apache.xmlbeans.XmlException;\n 14 import org.xml.sax.SAXException;\n 15 \n 16 import java.io.*;\n 17 import java.nio.charset.StandardCharsets;\n 18 import java.util.ArrayList;\n 19 import java.util.List;\n 20 \n 21 /**\n 22  * Created by markw on 9/17/15.\n 23  */\n 24 public class ReadMbox {\n 25 \n 26   static private boolean DEBUG_PRINT_META_DATA = true;\n 27 \n 28   private String getTextContents(InputStream stream) {\n 29     BufferedReader br = new BufferedReader(new InputStreamReader(stream));\n 30     StringBuffer sb = new StringBuffer();\n 31     String subject = \"\";\n 32     String from = \"\";\n 33     String to = \"\";\n 34     try {\n 35       String line = null;\n 36       boolean inText = false;\n 37       while ((line = br.readLine()) != null) {\n 38         //System.err.println(\"-- line: \" + line);\n 39         if (!inText) {\n 40           if (line.startsWith(\"Subject:\")) subject = line.substring(9);\n 41           if (line.startsWith(\"To:\"))      to = line.substring(4);\n 42           if (line.startsWith(\"From:\"))    from = line.substring(6);\n 43         }\n 44         if (line.startsWith(\"Content-Type: text/plain;\")) {\n 45           inText = true;\n 46           br.readLine();\n 47         } else if (inText) {\n 48           if (line.startsWith(\"-----\")) break;\n 49           if (line.startsWith(\"--_---\")) break;\n 50           if (line.startsWith(\"Content-Type: text/html;\")) break;\n 51           sb.append(line + \"\\n\");\n 52         }\n 53       }\n 54     } catch (Exception ex) {\n 55       System.err.println(\"ERROR: \" + ex);\n 56     }\n 57     return \"To: \" + to + \"\\nFrom: \" + from + \"\\nSubject: \" +\n 58            subject + \"\\n\" + sb.toString();\n 59   }\n 60 \n 61   public ReadMbox(String filePath)\n 62       throws IOException, InvalidFormatException, XmlException,\n 63              TikaException, SAXException {\n 64 \n 65     FileInputStream inputStream = new FileInputStream(new File(filePath));\n 66 \n 67     BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));\n 68 \n 69     String line = null;\n 70     StringBuffer sb = new StringBuffer();\n 71     try {\n 72       // the outer loop splits each email to a separate string that we will use \\\n 73 Tika to parse:\n 74       while ((line = br.readLine()) != null) {\n 75         if (line.startsWith(\"From \")) {\n 76           if (sb.length() > 0) { // process all but the last email\n 77             String content = sb.toString();\n 78             InputStream stream =\n 79               new ByteArrayInputStream(content.getBytes(StandardCharsets.UTF_8));\n 80             emails.add(getTextContents(stream));\n 81             sb = new StringBuffer();\n 82           }\n 83         }\n 84         sb.append(line + \"\\n\");\n 85       }\n 86       if (sb.length() > 0) { // process the last email\n 87         String content = sb.toString();\n 88         InputStream stream =\n 89           new ByteArrayInputStream(content.getBytes(StandardCharsets.UTF_8));\n 90         emails.add(getTextContents(stream));\n 91       }\n 92 \n 93       br.close();\n 94     } catch (Exception ex) {\n 95       System.err.println(\"ERROR: \" + ex);\n 96     }\n 97   }\n 98 \n 99   public int size() {\n100     return emails.size();\n101   }\n102 \n103   public String getEmail(int index) {\n104     return emails.get(index);\n105   }\n106 \n107   public List<String> emails = new ArrayList<>();\n108 }\n

\n\n\n\n

The following output was generated from running the test program GoogleDriveTest.testMboxFileReader. Here is the test code and its output (edited for brevity and clarity):

\n\n

1   ReadMbox mbox = new ReadMbox(\"GoogleTakeout/Mail/bookexample.mbox\");\n2   System.err.println(\"\\nMBOX size = \" + mbox.size());\n3   for (int i=0, size=mbox.size(); i<size; i++) {\n4     System.err.println(\"\\n* * next email:\\n\\n\" + mbox.getEmail(i) + \"\\n\");\n5   }\n

\n\n\n\n

 1 MBOX size = 3\n 2 \n 3 * * next email:\n 4 \n 5 To: <mark@markwatson.com>\n 6 From: Ruby Weekly= <rw@peterc.org>\n 7 Subject: This week's Ruby news issue\n 8 \n 9 This week's Ruby and Rails news\n10 \n11 Ruby Weekly - A Weekly Ruby Newsletter\n12 Issue 264 - September 17=2C 2015\n13 \n14  ...  ...\n

\n\n\n\n

You now can parse and use information from exported Microsoft Office documents. As I mentioned before, when you export Google documents this process also generates documents in the Microsoft formats.

\n\n

Using Postgres as a Local Document Store with Text Search

\n\n

We will now develop a simple set of utiltity classes for easily string structured data and “less structured” JSON data in a Postgres database. We will also provide support for text search in database table columns containing text data.

\n\n

I use many different data stores in my work: Amazon S3 and DynamoDB, Google Big Table, Hadoop File System, MongoDB, CouchDB, Cassandra, etc. Lots of good options for different types of projects!

\n\n

That said, Postgres is my “Swiss Army knife” data store offering a rock solid relational database, full text indexing, and native support for schema less native JSON documents. I this short section I am going to cover JDBC access to Postgres and cover the common use cases that you might need if you also want to use Postgres as your “Swiss Army knife” data store.

\n\n

The following listing shows these utilities, encapsulated in the Java class PostgresUtilities. In line 16 we set the server name, Postgres account anme, and password for connecting to a database. Here the server name is “localhost” since I am running both the example program and the postgres server on my laptop. I pass the connection parameters in the connection URI on line 14 but the commented out lines 25 and 26 show you how to set connection properties in your code if you prefer that.

\n\n

  1 package com.markwatson.km;\n  2 \n  3 import java.sql.*;\n  4 import java.util.ArrayList;\n  5 import java.util.HashMap;\n  6 import java.util.List;\n  7 import java.util.Properties;\n  8 \n  9 /**\n 10  * Created by Mark on 6/5/2015.\n 11  */\n 12 public class PostgresUtilities {\n 13   private static String defaultConnectionURL =\n 14        \"jdbc:postgresql://localhost:5432/kmdexample?user=markw&password=\";\n 15   private Connection conn = null;\n 16 \n 17   public PostgresUtilities() throws ClassNotFoundException, SQLException {\n 18     this(defaultConnectionURL);\n 19   }\n 20 \n 21   public PostgresUtilities(String connectionURL)\n 22          throws ClassNotFoundException, SQLException {\n 23     Class.forName(\"org.postgresql.Driver\");\n 24     Properties props = new Properties();\n 25     //props.setProperty(\"user\",\"a username could go here\");\n 26     //props.setProperty(\"password\",\"a passwordc could go here\");\n 27     conn = DriverManager.getConnection(connectionURL, props);\n 28   }\n 29 \n 30   /**\n 31    * doQuery - performs an SQL update\n 32    * @param sql the update string\n 33    * @return List<HashMap<String,Object>> Each row is a map where columns are\n 34    *         fetched by (lower case) column name\n 35    * @throws SQLException\n 36    */\n 37   public int doUpdate(String sql) throws SQLException {\n 38     Statement st = conn.createStatement();\n 39     int rows_affected = st.executeUpdate(sql);\n 40     return rows_affected;\n 41   }\n 42 \n 43   /**\n 44    * doQuery - performs an SQL query\n 45    * @param sql the query string\n 46    * @return List<HashMap<String,Object>> Each row is a map where columns are\n 47              fetched by (lower case) column name\n 48    * @throws SQLException\n 49    */\n 50   public List<HashMap<String,Object>> doQuery(String sql) throws SQLException {\n 51     Statement st = conn.createStatement();\n 52     ResultSet rs = st.executeQuery(sql);\n 53     List<HashMap<String,Object>> ret = convertResultSetToList(rs);\n 54     rs.close();\n 55     st.close();\n 56     return ret;\n 57   }\n 58 \n 59   /**\n 60    * textSearch - search for a space separated query string in a specific column\n 61    *              in a specific table.\n 62    *\n 63    * Please note that this utility method handles a very general case. I usually\n 64    * start out with a general purpose method like <strong>textSearch</strong>\n 65    * and then as needed add additional methods that are very application\n 66    * specific (e.g., selecting on different columns, etc.)\n 67    *\n 68    * @param tableName\n 69    * @param coloumnToSearch\n 70    * @param query\n 71    * @return List<HashMap<String,Object>> Each row is a map where columns are\n 72    *                                      fetched by (lower case) column name\n 73    * @throws SQLException\n 74    */\n 75   public List<HashMap<String,Object>> textSearch(String tableName,\n 76                                                  String coloumnToSearch,\n 77                                                  String query)\n 78          throws SQLException {\n 79     // we need to separate query words with the & character:\n 80     String modifiedQuery = query.replaceAll(\" \", \" & \");\n 81     String qString = \"select * from \" + tableName + \" where to_tsvector(\" +\n 82                      coloumnToSearch + \") @@ to_tsquery('\" +\n 83                      modifiedQuery +  \"')\";\n 84     Statement st = conn.createStatement();\n 85     ResultSet rs = st.executeQuery(qString);\n 86     List<HashMap<String,Object>> ret = convertResultSetToList(rs);\n 87     rs.close();\n 88     st.close();\n 89     return ret;\n 90   }\n 91 \n 92   /**\n 93    * convertResultSetToList\n 94    *\n 95    * The following method is derived from an example on stack overflow.\n 96    * Thanks to stack overflow users RHT and Brad M!\n 97    *\n 98    * Please note that I lower-cased the column names so the column data for each\n 99    * row can uniformly be accessed by using the column name in the query as\n100    * lower-case.\n101    *\n102    * @param rs is a JDBC ResultSet\n103    * @return List<HashMap<String,Object>> Each row is a map where columns are\n104    *         fetched by (lower case) column name\n105    * @throws SQLException\n106    */\n107   private List<HashMap<String,Object>> convertResultSetToList(ResultSet rs)\n108           throws SQLException {\n109     ResultSetMetaData md = rs.getMetaData();\n110     int columns = md.getColumnCount();\n111     List<HashMap<String,Object>> list = new ArrayList<HashMap<String,Object>>();\n112     while (rs.next()) {\n113       HashMap<String,Object> row = new HashMap<String, Object>(columns);\n114       for(int i=1; i<=columns; ++i) {\n115         row.put(md.getColumnName(i).toLowerCase(),rs.getObject(i));\n116       }\n117       list.add(row);\n118     }\n119     return list;\n120   }\n121 }\n

\n\n\n\n

I use the private method convertResultSetToList (lines 107 through 102) to convert a result set object (class java.sql.ResultSet) to pure Java data. This method returns a list. Each element in the returned list contains a map representing a row. the keys of the map are table column names and the values are the row value for the column.

\n\n

Besides rhe constructor, this class contains three public methods: doUpdate, doQuery and textSearch. Both take as an argument a string containign an SQL statement. I assume that you know the SQL language, and if not the following test code for the class PostgresUtilities provides a few simple SQL statement examples for creating tables, adding and updating rows in a table, querying to find specific rows, and performing full text search.

\n\n

The test program PostgresTest contains examples for using this utility class. In the following snippets I will alternate code from PostgresTest with the outputs from the snippets:

\n\n

    PostgresUtilities pcon = new PostgresUtilities();\n    int num_rows = pcon.doUpdate(\n      \"create table test (id int, name varchar(40), description varchar(100))\");\n    System.out.println(\"num_rows = \" + num_rows);\n

\n\n\n\n

The doUpdate method returns the number of modified rows in the database. From the output you see that creating a table changes no rows in the database:

\n\n

num_rows = 0\n

\n\n\n\n

    PostgresUtilities pcon = new PostgresUtilities();\n    int num_rows = pcon.doUpdate(\n      \"create table test (id int, name varchar(40), description varchar(100))\");\n    System.out.println(\"num_rows = \" + num_rows);\n

\n\n\n\n

The output is:

\n\n

num_rows = 0\n

\n\n\n\n

    num_rows = pcon.doUpdate(\n      \"CREATE INDEX description_search_ix ON test USING gin(to_tsvector('english\\\n', description))\");\n    System.out.println(\"num_rows = \" + num_rows);\n    num_rows = pcon.doUpdate(\n\t  \"insert into test values (1, 'Ron', 'brother who lives in San Diego')\");\n    System.out.println(\"num_rows = \" + num_rows);\n    System.out.println(\"num_rows = \" + num_rows);\n

\n\n\n\n

The output is:

\n\n

num_rows = 0\n\nnum_rows = 1\n

\n\n\n\n

    num_rows = pcon.doUpdate(\n\t  \"insert into test values (1, 'Ron', 'brother who lives in San Diego')\");\n    System.out.println(\"num_rows = \" + num_rows);\n    num_rows = pcon.doUpdate(\"insert into test values (1, 'Anita', 'sister inlaw\\\n who lives in San Diego')\");\n    System.out.println(\"num_rows = \" + num_rows);\n

\n\n\n\n

The output is:

\n\n

num_rows = 1\n

\n\n\n\n

    List<HashMap<String,Object>> results = pcon.doQuery(\"select * from test\");\n    System.out.println(\"results = \" + results);\n

\n\n\n\n

The output is:

\n\n

results = [{name=Ron, description=brother who lives in San Diego, id=1},\n           {name=Anita, description=sister inlaw who lives in San Diego, id=1}]\n

\n\n\n\n

    String name = \"Ron\";\n    String search_string = \"sister\";\n    results = pcon.doQuery(\n        \"select * from test where name = '\"  +name +\n        \"' and to_tsvector(description) @@ to_tsquery('\" + search_string + \"')\");\n    System.out.println(\"results = \" + results);\n

\n\n\n\n

The output is:

\n\n

results = []\n

\n\n\n\n

    results = pcon.doQuery(\n        \"select * from test where to_tsvector(description) @@ to_tsquery('\" +\n        search_string + \"')\");\n    System.out.println(\"results = \" + results);\n

\n\n\n\n

The output is:

\n\n

results = [{name=Anita, description=sister inlaw who lives in San Diego, id=1}]\n

\n\n\n\n

    num_rows = pcon.doUpdate(\"drop table test\");\n    System.out.println(\"num_rows = \" + num_rows);\n

\n\n\n\n

The output is:

\n\n

num_rows = 0\n

\n\n\n\n

There is a lot of hype concerning NoSQL data stores and some of this hype is well deserved: when you have a massive amount of data to handle then splitting it accross many servers with systems like the Hadoop File System or Cassandra allows you to process large data sets using many lower cost servers. That said, for many applications, you don’t have very large data sets so general purpose database systems like Postgres make more sense. Ideally you will wrap data access in your own code to minimize future code changes if you switch data stores.

\n\n

Wrap Up

\n\n

Much of my career has involved building systems to store and provide access to information to support automated and user in the loop systems. Knowledge Management is the branch of computer science that helps turn data into data into information and information into knowledge. I titled this chapter “Knowledge Management-Lite” because I am just providing you with ideas for your own KM projects and hopefully changed for the better how you look at the storage and use of data.

\n\n\n

Book Wrap Up

\n\n

This book has provided you with my personal views on how to process information using machine learning, natural language processing, and various data store technologies. My hope is that at least some of my ideas for building software systems has been of use to you and given you ideas that you can use when you design and build your applications.

\n\n

My philosophy in general for building systems that solve real problems in the world includes the following system:

\n\n

Learn new technologies as a background exercise to “keep a full toolbelt.” This is especially important now that technology is changing rapidly and often new technologies make it easier to implement new functionality. A major reason that I wrote this book is to introduce you to some of the technology that I have found to be most useful.
Make sure you solve the right problems. Build systems that have an impact on your company or organization, and on the world in general by building things that you think will have the greatest positive impact.

\n\n

Best regards,\nMark Watson  November 10, 2015\n

\n\n\n\n

\n\n","tocStructure",[283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338],{"_76":77,"_43":78,"_70":-5,"_79":80,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":86,"_70":-5,"_79":87,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":92,"_70":-5,"_79":93,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":77,"_43":95,"_70":-5,"_79":96,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":99,"_70":-5,"_79":100,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":102,"_70":-5,"_79":103,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":105,"_70":-5,"_79":106,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":108,"_70":-5,"_79":109,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":111,"_70":-5,"_79":112,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":114,"_70":-5,"_79":115,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":117,"_70":-5,"_79":118,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":77,"_43":120,"_70":-5,"_79":121,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":124,"_70":-5,"_79":125,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":127,"_70":-5,"_79":128,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":130,"_70":-5,"_79":131,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":133,"_70":-5,"_79":134,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":77,"_43":136,"_70":-5,"_79":137,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":140,"_70":-5,"_79":141,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":143,"_70":-5,"_79":144,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":146,"_70":-5,"_79":147,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":149,"_70":-5,"_79":150,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":152,"_70":-5,"_79":153,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":155,"_70":-5,"_79":156,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":158,"_70":-5,"_79":159,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":161,"_70":-5,"_79":162,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":164,"_70":-5,"_79":165,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":167,"_70":-5,"_79":168,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":170,"_70":-5,"_79":171,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":77,"_43":173,"_70":-5,"_79":174,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":177,"_70":-5,"_79":178,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":180,"_70":-5,"_79":181,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":183,"_70":-5,"_79":184,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":186,"_70":-5,"_79":187,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":77,"_43":189,"_70":-5,"_79":190,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":193,"_70":-5,"_79":194,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":196,"_70":-5,"_79":197,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":199,"_70":-5,"_79":200,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":77,"_43":202,"_70":-5,"_79":203,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":206,"_70":-5,"_79":207,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":209,"_70":-5,"_79":210,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":212,"_70":-5,"_79":213,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":77,"_43":215,"_70":-5,"_79":216,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":219,"_70":-5,"_79":220,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":222,"_70":-5,"_79":223,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":225,"_70":-5,"_79":226,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":228,"_70":-5,"_79":229,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":231,"_70":-5,"_79":232,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":234,"_70":-5,"_79":235,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":237,"_70":-5,"_79":238,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":240,"_70":-5,"_79":241,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":77,"_43":243,"_70":-5,"_79":244,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":247,"_70":-5,"_79":248,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":250,"_70":-5,"_79":251,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":253,"_70":-5,"_79":254,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":85,"_43":212,"_70":-5,"_79":256,"_72":81,"_88":-5,"_89":-5,"_90":-5},{"_76":77,"_43":258,"_70":-5,"_79":259,"_72":81,"_88":-5,"_89":-5,"_90":-5},"publicChapterContent","nextChapter","sections",[],"routes/_store+/_layout","routes/_store+/_readers+/_layout","actionData","errors"]

Table of Contents

Preface

Book Outline

If You Did Not Buy This Book

Network Programming Techniques for the Internet of Things

Motivation for IoT

Running the example programs

Build

Run service discovery

Run UDP experiments:

Design Pattern

Directory Lookups

Create and Register a Service

Lookup a Service by Name

User Data Protocol Network Programming

Example UDP Server

Example UDP Client

Multicast/Broadcast Network Programming

Multicast Broadcast Server

Multicast Broadcast Listener

Wrap Up on IoT

Natural Language Processing Using OpenNLP

Using OpenNLP Pre-Trained Models

Training a New Categorization Model for OpenNLP

Using Our New Trained Classification Model

Using the OpenNLP Parsing Model

Machine Learning Using Apache Spark

Setting Up Spark On Your Laptop

Hello Spark - a Word Count Example

Introducing the Spark MLlib Machine Learning Library

MLlib Logistic Regression Example Using University of Wisconsin Cancer Database

MLlib SVM Classification Example Using University of Wisconsin Cancer Database

MLlib K-Means Example Program

Converting Text to Numeric Feature Vectors

Using K-Means to Cluster Wikipedia Articles

Using SVM for Text Classification

Using word2vec To Find Similar Words In Documents

Chapter Wrap Up

Anomaly Detection Machine Learning Example

Motivation for Anomaly Detection

Math Primer for Anomaly Detection

AnomalyDetection Utility Class

Example Using the University of Wisconsin Cancer Data

Deep Learning Using Deeplearning4j

Deep Belief Networks

Deep Belief Example

Deep Learning Wrapup

Web Scraping Examples

Motivation for Web Scraping

Using the jsoup Library

Wrap Up

Linked Data

Example Code

Overview of RDF and SPARQL

SPARQL Query Client

JSON Client

DBPedia Entity Lookup

Java Data Client

JSON Client

Annotate Text with DBPedia Entity URIs

Resolving Named Entities in Text to Wikipedia URIs

Java Named Entity Recognition Library

Combining Data from Public and Private Sources

Wrap Up for Linked Data

Java Strategies for Working with Cloud Data: Knowledge Management-Lite

Motivation for Knowledge Management

Using Google Drive Cloud Takeout Service

Processing Microsoft Excel Spreadsheet xlsx and Microsoft Word docx Files

Processing iCal Calendar Files

Processing Email mbox Files

Using Postgres as a Local Document Store with Text Search

Wrap Up

Book Wrap Up