The publisher analyzer attempts to determine the publisher of a web page from analyzing the web page's content. There are multiple moving parts to this analyzer, but its general internal workflow is as follows:

1. It crawls a given webpage URL
2. It extracts the page's content and extracts its meta-data (including "defluffing", which is the removal of navigation, ads, and normal Web page boilerplate)
3. It tags all of the organizations (anything that is considered an organization in KBpedia) across the extracted content using the organization entities that exist in the knowledge base
4. It conducts certain specialty analysis related to page "signals" that might indicate an organizational entity
5. It detects unknown entities that will eventually be added to the knowledge base after curation
6. It performs an in-depth analysis of the organization entities (known or unknown) that got tagged in the content of the web page, and analyzes which of these is the most likely to be the publisher of the web page.

The machine learning system leverages existing algorithms to calculate the likelihood that an organization is the publisher of a web page and to detect unknown organizations. These are conventional uses of these algorithms. What differentiates the publisher analyzer is its knowledge base. We leverage KBpedia to detect known organization entities. We use the knowledge in KBpedia's combined KBs for each of these entities to improve the analysis process. We constrain the analysis to certain types (by inference) of named entities, etc. The special sauce of this entire process is the fully integrated set of knowledge bases that comprise KBpedia, including its hundreds of thousands of concepts, 39,000 reference concepts, and 20 million known entities.

This use case begins with three public datasets already in KBpedia: Wikipedia (via DBpedia), Freebase and USPTO.

These public datasets are then compared to two private datasets, which contain high-quality, curated, and domain-related listings of organizations. The numbers of organizations contained in these private datasets are much smaller than those in the public ones, but they are also more relevant to the domain. These private datasets are fairly typical of the specific information that an enterprise may have available in its own domain.

The reference standard, or "gold standard", employed in this use case is composed of 511 randomly selected Web pages that are manually vetted and characterized. (As a general rule of thumb we find about 500 examples in the positive training set to be adequate.)

The gold standard is really simple. For each of the URLs we have in the standard, we determine the publishing organization manually. Then once the organization is determined, we search in each dataset to see if the entity is already existing. If it is, then we add the URI (unique identifier) of the entity in the knowledge base into the gold standard. It is this URI reference that is used to determine if the publisher analyzer properly detects the actual publisher of the web page.

We also add a set of 10 web pages manually for which we are sure that no publisher can be determined for the web page. These are the 10 True Negative (see below) instances of the gold standard.

The gold standard also includes the identifier of possible unknown entities that are the publishers of the web pages. These are used to calculate the metrics when considering the unknown entities detected by the system.

The goal of the analysis is to determine how good the analyzer is to perform the task (detecting the organization that published a web page on the Web). What we have to do is to use a set of metrics that will help us understand the performance of the system. The metrics calculation is based on the confusion matrix.

When we are processing a new run, all results are characterized according to four possible scoring values:

1. True Positive (TP): test identifies the same entity as in the gold standard
2. False Positive (FP): test identifies a different entity than what is in the gold standard
3. True Negative (TN): test identifies no entity; gold standard has no entity
4. False Negative (FN): test identifies no entity, but the gold standard has one

The True Positive, False Positive, True Negative and False Negative (see Type I and type II errors for definitions) should be interpreted in the manner common to in named entity recognition tasks.

This simple scoring method allows us to apply a series of metrics based on the four possible scoring values:

1. Precision: is the proportion of properly predicted publishers amongst all of the publishers that exists in the gold standard (TP / TP + FP)
2. Recall: is the proportion of properly predicted publishers amongst all the predictions that have been made (good and bad) (TP / TP + FN)
3. Accuracy: is the proportion of correctly classified test instances; the publishers that could be identified by the system, and the ones that couldn't (the web pages for which no publisher could be identified). ((TP + TN) / (TP + TN + FP + FN))
4. f1: is the test's equally weighted combination of precision and recall
5. f2: is the test's weighted combination of precision and recall, with a preference for recall
6. f0.5: the test's weighted combination of precision and recall, with a preference for precision.

The F-score is a common combined score of the general prediction system. The F-score is a measure that combines precision and recall via their harmonic mean. The f2 measure weighs recall higher than precision (by placing more emphasis on false negatives), and the f0.5 measure weighs recall lower than precision (by attenuating the influence of false negatives). Cognonto includes all three F-measures in its standard reports to give a general overview of what happens when we put an emphasis on precision or recall. Some clients prefer limiting false positives at the sake of lower recall; others want fuller coverage.

Still, in most instances, we have found customers find accuracy to be the most useful metric. We particularly emphasize that metric in the results below.

The first step is to create the starting basis that includes no dataset. Then we will add different datasets, and try different combinations, when computing against the gold standard such that we know the impact of each on the metrics.

(table (generate-stats :js :execute :datasets []))

True positives:  2
False positives:  5
True negatives:  19
False negatives:  485

+--------------+--------------+
| key          | value        |
+--------------+--------------+
| :precision   | 0.2857143    |
| :recall      | 0.0041067763 |
| :accuracy    | 0.04109589   |
| :f1          | 0.008097166  |
| :f2          | 0.0051150895 |
| :f0.5        | 0.019417476  |
+--------------+--------------+

Now, let's see the impact of each of the datasets that exist in the knowledge base we created to perform these tests. This will gives us an indicator of the inherent impact of each dataset on the prediction task.

A more realistic analysis is to use a combination of datasets. Let's see what happens to the performance metrics if we start combining the public datasets only.

Now let's see the impact of using adding the private datasets. We will continue to use the combination of the three public datasets (Wikipedia, Freebase and USPTO) to which we will add the private datasets (PD #1 and PD #2).

There is one last feature with the publisher analyzer that we should highlight. The analyzer also allows us to identify unknown entities from the web page. (An "unknown entity" is identified as a likely organization entity, but which does not already exist in the KB.) Sometimes, it is the unknown entity that is the publisher of the web page. The usefulness of the unknown entity identification is to flag new possible entities (organizations, in this case) that should be considered for addition to the overall knowledge base.

(table (generate-stats :js :execute :datasets :all))

True positives:  345
False positives:  104
True negatives:  19
False negatives:  43

+--------------+------------+
| key          | value      |
+--------------+------------+
| :precision   | 0.76837415 |
| :recall      | 0.88917524 |
| :accuracy    | 0.7123288  |
| :f1          | 0.82437277 |
| :f2          | 0.86206895 |
| :f0.5        | 0.78983516 |
+--------------+------------+

As we can see, the overall accuracy improved by 19.73% when considering the unknown entities compared to the public and private datasets.