Recently Daniel Linna, Professor of Law and the Director of The Center for Legal Services Innovation at Michigan State University, published the first iteration of The Legal Services Innovation Index (LSII) The first part of this ambitious project is the Innovation Catalog of innovative products, legal services and consulting services offered by large law firms. It appears that the already impressively large catalog will be primarily managed by Linna’s law student support team and other volunteers based on their own research efforts and communal submissions. The second part of the project is the Law Firm Index (LFI) of ten innovation categories devised by Linna and his team and scored by applying a standardized set of keyword and key phrase searches of large law firm websites utilizing Google Advanced Search and recording the results counts in the LFI. The premise here is that innovative law firms will use innovation-related terms more frequently on their websites compared to non-innovative firms. The hope is that the Google search results counts will provide a reasonably objective means of measuring large law firm innovation across various temporal and demographic dimensions. Good idea…but a deeply flawed implementation.
Granted, Linna is quick to caution against reading too much into this first generation of the index. He dedicates a large portion of the site to the various limitations and challenges posed by relying on Google search results counts generated from what is basically law firm-generated propaganda. Nevertheless, he seems to be comfortable enough with the fundamental premise to release the details into the “wild” via an online Tableau-based data visualization tool that allows anyone to perform rudimentary grouping and sorting of the index. Already, we are seeing the first interpretive fruits of this initiative. See, for instance, here. Professor Linna himself mostly refrains from engaging in the fun and games of data interpretation. Instead, his message is more along the lines of an appeal for feedback on the effort. It is in response to that appeal that I offer my comments here. [Note: I want to emphasize that my specific criticisms of the methodology used to construct the LFI are also identified in the Introduction and Methodology section that accompanies the LFI itself. Please refer to comment section below for Professor Linna’s response to this post and my reply to Professor Linna.]
Applying the Knowledge Management Smell Test
KM is one of the ten innovation categories tracked by the LFI. The specific Google search used in the measurement is: “knowledge management” OR “knowledge engineering.” The number of times I’ve personally used the term “knowledge engineering” or personally heard it used by a legal KM peer I could count on the fingers of one hand and still have enough fingers left over to flip the bird at anyone who claims to be a “knowledge engineer.” We’ll let that one slide because its almost universal irrelevance to the topic of legal KM has no meaningful effect on the results. Below is a screen grab of the KM index sorted by search result count (only the “top” 46 firms are shown here). 
The numbers shown above represent the number of pages on the law firms’ websites that purportedly refer to “knowledge management or “knowledge engineering.” Do those posted numbers pass your smell test? They certainly do NOT pass mine. Sure, I would expect to see Littler and Ogletree ranked high. They share two of the five most recent ILTA KM Professional of the Year Award winners (Susan Woodhouse and Patrick DiDomenico); but by that standard White & Case (Oz Benamram) is nowhere to be found, and Wilson Sonsini (Chris Boyd) and Debevoise (Steven Lastres) only make it into the lower half of the top 46. What I see is a mix of firms that legitimately belong somewhere near the top but also quite a few firms that I rarely or never come across as KM leaders and innovators. Just as importantly, quite a few of the firms I would expect to see in the top 46 are not listed. I strongly suspect that any random selection of 46 of the largest 263 firms with a weighting bias added based on firm (and website) size would yield a top 46 that “feels” about as valid as this one.
Over 5000 Pages that Refer to KM…Really?
One thing that should be immediately obvious to any legal KM practitioner is the absurd number of KM hits reported for many firms. There is simply no way that KM is substantively referenced on more than, say, 10-15 pages for any given law firm website (a few very large firms with an extraordinary number of KM professionals may exceed that number due to posted biography pages). Indeed, show me a firm that actually dedicates more than one single substantive website page to its KM efforts, and I’ll show you a very rare exception to the rule. Despite that harsh reality, according to the LFI over 50 of the monitored firms have more than 15 KM pages. And then there’s Shearman’s and Ogletree’s 2000+ pages and Littler’s 5000+ pages!
Consistent with my skeptical expectations, I found that none of the dozen or so sites that I manually checked legitimately match the number of hits returned by Google. By cross-checking search results from Google with the individual websites’ internal search engines and by examining obvious false-hit pages returned by Google, I found four primary reasons for the gross search hit inflation:
- Some sites with the biggest hit inflation (e.g., Shearman, Ogletree and Littler) include either visible or hidden HTML code in page headers or footers that reference “knowledge management.”
- Some sites are structured to repeat the same page a number of times. For instance, a lawyer bio page may end up being counted as multiple hits because it appears in multiple practice or office subsites. Thus, a KM lawyer page may be counted multiple times and Google does not necessarily exclude the duplicates.
- Google’s Advanced Search page count is NOT based on exact matches of searched phrases. It is not uncommon for the words “knowledge” and “management” to appear somewhere on a page posted on a law firm website and Google will usually include those pages in its count. It will even include pages that have only one of the words. (Note: Using the “Verbatim” option in Google restricts results to the exact phrase specified in the search but does not return a total page count and, thus, was not used in the LFI measurement.)
- A KM lawyer (co)authors and posts numerous substantive legal alerts/articles on a firm’s website. Knowledge Management appears in the author title or credits included in the article but not substantively in the alert/article itself.
With these considerations in mind, and by way of example, let’s take a closer look at Littler’s numbers:
| Source | Count | Comment |
| Google Adv Search | 5700 | Total # of Littler KM pages found by LSII using Google’s advanced search. Wow! |
| Google Adv Search | 5520 | Total # of Littler KM pages I found when I used Google advanced search. (Google returns different totals for different users based on the users’ prior search activity. Professor Linna also noted this problem.) |
| Littler Website Search Tool | 4612 | Total # of Littler pages found when using knowledge management OR knowledge engineering [no quotes] as the search criteria. |
| Google Adv Search | 488 | Total # of Littler KM pages found when the Verbatim filter in Google’s advanced search results is applied. (Required manual counting.) |
| Littler Website Search Tool | 93 | Total # of Littler pages found when using “knowledge management” OR “knowledge engineering” [with quotes] as the search criteria |
| Manual Review of Littler Website Search Results | 31 | Total # of Littler pages found that use “knowledge management” other than in the author title or credits of a legal alert/article page. |
| Littler Website Search Tool | 23 | Total # of Littler professionals’ bio pages in which “knowledge management” appears. |
| Manual Review of Littler Website Search Results | 8 | Total # of Littler pages that substantively refer to “knowledge management” (excluding bio pages). |
| Littler Website Search Tool | 2 | Total # of Littler pages specifically dedicated to Littler’s KM program, services and products. |
The inflated number reported for Littler in the LFI is almost completely due to the presence of knowledge management in (non-visible) HTML script that probably appears on every page on the site. When that site structuring issue is removed, Littler’s numbers fall in line with the rest of the pack. To me, the page counts highlighted in red above are the meaningful ones for Littler. To be sure, Littler is a widely recognized KM leader and innovator. As already noted, the firm deserves to appear at or near the top of any law firm innovation index. However, the fact that the chosen methodology for scoring the LFI happens to place Littler at the top is really just a lucky accident of how the firm’s website is structured and not because the specific LFI analytics strategy works. Littler is an extreme case, but many, if not all, of the other firms that ended up toward the top of the heap also benefited from some combination of the noted artificial boosts to their numbers, while other deserving firms are buried because they simply chose not to deploy similar website design and style decisions.
Knowledge Management-to-Noise Ratio
Even the most innovative firms are likely to reference their KM efforts, projects and awards relatively few times on their public websites. As noted, some larger firms with big KM staffs will also have more KM professional biography pages, but that only marginally signifies innovation and will perhaps inappropriately bias the index in favor of larger firms simply because they are larger and not because they are more innovative. Thus, the fundamental problem here is that this website search-based strategy generates a very low “signal” in relation to the “noise” generated by false hits from uses of these common words in unrelated contexts and magnified by duplicate pages and varying website structuring/design strategies. The result is an untrustworthy and largely random ordering.
What About the Other Categories?
KM is only one of the ten innovation categories tracked by the LFI, and it is the only one that I looked at in depth. However, even a cursory examination of the other categories reveals similar problems related to false hits caused by coincidental website structuring and styling, page duplication and over-inclusion based on Google’s miscounts of phrase searches that contain words that commonly appear in multiple legal website contexts. Consider, for example, the case of Berwin Leighton Paisner (BLP) in the artificial intelligence category. BLP is widely recognized for its pioneering AI work with RAVN, so it is hardly surprising that it cracks the top 15 in the AI category with a reported total of 269 webpage hits. Likewise, when I run the specified Google search (“machine learning” OR “deep learning” OR “artificial intelligence” site:blplaw.com) I get 277 results. However, by simply adding “-oil” to the search string, the hit count plummets to just nine results. What does oil have to do with AI? As it turns out, both Artificial Intelligence and Oil and Gas are filters in the press release section of BLP’s website. Both of those terms appear on several hundred posted press release pages as selectable filters but neither appears in the (substantive) body of the press releases. In fact, only one posted press release appears to have the Artificial Intelligence tag applied; and as best I can tell, only about 12 pages on the BLP site actually relate to or substantively describe or promote BLP’s AI systems and services.
A search strategy that claims 1000+ results from individual law firm websites for ANY topic (let alone law firm operational topics) is obviously flawed. Nevertheless, 1000+ law firm results are reported for eight out of ten of the categories. In short, the same kind of signal-to-noise problem I saw in my more extensive review of the KM category appears to infest the other categories as well. You simply cannot assume that any individual law firm result is valid, which means that the index, as a comparative tool, is unreliable as well.
Shhhhhh…Don’t Tell the Law Firm Business Development and PR People About This
The worthy purpose behind the LSII is to stimulate large law firm innovation and ultimately to improve the delivery of legal services generally. An objective benchmark for comparing law firms will arm clients with better information and enable them to select more progressive firms. This could trigger a virtuous innovation improvement loop where law firms pay more attention to new legal service delivery models and competitively respond to improvements implemented by other firms. Sounds great…just one problem though. The index measures words, not deeds. Worse, it measures the words primarily generated by the PR function of law firms. Thus, the more attention and success the LSII achieves, the greater the risk that law firms will respond with SEO strategies targeted at raising their scores in the LFI rather than genuine (but far more expensive and difficult to implement) innovations. With no good way to validate the measurement itself and no way to distinguish between innocent PR padding and abusive optimizations, the credibility of the project will remain highly questionable.
The Not So Fatal Flaw?
Getting law firms to pay more lip service to innovation is a step in the right direction even if the methodology is flawed to begin with and vulnerable to being gamed, right? That seems to be argument some are making in support of the LSII undertaking, but I disagree. As things stand now, the LFI portion of the LSII project is rather obviously broken and untrustworthy. Setting aside the risks of SEO manipulation in the future, the clear and present danger is that the whole undertaking will reinforce the already hyper skepticism of the legal establishment toward virtually all of the “buzzwords” that comprise the search parameters at the core of the index. As an exercise in data analytics the index doubly fails: it fails to reasonably model the real status of law firm innovation efforts; and it fails to demonstrate the usefulness of data analytics itself as a tool to be embraced by lawyers. My advice here is to fail fast and move on to a better strategy for building a useful law firm index to supplement the otherwise worthwhile catalog portion of the project.
Brent,
Thank you for your feedback. As I mentioned in our discussion on Twitter, I raised each of these points when I released Phase 1, version 1.0 of this project. You alluded to this, but I believe you’ve missed substantial portions of the ground that I covered.
For the methodology related to the Law Firm Index searches, please see: http://www.legaltechinnovation.com/law-firm-index/
For the overview of the project, please see: http://www.legaltechinnovation.com/overview/
Most relevant is my note on the Law Firm Index searches page:
“We recorded the number of results reported in the first page returned by Google Advanced Search after submitting the search query. This first number returned by Google is an estimate of the number of hits. (For information about Google search, visit How Google Search Works.) The actual number could be higher or lower. Nevertheless, because the same searches were used for all law firms the resulting information, even if not exact in absolute values, is useful as a relative measure.”
The data was used as a relative measure to create the “Percentile Order by Category” measure, as fully discussed on that page. The other points you raised are addressed on the Law Firm Index searches page.
I appreciate that you recognize the value of the Catalog. I’ve received nearly 20 updates which will soon be added to the Catalog. That said, the searches also provide value. They should not be used to rank firms, for all of the reasons I’ve stated. But the searches of firm’s websites provide valuable information, particularly in a space where we’ve had little more than anecdotes about innovation.
Thank you, again.
Dan
Dan, thanks for the comment. I do not believe that I “missed substantial portions of the grounds covered” in the Introduction and Methodology section (or elsewhere) of the documentation you published along with the index and catalog. In fact, I specifically noted at the beginning of the post that you dedicated a significant portion of the site to the various limitations and challenges of the Google search based strategy. However, to further clarify and acknowledge that you address at least in general terms the issues I explore in the post, I’ve added an additional note in the post itself. Nevertheless, I believe that my efforts very tangibly illustrate the problems you acknowledged in your methodology notes. Without this sort of concrete description of the extent of the problem, the caution you sounded remains too abstract and too easily disregarded by readers anxious to jump right to the index itself.
With respect to the “most relevant” methodology note that you’ve included in your comment above, I will point out three things. First, your assertion that the index is “useful as a relative measure” because the same Google search strategy was consistently applied to all law firms is simply that: an assertion. You have not supported it with any validation testing based on manual review of a sufficient number of individual sites to confirm that the wide range of confounding variables do not meaningfully negate the results. As I explained (and supported with examples) the number of valid hits (i.e., those based on substantive uses of the search terms) is going to be rather small for any given law firm website for most of the search categories. That makes the index highly vulnerable to the large number of false positive and missed hits (also demonstrated) that can easily occur and swamp the valid hits. Second, you have not shown that the problem at the individual category level is offset rather than magnified by combining the categories into a master score. My personal guesstimate is that the same structuring and site size issues that affect one category also affect others. Third, “The Percentile Order by Category” does not overcome the false boosting problem at the individual category level. It does not change the ordering at that level but, rather, simply smooths out the scores and hides what is otherwise obvious from the raw scores (namely, that some firm scores are grossly boosted by false positives). The ranking outcome from the combined score is affected somewhat by the smoothing. But, again, we have no basis for assuming that combining the categories for each firm suppresses rather than amplifies the measurement errors at the individual category levels. What we are left with is an analytical edifice built on acknowledged problematic data. To me, that is the wrong way to launch an otherwise worthwhile undertaking like the LSII.