Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
26,665 characters · 13 sections · 18 citation commands
BERT based freedom to operate patent analysis
\baselineskip=1.3\normalbaselineskip
With new inventions and the launch of new products, an inventor or entrepreneur is often faced with the question of possible patent infringement. There are risks of high procedural costs, expensive penalties and compensation as well as a ban on the sale of the product, despite high development expenditures. Therefore, inventors try to assess the risks of patent infringement before commencing commercial activities, which can possibly be associated with an extensive freedom to operate (FTO) analysis.
FTO patent analysis are generally very time-consuming. Patent claims must be identified, which could cover the invention or product. It should be taken into account, that the claims of pending patent applications can still change during the examination process. Often the claims become modified by adding further technical features.
Currently, the vast majority of patent searches are performed based on Boolean methods. Patents are preselected on the basis of keywords, usually in combination with suitable IPC (International Patent Classification\footnote{\url{https://www.wipo.int/classifications/ipc/en}.}) and CPC (Cooperative Patent Classification\footnote{\url{https://worldwide.espacenet.com/classification?locale=en_EP}.}) patent classes. Nevertheless, usually hundreds to more than a thousand patent documents remain that have to be viewed and assessed by a patent expert. Even with only a small amount of time spent per patent, the total effort is enormous and it is not uncommon for relevant patents to be overlooked. With a Boolean method, all synonyms must also be considered in the keyword concepts, which can be very difficult, especially in patent language. Also the relationship of the keywords to one another, which have to be modeled in a complex manner with distance operators, can lead to lose interesting patents. Heteronyms can lead to a large number of irrelevant documents, which makes evaluation by the patent expert more extensive and error-prone. The work is also becoming more and more complex due to the rapidly growing number of new patent applications and increasing number of technological interdisciplinary inventions.
In this paper we try out a method to support FTO patent searches and analysis with the help of machine learning and to improve the efficiency of the search and patent evaluation.
This work essentially reverses the procedure described in the publication freunek2021bert, where BERT is applied to patent novelty searches. While a relevant description is searched for a claim as input in freunek2021bert, here, BERT is fine-tuned for the task to identify FTO relevant independent claims for a description of an invention or a product.
For this reason, we concatenate descriptions to their independent claims. Each description represents an invention, which is protected by the corresponding claims. The idea is, by fine-tuning BERT on that task, BERT learns to identify FTO relevant claims based on a description of an invention or a product as input.
In this paper we describe this method in detail and as a test, apply BERT to five DOCDC abstracts of inventions. To our knowledge, the method described in this work has not been published yet. The method presented here offers the following advantages (Freunek, Bodmer):
The paper is structured as follows: First, we identify related work in Chapter (ref). Then we show our approach to train BERT on FTO analysis in Chapter (ref): How we generate the data and which BERT model we use, how we practically apply the trained BERT for FTO patent searches or analysis. In Chapter (ref), we will apply the described method and train BERT on real patent data and finally test the trained BERT on five inventions to identify FTO relevant patents. In Chapter (ref) we give some conclusions and possibilities for future research tasks.\\
This paper mainly refers to the work in freunek2021bert. There, the idea is to concatenate the patent claims to their own description to fine-tune BERT for patent novelty searches. Such a trained BERT should be able to identify novelty destroying descriptions for claims. This work is very closely related to this idea, hence this paper shows many parallels to the cited work.
BERT is a pre-trained model for tasks in natural language processing and is based on a transformer architecture. The self-attention mechanism relates different positions in a sentence or input sequence to each other, as described in Vaswani2017 and Devlin2018. The pre-trained BERT is fine-tuned to the task of interest.
Some papers describe the application of BERT in the field of patent classification like in Sun2019 and Lee2020. In addition to the work in freunek2021bert, the paper Risch2020 also describes the use of BERT for novelty searches. A detailed study on patent novelty detection can be found in Chikkamath2020AnES. The authors achieved their best results NBSVM (Naive Bayes Support Vector Machine). A further study about prior art searches in combination with GPT2 can be found in Lee2020a.
A survey about deep learning methods applied to patent analysis is given in KRESTEL2021102035. Among other things, the authors identified the trend towards more complex models such as BERT. In the work of Setchi2021, a platform for comparison of state of the art AI techniques in patent searches are demonstrated. A further literature overview of applying AI to intellectual property can be found in Aristodemou2018 or the application to patent landscaping has been reported in Choi2019.
The authors of this paper are not aware of any publications focused on FTO patent searches based on machine learning methods. Possibly, methods that deal with the search of prior art could also be applied to the subject of FTO analysis. \\
The work is based on the following idea: With the development of an invention or a product, the question of its commercial usability arises, whether the invention or the product is already protected by a patent. We will describe how to train BERT on this question. The invention or the product can be illustrated by a brief description. Based on this brief description, the relevant patents are to be identified with the help of BERT. In each patent, there is a description for an invention, which is covered by the corresponding claims. Therefore, in this work we will link the descriptions in the patents with their own independent claims. BERT should thus learn relevant claims to descriptions and finally be able to identify claims or patents FTO relevant to a description as input. \\
To generate the training data for BERT one has to be aware of two things: 1: BERT can process input lengths up to 512 tokens. 2: We suppose that a suitable description of an invention or product is of size of a typical patent abstract, which has a length of roughly 200 words. Since a patent description can amount to several thousand words, we have to slice the description into pieces, which should have the typical length of a patent abstract and, concatenated to an independent claim, fits the input size of BERT with respect to special tokens, like the CLS (for classification) and the SEP (for separation) token. The procedure of slicing patent descriptions is explained in some detail in freunek2021bert. We do not select specific text passages from the description, but rather link the entire description to the independent claims. Of course, the concatenated claim will not always match the item of description piece. For example, when the description cites the related prior art and its disadvantages. However, the procedure described should be a good approximation from a statistical point of view, since the majority of the description in a patent is directly dedicated to the actual invention and its embodiments.
In more detail, this works as follows: Each description-piece $k$ of patent $i$ is combined with the independent claim(s) of the same patent $i$ yielding the first label data:
\\ Now, having the data for the first label, we need to complete the training data set by generating data with the second label, where is no matching between description piece and independent claim, or in other words, where the claim does not cover the description and thus is of no FTO relevance. We make use of the observation , that most independent claims of patent $i$ are of no FTO relevance of the description of patent $j$, with $i \ne j$. Applying this observation, we can generate the second label data according to:
\\ where $i \ne j$. Description pieces and the independent claims are combined in a random manner. The only limitation is that claim and description piece belong to different patents.
By taking the same claims and descriptions pieces for the second label as for the first label, one can make sure, that both labels have the same frequency and BERT learns to distinguish between relevant and non-relevant claims for patent descriptions. Of course, there is a probability of concatenating a description piece randomly to a FTO relevant claim of another patent, but at the end the effect a "confusing" during training should be very low. Finally, each input has the structure:
\\ where the length of the input is padded with PAD tokens during tokenization to the defined maximum input length, if the length of the concatenated description piece and claim is smaller than the maximum input length. Otherwise we get:
\\ The maximum input length can be set to lower values than 512 tokens, but it is expected to get the best results, when the description pieces and claim concatenations fit the maximum input size of BERT. \\
For training BERT, we applied the tokenizer {\bf bert-base-uncased}. An important parameter here is the maximum input sequence length. BERT allows a maximum input length of 512 tokens. It is intuitively obvious to go close to this limit with this model, provided that the computer capacity allows it. In our experiment in chapter (ref) we set the input length to 500 tokens.
Hyperparameters like learning rate, batch size, number of epochs have to be adjusted and a BERT model has to be selected. The probably model of choice is the {\bf Bert For Next Sentence Prediction}. As in freunek2021bert, we will chose the model {\bf Bert For Sequence Classification}.
Once BERT is trained (fine-tuned), BERT can be applied to FTO patent analysis or patent searches with a short description of an invention or a product as input. In our studies we focused training BERT on selected technology fields as in freunek2021bert. The required computer capacity and the volume of data is therefore significantly lower. Also, it cannot be ruled out that a technology-specialized BERT is superior to a general trained BERT (trained to all or at least to several technology fields). The disadvantage, however, is that BERT must be trained specifically for a task, provided that the technology has not been trained before.
To prepare the input, we prepare the patents similar to the data generation procedure described in chapter (ref): The independent claims, which we want to analyze according to FTO relevance for the description of interest, are concatenated according to the structure ("equation") (ref) or (ref). The concatenations should fit the input size of BERT without slicing, since descriptions of inventions or products have typically the size of an abstract and typical abstract and claim concatenations have lengths below 512 words.
To be clear: We train BERT on the complete description of the patents. But the FTO analysis is carried out for a short description of an invention or product, which has the size of an abstract.
Then, the description of the invention or product - $\textrm{description}_\textrm{ip}$ - is concatenated to the independent claims, we get the following structure (by neglecting the CLS and PAD token):
According to chapter (ref), BERT assigns to the analyzed claims logits numbers for both label. This logits represent the assessment of the claims of being FTO relevant (first label) or being non-relevant (second label). If desired (but not necessary), the logits can be further calculated with the Softmax function. The claims should be arranged according to the values of the logits or the Softmax function according to the first label. This results in a sequence according to the relevance of the claims according to BERT´s calculation.
Basically we are faced with the problem, that there is no reference data set and no strict evaluation scheme to assess the performance of BERT or another machine learning model on a FTO task. For a classification task, the precision, the recall and the F1 score could be calculated. This is hardly possible for a FTO analysis. This experiment is therefore intended to give an initial indication of whether the method described above can be used reasonably.
We applied the method to the DOCDB abstracts of five patents (reference patents). The abstracts should represent a short description of the corresponding invention or product, for which we want to make the FTO patent search or patent analysis. BERT should identify the most FTO relevant claims for the five reference patents.
To this end, we performed the following procedure:
The FTO analysis of the first claims is performed for the following patent applications (reference patents): AU 2012315252 B2, US 9,659,410 B2, \\US 9,911,174 B2, US 10,375379 B2 and US 2015/0302541 A1.
All 2577 to-be-searched patents, including the reference patents, are assigned to the IPC or CPC patent class G06T1/60. The 1059 training patents are randomly selected from G06T1/00 and below. All patents are US, GB, AU, CA or IN patents in English language. It was checked, that BERT was not trained on the reference and the other to-be-searched patents, although this is not a condition or limitation of the method presented. Between the randomly chosen training patents and the randomly chosen to-be-searched patents is no overlap. The descriptions of the patents were sliced into pieces of random size in the range of 100 to 200 words. Future research could investigate a dynamic slicing to fit exactly the size of the concatenated claim and description piece to the maximum sequence length, as described in chapter (ref). The maximum sequence length for BERT has been chosen to 500 tokens. Slicing the descriptions of the training patents and the generation of label 0 and label 1 description pieces and claims according to chapter (ref), yields 74'498 training input sequences. We split randomly 9'250 concatenated claim and description pieces for validation. \\
We trained BERT for 2 epochs. The FTO patent search or patent analysis in the to-be-searched group with 2577 patents was performed only on the first claim. The trained BERT yields the results shown in the tables (ref) - (ref). The abstracts and claims of the patents are listed in detail in the Appendix. The results are discussed in the following section.\\
The result is difficult to assess. There is neither a standardized data set nor can one clearly assign relevance or non-relevance to each patent. In addition, in the selected patent class G06T1/60, the claims are very long, i.e. have many features. In practice, each of these features can determine whether the patent is FTO relevant or not. An assessment would be much easier if one were to test patents of a patent class which on average require fewer features in the independent claims. Ultimately, the method presented here will have to be tested in practice to assess its efficiency. Apparently the result looks promising.
The first position in the results of the selected FTO patents is noticeable: this is the patent itself, the abstract of which was used as input to BERT. BERT has not previously learned a direct relationship between this abstract and its first claim. The result shows that, among all claims from the 2577 to-be-searched patents, BERT rated the claim to the patent for which we carried out an FTO analysis as the most relevant, without having "seen" the claim beforehand. This result clearly indicates that BERT is able to identify relevant claims, if existing.
The following points should be noted here, which distinguish the test presented here from a real FTO, but do not represent any restriction with regard to the practicability of the method described: The documents evaluated by BERT may not be complete as other patent classes were completely disregarded. It was also not taken into account whether the patents are still active and it did not distinguish between A-documents (applications) and B-documents (granted patents). Also, only the first claim was evaluated, but not, if existing, further independent claims. \\
In this work, we presented a new method to train BERT for a freedom to operate (FTO) patent search or patent analysis, by concatenating patent descriptions to their independent claims. The descriptions of the patents are sliced into description pieces of a certain length, which should be adopted to the length of the trained or searched claim. In our tests, we sliced the description into pieces with a random length in the range of 100 to 200 tokens. We applied the trained BERT to a FTO patent search or patent analysis for five inventions, which are described in five DOCDB abstracts. The result is difficult to evaluate, since there is neither a standardized data set nor can one clearly assign relevance or non-relevance to each patent. The apparent result is promising, but the method needs to be further evaluated in practice.
The following points could be interesting for future research:
We would like to thank Jochen Spuck, EconSight (Switzerland, Basel) and Carsten Guderian, PatentSight (Germany, Bonn), for their helpful and valuable discussions and support on patent data.