• Corpus ID: 174799315

PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model

@article{Lee2019PatentBERTPC,
  title={PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model},
  author={Jieh-Sheng Lee and Jieh Hsiang},
  journal={ArXiv},
  year={2019},
  volume={abs/1906.02124}
}
In this work we focus on fine-tuning a pre-trained BERT model and applying it to patent classification. When applied to large datasets of over two millions patents, our approach outperforms the state of the art by an approach using CNN with word embeddings. In addition, we focus on patent claims without other parts in patent documents. Our contributions include: (1) a new state-of-the-art method based on pre-trained BERT model and fine-tuning for patent classification, (2) a large dataset USPTO… 
View PDF on arXiv
57 Citations
Highly Influential Citations
4
Background Citations
23
Methods Citations
20
Results Citations
4

Tables from this paper

57 Citations

Evaluating Neural Multi-Field Document Representations for Patent Classification

This work proposes a new architecture combining the neural transformer-based representations of the various fields into a meta-embedding, which is demonstrated to outperform the tf.idf-based counterparts especially on less frequent classes.

PatentNet: multi-label classification of patent documents using deep learning based language understanding

Fine-tuning the pre-trained language models on the patent text improves the multi-label patent classification performance, and it is concluded that XLNet performs the best and achieves a new state-of-the-art classification performance.

CoPatE: A Novel Contrastive Learning Framework for Patent Embeddings

The proposed CoPatE is a novel Contrastive Learning Framework for Patent Embeddings to capture the high-level semantics of the large-scale patents, where a patent semantic compression module learns the informative claims to reduce the computational complexity, and a tags auxiliary learning module is to enhance the semantics of a patent from the structure.

PatentTransformer-1.5: Measuring Patent Claim Generation by Span Relevancy

This work proposes a span-based approach and a generic framework to measure patent claim generation quantitatively, and fine-tune a pre-trained Google BERT model to measure the patent claim spans generated by a fine-tuned OpenAI GPT-2 model.

Patent prior art search using deep learning language model

This paper addressed a method to effectively handle the patent documents using BERT, one of the major deep learning-based language models and proved through experiments that the model had outperformed the conventional approaches and the combinations of the key components with the recall value of up to '94.29%' from the real patent dataset.

Towards Comprehensive Patent Approval Predictions:Beyond Traditional Document Classification

This work proposes a novel framework that unifies the document classifier with handcrafted features, particularly time-dependent novelty scores, and imposes a new regularization term into the classification objective to enforce the monotonic change of approval prediction w.r.t. novelty scores.

Measuring Patent Claim Generation by Span Relevancy

This work proposes a span-based approach and a generic framework to measure patent claim generation quantitatively and shows the effectiveness of the span-pair classifier after fine-tuning the pre-trained model, and validates the quantitative metric of span relevancy in patent claimgeneration.

MEXN: Multi-Stage Extraction Network for Patent Document Classification

A neural network-based document classification for patent documents is proposed by designing a novel multi-stage feature extraction network (MEXN), which comprise of paragraphs encoder and summarizer for all paragraphs.

PatentTransformer: A Framework for Personalized Patent Claim Generation

The PatentTransformer framework to generate and measure personalized patent claims leverages the recent transfer learning in the Deep Learning field, particularly the state-of-the-art Transformer-based models.

PatentTransformer-2: Controlling Patent Text Generation by Structural Metadata

This second version of PatentTransformer, the codename for patent text generation based on Transformer-based models, leverages more of the structural metadata in patents to build a text-to-text generation flow.
...

References

SHOWING 1-10 OF 13 REFERENCES

Patent Classification Experiments with the Linguistic Classification System LCS

It is found that adding full descriptions to abstracts gives a clear improvement; the first 400 words of the description also improves classification but to a lesser degree; the most important finding however is the importance of the threshold on the class selection.

A Hierarchical Feature Extraction Model for Multi-Label Mechanical Patent Classification

A hierarchical feature extraction model (HFEM) is proposed for multi-label mechanical patent classification, which is able to capture both local features of phrases as well as global and temporal semantics.

Universal Language Model Fine-tuning for Text Classification

This work proposes Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduces techniques that are key for fine- Tuning a language model.

Universal Language Model Fine-tuning for Patent Classification

The methods used for the 2018 ALTA Shared Task to automatically classify Australian patents into their main International Patent Classification section used a Support Vector Machine and Universal Language Model with Fine-tuning.

IPC Multi-label Classification Based on the Field Functionality of Patent Documents

It is verified that the technical field and background field play an important role in improving the precision of IPC multi-label classification at the IPC subclass level.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.

Deep Contextualized Word Representations

A new type of deep contextualized word representation is introduced that models both complex characteristics of word use and how these uses vary across linguistic contexts, allowing downstream models to mix different types of semi-supervision signals.

Analysis of the patent documentation coverage of the CPC in comparison with the IPC with a focus on Asian documentation

Quantifying the Challenges in Parsing Patent Claims

The 1st International Workshop on Advances in Patent Information Retrieval (AsPIRe-2010), 28 maart 2010 presents new ideas on how to improve the quality of patent information retrieval in the rapidly changing environment.

DeepPatent: patent classification with convolutional neural networks and word embedding

DeepPatent is proposed, a deep learning algorithm for patent classification based on CNN and word vector embedding that outperformed all the existing algorithms that used the same information for training and achieved a classification precision of 83.98%.