Chapter 8, Exploring Sentence- and Domain-Specific BERT

1. Sentence-BERT (SBERT) was introduced by the Ubiquitous Knowledge Processing Lab (UKP-TUDA). As the name suggests, SBERT is used to obtain fixed-length sentence representations. SBERT extends the pre-trained BERT model (or its variants) to obtain the sentence representation.

2. If we obtain a sentence representation by applying mean pooling to the representation of all the tokens, then essentially the sentence representation holds the meaning of all the words (tokens), and if we obtain a sentence representation by applying max pooling to the representation of all the tokens, then essentially the sentence representation holds the meaning of important words (tokens).
3. ClinicalBERT is the clinical domain-specific BERT pre-trained on a large clinical corpus. The clinical notes or progress notes contain very useful information about the patient. This includes a record of patient visits, their symptoms, diagnosis, daily activities, observations...