WikiText数据集_自然语言处理
WikiText数据集来自维基百科(Wiki)的词条，经过验证的优质文章内容被收录，总数超过1亿个词条(原词叫token，令牌)。
与Penn Treebank（PTB）的预处理版本相比，WikiText-2要大2倍多，WikiText-103要大110倍多。WikiText数据集还具有更大的词汇表，并保留了原始大小写、标点符号和数字，所有这些都在PTB中删除。由于数据集由完整的文章组成，因此它非常适合可以利用长期依赖关系的模型。
WikiText-2 约 4.3 MB，WikiText-103 约 181 MB。
示例
数据集保存在txt中，大概的格式如下：
= Homarus gammarus =
Homarus gammarus , known as the European lobster or common lobster , is a species of lobster from the eastern Atlantic Ocean , Mediterranean Sea and parts of the Black Sea . It is closely related to the American lobster , H. americanus . It may grow to a length of 60 cm ( 24 in ) and a mass of 6 kilograms ( 13 lb ) , and bears a conspicuous pair of claws . In life , the lobsters are blue , only becoming " lobster red " on cooking . Mating occurs in the summer , producing eggs which are carried by the females for up to a year before hatching into larvae . Homarus gammarus is a highly esteemed food , and is widely caught using lobster pots , mostly around the British Isles .
= = Description = =
Homarus gammarus is a large , with a body length up to 60 centimetres ( 24 in ) and weighing up to 5 – 6 kilograms ( 11 – 13 lb ) , although the lobsters caught in lobster pots are usually 23 – 38 cm ( 9 – 15 in ) long and weigh 0 @.@ 7 – 2 @.@ 2 kg ( 1 @.@ 5 – 4 @.@ 9 lb ) . Like other crustaceans , lobsters have a hard which they must shed in order to grow , in a process called ( ) . This may occur several times a year for young lobsters , but decreases to once every 1 – 2 years for larger animals .
The first pair of is armed with a large , asymmetrical pair of claws . The larger one is the " " , and has rounded used for crushing prey ; the other is the " cutter " , which has sharp inner edges , and is used for holding or tearing the prey . Usually , the left claw is the , and the right is the cutter .
The is generally blue above , with spots that , and yellow below . The red colour associated with lobsters only appears after cooking . This occurs because , in life , the red pigment is bound to a protein complex , but the complex is broken up by the heat of cooking , releasing the red pigment .
The closest relative of H. gammarus is the American lobster , Homarus americanus . The two species are very similar , and can be crossed artificially , although hybrids are unlikely to occur in the wild since their ranges do not overlap . The two species can be distinguished by a number of characteristics :
The of H. americanus bears one or more spines on the underside , which are lacking in H. gammarus .
The spines on the claws of H. americanus are red or red @-@ tipped , while those of H. gammarus are white or white @-@ tipped .
The underside of the claw of H. americanus is orange or red , while that of H. gammarus is creamy white or very pale red .
= = Life cycle = =
Female H. gammarus reach sexual maturity when they have grown to a carapace length of 80 – 85 millimetres ( 3 @.@ 1 – 3 @.@ 3 in ) , whereas males mature at a slightly smaller size . Mating typically occurs in summer between a recently female , whose shell is therefore soft , and a hard @-@ shelled male . The female carries the eggs for up to 12 months , depending on the temperature , attached to her . Females carrying eggs are said to be " " and can be found throughout the year .
The eggs hatch at night , and the larvae swim to the water surface where they drift with the ocean currents , preying on . This stage involves three and lasts for 15 – 35 days . After the third moult , the juvenile takes on a form closer to the adult , and adopts a lifestyle . The juveniles are rarely seen in the wild , and are poorly known , although they are known to be capable of digging extensive burrows . It is estimated that only 1 larva in every 20 @,@ 000 survives to the phase . When they reach a carapace length of 15 mm ( 0 @.@ 59 in ) , the juveniles leave their burrows and start their adult lives . .....
以上例子是词条 = Homarus gammarus =的介绍 （一种龙虾品种）。其也是wiki词条下的一级标题，后面的 = = Description = = 和 = = Life cycle = = 是二级标题，大意为“简述”和“生命周期“。
同理，=== xxx === 这样的格式就是该词条下的三级标题。
标识符
这里需要注意NLP里面常用的两个标识符经常出现
意思是这个单词是低频词，不在统计词频范围内
- @
这个是连接符，比如词条中有one-apple,那么数据库文本中是这样记录 one @-@ apple
下载和参考文献
- https://blog.salesforceairesearch.com/the-wikitext-long-term-dependency-language-modeling-dataset/
- https://paperswithcode.com/dataset/wikitext-2