新闻文本聚类

http://74.125.155.132/scholar?q=cache:x2h4e3WvjZYJ:scholar.google.com/+%E6%B1%89%E8%AF%AD%E6%96%B0%E9%97%BB%E6%8A%A5%E9%81%93%E4%B8%AD%E7%9A%84%E8%AF%9D%E9%A2%98%E8%B7%9F%E8%B8%AA%E4%B8%8E%E7%A0%94%E7%A9%B6&hl=zh-CN&as_sdt=2000

1 在最初的研究阶段（1999 年前），话题与事件含义相同。

2 在目前使用的话题概念要相对宽泛一些，它包括一个核心事件或活动以及所有与之直接相关的事件和活动（A topic is defined to be a seminal event or activity, along with all directly related events and activities）。

3 与话题相应的一个概念是主题（Subject），它的含义更广些。话题与某个具体事件相关，而主题可以涵盖多个类似的具体事件或者根本不涉及任何具体事件。如，“飞机失事”是一个主题，而“2002 年 5 月 7 日北方航空公司一架客机在大连失事”则是一个话题。

4 目前，最成功的报道关系识别系统使用一个余弦相似性计算函数，并用到以下一些资源或技术：停用词，词干分析，二元术语向量，增量修订 TF*IDF 的取值，以及基于时的罚分策略（即如果两篇报道出现的时间间隔越大，那么它们描述同一话题的可能性就越小）。

5 IBM 公司开发的一个相对比较成功的话题识别系统采用了一种两层聚类策略，使用对称的 Okapi 公式来比较两篇报道的相似性。

 

http://74.125.155.132/scholar?q=cache:F-_PID03qqUJ:scholar.google.com/+%E6%B1%89%E8%AF%AD%E6%96%B0%E9%97%BB%E6%8A%A5%E9%81%93%E4%B8%AD%E7%9A%84%E8%AF%9D%E9%A2%98%E8%B7%9F%E8%B8%AA%E4%B8%8E%E7%A0%94%E7%A9%B6&hl=zh-CN&as_sdt=2000

1 TDT会议采用的语料是由会议组织者提供并由语言数据联盟（Linguistic Data Consortium，以下简称LDC）对外发布的TDT系列语料，目前已公开的训练和测试语料包括TDT预研语料（TDT Pilot Corpus）、TDT2和TDT3，这些语料都人工标注了若干话题作为标准答案。

2 从1999年开始，TDT会议引入了对汉语话题的评测。

3 可以看到，话题检测与跟踪和信息抽取研究一样，其建立与发展是以评测驱动的方式进行的。这种评测研究的方法具有以下一些特点：明确的形式化的研究任务、公开的训练与测试数据、公开的评测比较。它将研究置于公共的研究平台上，使得研究之间的比较更加客观，从而让研究者认清各种技术的优劣，起到正确引导研究发展方向的目的。

4 向量空间模型是目前最简便高效的文本表示模型之一。其基本思想是：给定一自然语言文档D＝D（t1，w1；t2，w2；…；tN，wN），其中ti是从文档D中选出的特征项，wi是项的权重，1≤i≤N。在实际的参评系统中，基本上都以词作为文本特征项。特征（词）加权采用的是IR系统中常用的tf*idf加权策略。

<!-- /* Font Definitions */ @font-face {font-family:宋体; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-alt:SimSun; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 135135232 16 0 262145 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-1610611985 1107304683 0 0 159 0;} @font-face {font-family:"/@宋体"; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 135135232 16 0 262145 0;} @font-face {font-family:"Liberation Serif"; mso-font-alt:"Times New Roman"; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} @font-face {font-family:"DejaVu LGC Sans"; mso-font-alt:"Times New Roman"; mso-font-charset:0; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0cm; margin-bottom:.0001pt; text-align:justify; text-justify:inter-ideograph; mso-pagination:none; mso-hyphenate:none; text-autospace:ideograph-other; font-size:12.0pt; font-family:"Liberation Serif","serif"; mso-fareast-font-family:宋体; mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:"DejaVu LGC Sans"; mso-font-kerning:1.5pt; mso-bidi-language:HI;} .MsoChpDefault {mso-style-type:export-only; mso-default-props:yes; font-size:12.0pt; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; mso-ascii-font-family:"Liberation Serif"; mso-hansi-font-family:"Liberation Serif"; mso-bidi-font-family:"DejaVu LGC Sans"; mso-font-kerning:1.5pt; mso-bidi-language:HI;} .MsoPapDefault {mso-style-type:export-only; mso-pagination:none; mso-hyphenate:none; text-autospace:ideograph-other;} /* Page Definitions */ @page {mso-page-border-surround-header:no; mso-page-border-surround-footer:no;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 90.0pt 72.0pt 90.0pt; mso-header-margin:36.0pt; mso-footer-margin:36.0pt; mso-paper-source:0;} div.WordSection1 {page:WordSection1;} -->   向量空间模型和中心向量模型通常采用 cosine 公式来计算报道－话题的相似度。 <!-- /* Font Definitions */ @font-face {font-family:宋体; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-alt:SimSun; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 135135232 16 0 262145 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-1610611985 1107304683 0 0 159 0;} @font-face {font-family:"/@宋体"; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 135135232 16 0 262145 0;} @font-face {font-family:"宋体/, SimSun"; mso-font-alt:"Times New Roman"; mso-font-charset:0; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} @font-face {font-family:"Liberation Serif"; mso-font-alt:"Times New Roman"; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} @font-face {font-family:"DejaVu LGC Sans"; mso-font-alt:"Times New Roman"; mso-font-charset:0; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0cm; margin-bottom:.0001pt; text-align:justify; text-justify:inter-ideograph; mso-pagination:none; mso-hyphenate:none; text-autospace:ideograph-other; font-size:12.0pt; font-family:"Liberation Serif","serif"; mso-fareast-font-family:宋体; mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:"DejaVu LGC Sans"; mso-font-kerning:1.5pt; mso-bidi-language:HI;} p.Standard, li.Standard, div.Standard {mso-style-name:Standard; mso-style-unhide:no; mso-style-parent:""; margin:0cm; margin-bottom:.0001pt; text-align:justify; text-justify:inter-ideograph; mso-pagination:none; mso-hyphenate:none; text-autospace:ideograph-other; font-size:10.5pt; mso-bidi-font-size:12.0pt; font-family:"Times New Roman","serif"; mso-fareast-font-family:"宋体/, SimSun"; mso-font-kerning:1.5pt;} .MsoChpDefault {mso-style-type:export-only; mso-default-props:yes; font-size:12.0pt; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; mso-ascii-font-family:"Liberation Serif"; mso-hansi-font-family:"Liberation Serif"; mso-bidi-font-family:"DejaVu LGC Sans"; mso-font-kerning:1.5pt; mso-bidi-language:HI;} .MsoPapDefault {mso-style-type:export-only; mso-pagination:none; mso-hyphenate:none; text-autospace:ideograph-other;} /* Page Definitions */ @page {mso-page-border-surround-header:no; mso-page-border-surround-footer:no;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 90.0pt 72.0pt 90.0pt; mso-header-margin:36.0pt; mso-footer-margin:36.0pt; mso-paper-source:0;} div.WordSection1 {page:WordSection1;} -->

  近来有些系统开始尝试用 OKAPI 公式来计算报道 - 话题相似度，其形式是：

       Ok(d¹ ,d² ;cl) =

<!-- /* Font Definitions */ @font-face {font-family:宋体; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-alt:SimSun; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 135135232 16 0 262145 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-1610611985 1107304683 0 0 159 0;} @font-face {font-family:"/@宋体"; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 135135232 16 0 262145 0;} @font-face {font-family:"宋体/, SimSun"; mso-font-alt:"Times New Roman"; mso-font-charset:0; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} @font-face {font-family:"Liberation Serif"; mso-font-alt:"Times New Roman"; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} @font-face {font-family:"DejaVu LGC Sans"; mso-font-alt:"Times New Roman"; mso-font-charset:0; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0cm; margin-bottom:.0001pt; text-align:justify; text-justify:inter-ideograph; mso-pagination:none; mso-hyphenate:none; text-autospace:ideograph-other; font-size:12.0pt; font-family:"Liberation Serif","serif"; mso-fareast-font-family:宋体; mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:"DejaVu LGC Sans"; mso-font-kerning:1.5pt; mso-bidi-language:HI;} p.Standard, li.Standard, div.Standard {mso-style-name:Standard; mso-style-unhide:no; mso-style-parent:""; margin:0cm; margin-bottom:.0001pt; text-align:justify; text-justify:inter-ideograph; mso-pagination:none; mso-hyphenate:none; text-autospace:ideograph-other; font-size:10.5pt; mso-bidi-font-size:12.0pt; font-family:"Times New Roman","serif"; mso-fareast-font-family:"宋体/, SimSun"; mso-font-kerning:1.5pt;} .MsoChpDefault {mso-style-type:export-only; mso-default-props:yes; font-size:12.0pt; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; mso-ascii-font-family:"Liberation Serif"; mso-hansi-font-family:"Liberation Serif"; mso-bidi-font-family:"DejaVu LGC Sans"; mso-font-kerning:1.5pt; mso-bidi-language:HI;} .MsoPapDefault {mso-style-type:export-only; mso-pagination:none; mso-hyphenate:none; text-autospace:ideograph-other;} /* Page Definitions */ @page {mso-page-border-surround-header:no; mso-page-border-surround-footer:no;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 90.0pt 72.0pt 90.0pt; mso-header-margin:36.0pt; mso-footer-margin:36.0pt; mso-paper-source:0;} div.WordSection1 {page:WordSection1;} -->

所得结果表示文档和文档之间的距离，其中 d1,d2 是两个文档， cl 是 d1,d2 中较早出现的那个文档所属的话题。 是词 w 在文档 i 中调整后的词频，对其进行归一化处理使得 独立于 的长度， idf(w) 是词 w 的倒文档频率， 是包含词 w 的文档数目， 是话题 cl 中文档的数目， 是话题 cl 中包含词 w 的文档的数目， λ 是控制词的权值中和话题相关的那部分 “ 动态权值 ” 的可调参数。

文档和话题之间的分数是一个平均值：

       Ok(d,cl) = |cl|^-1 。