<legend id='4Ev0d'><style id='4Ev0d'><dir id='4Ev0d'><q id='4Ev0d'></q></dir></style></legend>
<tfoot id='4Ev0d'></tfoot>
        • <bdo id='4Ev0d'></bdo><ul id='4Ev0d'></ul>
        <i id='4Ev0d'><tr id='4Ev0d'><dt id='4Ev0d'><q id='4Ev0d'><span id='4Ev0d'><b id='4Ev0d'><form id='4Ev0d'><ins id='4Ev0d'></ins><ul id='4Ev0d'></ul><sub id='4Ev0d'></sub></form><legend id='4Ev0d'></legend><bdo id='4Ev0d'><pre id='4Ev0d'><center id='4Ev0d'></center></pre></bdo></b><th id='4Ev0d'></th></span></q></dt></tr></i><div id='4Ev0d'><tfoot id='4Ev0d'></tfoot><dl id='4Ev0d'><fieldset id='4Ev0d'></fieldset></dl></div>

      1. <small id='4Ev0d'></small><noframes id='4Ev0d'>

        用 lucene 提取 tf-idf 向量

        时间:2023-09-29

        • <bdo id='AhfvT'></bdo><ul id='AhfvT'></ul>
          <legend id='AhfvT'><style id='AhfvT'><dir id='AhfvT'><q id='AhfvT'></q></dir></style></legend>

              <i id='AhfvT'><tr id='AhfvT'><dt id='AhfvT'><q id='AhfvT'><span id='AhfvT'><b id='AhfvT'><form id='AhfvT'><ins id='AhfvT'></ins><ul id='AhfvT'></ul><sub id='AhfvT'></sub></form><legend id='AhfvT'></legend><bdo id='AhfvT'><pre id='AhfvT'><center id='AhfvT'></center></pre></bdo></b><th id='AhfvT'></th></span></q></dt></tr></i><div id='AhfvT'><tfoot id='AhfvT'></tfoot><dl id='AhfvT'><fieldset id='AhfvT'></fieldset></dl></div>
              <tfoot id='AhfvT'></tfoot>

                    <tbody id='AhfvT'></tbody>

                • <small id='AhfvT'></small><noframes id='AhfvT'>

                  本文介绍了用 lucene 提取 tf-idf 向量的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

                  问题描述

                  我已经使用 lucene 索引了一组文档.我还为每个文档内容存储了 DocumentTermVector.我写了一个程序,得到了每个文档的词频向量,但是如何获取每个文档的 tf-idf 向量呢?

                  I have indexed a set of documents using lucene. I also have stored DocumentTermVector for each document content. I wrote a program and got the term frequency vector for each document, but how can I get tf-idf vector of each document?

                  这是我在每个文档中输出词频的代码:

                  Here is my code that outputs term frequencies in each document:

                  Directory dir = FSDirectory.open(new File(indexDir));
                      IndexReader ir = IndexReader.open(dir);
                      for (int docNum=0; docNum<ir.numDocs(); docNum++) {
                          System.out.println(ir.document(docNum).getField("filename").stringValue());
                          TermFreqVector tfv = ir.getTermFreqVector(docNum, "contents");
                          if (tfv == null) {
                          // ignore empty fields
                          continue;
                          }
                          String terms[] = tfv.getTerms();
                          int termCount = terms.length;
                          int freqs[] = tfv.getTermFrequencies();
                  
                          for (int t=0; t < termCount; t++) {
                          System.out.println(terms[t] + " " + freqs[t]);
                          }
                      }
                  

                  lucene 中是否有任何内置函数可以让我这样做?

                  Is there any buit-in function in lucene for me to do that?

                  没有人帮忙,我自己做了:

                  Nobody helped, and I did it by myself:

                      Directory dir = FSDirectory.open(new File(indexDir));
                      IndexReader ir = IndexReader.open(dir);
                  
                      int docNum;
                      for (docNum = 0; docNum<ir.numDocs(); docNum++) {
                          TermFreqVector tfv = ir.getTermFreqVector(docNum, "title");
                          if (tfv == null) {
                                  // ignore empty fields
                                  continue;
                          }
                          String tterms[] = tfv.getTerms();
                          int termCount = tterms.length;
                          int freqs[] = tfv.getTermFrequencies();
                  
                          for (int t=0; t < termCount; t++) {
                              double idf = ir.numDocs()/ir.docFreq(new Term("title", tterms[t]));
                              System.out.println(tterms[t] + " " + freqs[t]*Math.log(idf));
                          }
                      }
                  

                  有什么方法可以找到每个词条的ID号吗?

                  is there any way to find the ID number of each term?

                  没有人帮忙,我自己又做了一次:

                  Nobody helped, and I did it by myself again:

                      List list = new LinkedList();
                      terms = null;
                      try
                      {
                          terms = ir.terms(new Term("title", ""));
                          while ("title".equals(terms.term().field()))
                          {
                          list.add(terms.term().text());
                          if (!terms.next())
                              break;
                          }
                      }
                      finally
                      {
                          terms.close();
                      }
                      int docNum;
                      for (docNum = 0; docNum<ir.numDocs(); docNum++) {
                          TermFreqVector tfv = ir.getTermFreqVector(docNum, "title");
                          if (tfv == null) {
                                  // ignore empty fields
                                  continue;
                          }
                          String tterms[] = tfv.getTerms();
                          int termCount = tterms.length;
                          int freqs[] = tfv.getTermFrequencies();
                  
                          for (int t=0; t < termCount; t++) {
                              double idf = ir.numDocs()/ir.docFreq(new Term("title", tterms[t]));
                              System.out.println(Collections.binarySearch(list, tterms[t]) + " " + tterms[t] + " " + freqs[t]*Math.log(idf));
                          }
                      }
                  

                  推荐答案

                  你可能找不到 tf-idf 向量.但正如您已经完成的那样,您可以手动计算 IDF.最好使用 DefaultSimilarity(或您使用的任何相似性实现)为您计算它.

                  You'll probably not found a tf-idf vector. But as you've already done, you can calculate IDF by hand. It is probably better to use the DefaultSimilarity (or whatever Similarity implementation you are using) to calculate it for you.

                  关于 Term ID,我认为目前你不能.至少直到 Lucene 4.0,见 这个.

                  Regarding Term ID, I think currently you can't. At least not until Lucene 4.0, see this.

                  这篇关于用 lucene 提取 tf-idf 向量的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持html5模板网!

                  上一篇:MongoDB 是关系 db + lucene 的有效替代品吗? 下一篇:渗透器在弹性搜索中的含义/作用是什么?

                  相关文章

                  最新文章

                  1. <tfoot id='x2EQ2'></tfoot>

                    1. <i id='x2EQ2'><tr id='x2EQ2'><dt id='x2EQ2'><q id='x2EQ2'><span id='x2EQ2'><b id='x2EQ2'><form id='x2EQ2'><ins id='x2EQ2'></ins><ul id='x2EQ2'></ul><sub id='x2EQ2'></sub></form><legend id='x2EQ2'></legend><bdo id='x2EQ2'><pre id='x2EQ2'><center id='x2EQ2'></center></pre></bdo></b><th id='x2EQ2'></th></span></q></dt></tr></i><div id='x2EQ2'><tfoot id='x2EQ2'></tfoot><dl id='x2EQ2'><fieldset id='x2EQ2'></fieldset></dl></div>

                    2. <legend id='x2EQ2'><style id='x2EQ2'><dir id='x2EQ2'><q id='x2EQ2'></q></dir></style></legend>

                      <small id='x2EQ2'></small><noframes id='x2EQ2'>

                        <bdo id='x2EQ2'></bdo><ul id='x2EQ2'></ul>