Skip to main content
DennisWa
Jul 3, 2025 4:38 AM UTC
Question

Stopwords Not Working as Expected in Custom Lucene Index (e.g. "are", "was", "that")

Hi Everyone,

I'm currently adding a custom stopwords.txt file to my custom Lucene index in AEM to filter out common stopwords during search. While most of the words in my list are being excluded as expected, I've noticed that some very common ones like "are", "was", and "that" are still being indexed and returned in search results.

My stopwords.txt file includes all of these terms (one per line), and I’ve confirmed the file is correctly referenced in the analyzer configuration for the index.

I’m wondering if anyone else has experienced this issue? Is there anything I might be missing related to:

  • File encoding or formatting of the stopwords.txt file?

  • Analyzer or tokenizer order/configuration?

  • Case sensitivity issues even with ignoreCase = true?

  • Possible overrides by other filters or analyzers?

Any suggestions or shared experiences would be greatly appreciated!

Thanks in advance!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

7 replies

DennisWaAuthor
Jul 3, 2025 4:46 AM UTC

Just several words are still being indexed, the rest of them work as expected! Here is my xml file:

 

 

Translate
    Nilesh_Mali
    Nilesh_Mali
    Level 3
    Jul 3, 2025 5:04 AM UTC

    @denniswa 

     

    Verify your custom index's analyzers and filters in your oak:index definition.

    The StopwordFilterFactory must be correctly placed after any lowercasing if ignoreCase="true".

     

    It should be something like this

    <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" /> </analyzer>

     

    As per your details it looks like you are missing LowerCaseFilterFactory.

    Make sure trigger a reindex after changes.

     

    Translate
      DennisWaAuthor
      Jul 3, 2025 5:21 AM UTC

      Hi Nilesh,

       

      Thanks for replying, I just tried adding the ignoreCase = true but they are still being indexed.

       
      <Stop jcr:primaryType="nt:unstructured" ignoreCase="{Boolean}true" words="stopwords.txt"> <stopwords.txt/> </Stop>
      Translate
      Nilesh_Mali
      Nilesh_Mali
      Level 3
      Jul 3, 2025 6:08 AM UTC

      <WhitespaceTokenizerFactory />

      This tokenizer is very basic — it only splits on whitespace and does not normalize punctuation, possessives, or word boundaries the StandardTokenizerFactory way does.

       

      <tokenizer jcr:primaryType="nt:unstructured" name="Standard"> <StandardTokenizerFactory jcr:primaryType="nt:unstructured"/> </tokenizer>

       

      Update stop filter to

      <Stop jcr:primaryType="nt:unstructured" ignoreCase="{Boolean}true" words="stopwords.txt"> <stopwords.txt/> </Stop>
      Translate
        kautuk_sahni
        kautuk_sahni
        Level 9
        Jul 14, 2025 11:52 AM UTC

        @denniswa Just checking in — were you able to resolve your issue?
        We’d love to hear how things worked out. If the suggestions above helped, marking a response as correct can guide others with similar questions. And if you found another solution, feel free to share it — your insights could really benefit the community. Thanks again for being part of the conversation!

        Translate
        Kautuk Sahni
        PoorvaJa
        Level 2
        Sep 9, 2025 4:04 AM UTC

        Facing a similar issue. Any fixes?

        Translate
        Terms & ConditionsAccessibility statement

        Loading footer...