[Tech Preview] Lucene.Net-Powered Dynamo Search

Request for Feedback
1

Dear forum members,

Dynamo team is thrilled to share with you our latest updates in Dynamo Core, featuring new in-canvas search capability built on top of the robust Lucene.Net framework. Search is a daily driver in Dynamo workflows, there are many problems reported with the existing search. A lot of them are captured by you in the other thread by our PM Sol at [Request for Feedback] Searching in Dynamo. In the past, we also see power users trying to fix it in their own ways. It has been too long but we finally started the revamp of search. This is only the beginning to transform the way you search and retrieve information within our product. Allow me to dive into the technical details of what this means.

Our development team has diligently integrated Lucene.Net to provide a robust and scalable search solution that meets the growing demand for performant and accurate information retrieval. Here are some key technical highlights of our new in-canvas search feature:

  1. High-Performance Indexing and Searching: Lucene.Net leverages an inverted index structure that allows for lightning-fast indexing and searching. By tokenizing and normalizing text, Lucene.Net creates an index that enables efficient retrieval of matching documents. Our search feature harnesses this power, delivering near-instantaneous search results, even when dealing with vast amounts of data.
  2. Relevance-Driven Search: Lucene.Net employs advanced scoring algorithms to determine the relevance of documents to a given query. By analyzing factors such as term frequency, inverse document frequency, and document length normalization, Lucene.Net ranks search results based on their relevance. This ensures that users receive accurate and meaningful search results, minimizing irrelevant matches.
  3. Multilingual Support: Lucene.Net supports a wide range of languages, making our search feature language-agnostic. It handles diverse content and linguistic nuances seamlessly, ensuring accurate search results across various languages. This enables users worldwide to benefit from our search functionality, regardless of their preferred language.

What is included in the current beta

  1. OOTB nodes indexing (integration nodes and package nodes support on the way)
    Dynamo loads all the node libraries and indexes all the nodes as part of initialization when you see the following message from Dynamo Splash Screen. It is also one of the reasons why Dynamo launch would take some time to finish until you can see the Dynamo startup page.

    Loading
    Loading923×618 90.4 KB

    Now that we switched to using Lucene.Net, Dynamo indexes nodes’ information to local files instead of in memory for now. After launching the latest 2.19 sandbox, you are expected to see these index files in AppData folder.
    image
    image1038×278 13.7 KB

  2. In-Canvas search was switched to Lucene.Net searcher. Here are some examples of search cases that works better. We have not fully tested all cases yet but would welcome you to report anything you feel is not intuitive.
    a. Search terms with space in them, e.g. list map, curve trim

    List Map
    List Map1510×1139 229 KB

    image
    image1507×1134 234 KB

    b. Search terms with a dot in between, e.g. list.map
    List.Map
    List.Map1515×1137 100 KB

    c. General search results accuracy, e.g. unit
    image
    image1514×1134 120 KB

    d. Search terms with typos, e.g. comprwss
    image
    image1510×1137 109 KB

What’s coming
Starting with in-canvas search, we are gathering feedback, and optimizing how indexing and search until we ensure you are pleased with the performance and accuracy. Meanwhile, we will replace the search component for library search, package search, and other search related features.

We are eager to receive your feedback and suggestions as we continually improve and optimize our search feature. Your insights are invaluable in our quest to enhance performance and usability. To explore the full potential of Dynamo and try the new search out, ensure that you are using our latest 2.19 Dynamo sandbox from 05-26 onwards at https://dynamobuilds.com/.

image
image1061×302 41.9 KB

Thank you for your continued support and engagement within our community. We are committed to delivering innovative solutions, and the integration of Lucene.Net into our search feature is just the beginning of a series of exciting updates and features to come.

Cheers,
Aaron-

13 Likes
7

wah wah
image
image

4 Likes
8

Thank you John, this case is validated.

image
image1510×1135 104 KB

4 Likes
9

Heck yes!

4 Likes
10

First, I wanted to say I’m super excited to test this out. However, I did have a question regarding this comment.

Does this mean the team needs to add other nodes to the index at a later time? Via an update?

I hope not. :pleading_face:

I plan on building core for 2023/24 this next week to test in Revit too. But I wanted to verify that comment first.

3 Likes
11

Hi John,
This only means we have not finished the work to validate all cases supported yet, e.g. integrator specific nodes should work at some cost, our team is testing all integrations now. Another case is when user install a new package with nodes, the new nodes need to be added to the index before the search could take effect. There are other optimization as well, that’s why I am calling this a tech preview with limited scope. But I will be updating users about progress.

Cheers,
Aaron-

3 Likes