sauron development: new file indexing maintaining system

The default behavior can only jump to the import statement, let's say the following:

(require xxx)

(from-xxx ...)

I can only go to the position of xxx. This is not a wanted behavior, I expect it would go to the definition place in another file rather than go to the import place in the current file in the case. But before we go deep, it's time to explain why ensuring semantic might not be a problem.

1. The most used language is racket/base, racket, racket/gui, typed/racket, they are just well supported by DrRacket's tools.
2. Reusing the module system and getting semantic support is possible, in fact, the collection turnstile provide these nicely, you can also expand your language to reuse built-in forms in racket. For example, if your language says var x = 1, expanding it to (define x 1) will get support from DrRacket.
3. If your language really cannot base on these, I have a future proposal to solve it, that's your language can talk to an API about how to jump to definition or anything people need. I will back to this point at guessing class and API.

Back to the problem, I found racket-mode already solve it, so I check their code(below).

(define/override (syncheck:add-arrow/name-dup/pxpy
                  _def-src def-beg def-end _def-px _def-py
                  _use-src use-beg use-end _use-px _use-py
                  _actual? _level require-arrow? _name-dup?)
  (when (and (valid-beg/end? def-beg def-end)
             (valid-beg/end? use-beg use-end))
    ;; Consolidate the add-arrow/name-dup items into a hash table
    ;; with one item per definition. The key is the definition position.
    ;; The value is the set of its uses' positions.
    (hash-update! ht-defs/uses
                  (list (substring code-str def-beg def-end)
                        (match require-arrow?
                          ['module-lang 'module-lang]
                          [#t           'import]
                          [#f           'local])
                        (add1 def-beg)
                        (add1 def-end))
                  (λ (v) (set-add v (list (add1 use-beg)
                                          (add1 use-end))))
                  (set))
    (unless require-arrow?
      (send this syncheck:add-mouse-over-status "" use-beg use-end "defined locally"))))

(define/override (syncheck:add-jump-to-definition _src beg end id-sym path submods)
  ;; - drracket/check-syntax only reports the file, not the
  ;;   position within. We can find that using our def-in-file.
  ;;
  ;; - drracket/check-syntax uses identifier-binding which can't
  ;;   follow some contracting and renaming provides. As a result,
  ;;   the value of the id here can be wrong. For example it will
  ;;   report "provide/contract-id-make-traversal.1" for
  ;;   "make-traversal". When the reported id differs from that in
  ;;   the source text, we report both to try with def-in-file.
  ;;
  ;; However, calling def-in-file here/now for all jumps would be
  ;; quite slow. Futhermore, a user might not actually use the
  ;; jumps -- maybe not any, probably not most, certainly not all.
  ;;
  ;; Sound like a job for a thunk, e.g. racket/promise
  ;; delay/force? We can't marshal a promise between Racket back
  ;; end and Emacs front end. We can do the moral equivalent:
  ;; Simply return the info that the front end should give to the
  ;; "def/drr" command if/as/when needed.
  (when (and (valid-beg/end? beg end) (file-exists? path))
    (define src-str (substring code-str beg end))
    (define drracket-id-str (symbol->string id-sym))
    (interval-map-set! im-jumps beg end
                       (list (path->string path)
                             submods
                             (if (equal? drracket-id-str src-str)
                                 (list drracket-id-str)
                                 (list drracket-id-str src-str))))))

After researching, I found two points

1. syncheck-annotations<%> can build some helpful information as IDE, this is how DrRacket points out errors and provides many hints.
2. based on it building more is possible, racket-mode shows this.

So I create another one in sauron:

(define/override (syncheck:add-arrow/name-dup start-src-obj start-left start-right
                                              end-src-obj end-left end-right
                                              actual?
                                              level
                                              require-arrow?
                                              name-dup?)
  (define id (string->symbol (send text get-text end-left end-right)))
  (unless require-arrow?
    (interval-map-set! bindings
                       end-left
                       (add1 end-right)
                       (binding id start-left start-right #f))))

(define/override (syncheck:add-jump-to-definition source-obj start end id filename submods)
  (interval-map-set! bindings start (add1 end) (binding id #f #f filename)))

It's easy to understand the query flow, for a file, I build an annotation result(also called indexing in many IDE, or record in sauron) just like above. When the editor is asking the definition position of a range or a position, it would get something from the annotation result, due to the code it could be:

1. (binding id start-left start-right #f) when this is a local variable, collected via syncheck:add-arrow/name-dup
2. (binding id #f #f filename) when the identifier is an external one, collected via syncheck:add-jump-to-definition

The local one is boring, just move the cursor to the target range(what DrRacket already does), the external one is the point. Then I have to collect top-level definitions for each file via the below code.

(define/override (syncheck:add-definition-target source-obj start end id mods)
  (log:debug "syncheck:add-definition-target ~a:~a" source-obj id)
  (hash-set! defs id (binding id start end src)))

When an editor just finds it gets (binding id #f #f filename), it will stop and prepare to open another tab for the file, and then ask another record to get the definition of =id=(what above just collected)! The whole flow completes correctly just like racket-mode. Later, I also port this to racket-langserver, this makes all mainstream development tools share the same ability!

• DrRacket: sauron
• VSCode, Vim: racket-langserver
• Emacs: racket-mode

Is time to push our boundaries.

Quickly, I ambitious started to do the reverse, jump to usages from a definition. Unfortunately, this is inefficient in the end because of my implementation. I know I must create something differently from fundamental, the record management needs a new approach. Since per file per record, I quickly decide to use the model per file per thread. The idea was made last year, however, I cannot push it forward till recently.

I dig into the implementation, this section briefly shows what I know from this process.

1. I have a big hashmap for mapping from a file to its record. Thus, although each file gets a thread as an updater, the updater will get blocked on writing the hashmap since the hashmap is thread-safe. Hence, the read-write lock would also block the editor randomly even if the editor just wants to query something.
2. When any file changes, I create updaters for every program file in the project, this is inefficient.

As corresponding, the following are solutions.

1. Rather than store the data in hashmap, store them in the thread! Then the hashmap became from file to the thread, the thread called maintainer now! Now, all querying at least not blocking by any lock, only scheduling problem.
2. With the record maintainer, any changes became sending messages to the corresponding maintainer of a file via thread-send. Simplify the abstraction a lot.

Maintainer HashMap

        │
        │
        │ create
        │                         ┌────────────────┐
        │                         │     Editor     │
  ┌─────▼──────┐                  │                │
  │ maintainer ◄──────────────────┤  current-file  │
  └────────────┘                  │                │
                  send message    └────────────────┘

These all happened in commit 0b1714e45, and amend in commit 930db7f92.

The result is impressive.

1. opening a project only takes at most 2 seconds.
2. UI gets no blocking
3. The memory usage suppresses to less than 0.5 times at peak. Notice this is not an accurate measurement due to I didn't measure in the same version, I remember the peak is over 1000MB in Racket 8.2, but now is around 500MB in Racket 8.5.
4. Collect records and get prepared to respond is faster now

Finally, I have already reported the current situation of sauron, and am ready to show some future.

I show you how sauron came to today, and what's the future. I want to invite you to explore more problems in your daily development and tell in the community, or even contribute your code. Can't wait to see them! Have a nice day~~~