commit a983e6970814962a219566cee635b720ad88c64b
parent 1f2883e879761424151ed1e55743c1655b39a0df
Author: Justin M. Keyes <justinkz@gmail.com>
Date: Thu, 23 Oct 2025 17:03:16 +0200
docs: dev_tools, dev_arch
Diffstat:
5 files changed, 607 insertions(+), 704 deletions(-)
diff --git a/runtime/doc/debug.txt b/runtime/doc/debug.txt
@@ -1,163 +0,0 @@
-*debug.txt* Nvim
-
-
- VIM REFERENCE MANUAL by Bram Moolenaar
-
-
-Debugging Vim *debug-vim*
-
-This is for debugging Vim itself, when it doesn't work properly.
-For debugging Vim scripts, functions, etc. see |debug-scripts|
-
- Type |gO| to see the table of contents.
-
-==============================================================================
-
-1. Location of a crash, using gcc and gdb *debug-gcc* *gdb*
-
-When Vim crashes in one of the test files, and you are using gcc for
-compilation, here is what you can do to find out exactly where Vim crashes.
-This also applies when using the MingW tools.
-
-1. Compile Vim with the "-g" option (there is a line in the src/Makefile for
- this, which you can uncomment). Also make sure "strip" is disabled (do not
- install it, or use the line "STRIP = /bin/true").
-
-2. Execute these commands (replace "11" with the test that fails): >
- cd testdir
- gdb ../vim
- run -u unix.vim -U NONE -s dotest.in test11.in
-
-3. Check where Vim crashes, gdb should give a message for this.
-
-4. Get a stack trace from gdb with this command: >
- where
-< You can check out different places in the stack trace with: >
- frame 3
-< Replace "3" with one of the numbers in the stack trace.
-
-==============================================================================
-
-2. Locating memory leaks *debug-leaks* *valgrind*
-
-If you suspect Vim is leaking memory and you are using Linux, the valgrind
-tool is very useful to pinpoint memory leaks.
-
-First of all, build Vim with EXITFREE defined. Search for this in MAKEFILE
-and uncomment the line.
-
-Use this command to start Vim:
->
- valgrind --log-file=valgrind.log --leak-check=full ./vim
-
-Note: Vim will run much slower. If your vimrc is big or you have several
-plugins you need to be patient for startup, or run with the "-u NONE"
-argument.
-
-There are often a few leaks from libraries, such as getpwuid() and
-XtVaAppCreateShell(). Those are unavoidable. The number of bytes should be
-very small a Kbyte or less.
-
-==============================================================================
-
-3. Windows Bug Reporting *debug-win32*
-
-If the Windows version of Vim crashes in a reproducible manner, you can take
-some steps to provide a useful bug report.
-
-
-3.1 GENERIC ~
-
-You must obtain the debugger symbols (PDB) file for your executable: gvim.pdb
-for gvim.exe, or vim.pdb for vim.exe. The PDB should be available from the
-same place that you obtained the executable. Be sure to use the PDB that
-matches the EXE (same date).
-
-If you built the executable yourself with the Microsoft Visual C++ compiler,
-then the PDB was built with the EXE.
-
-If you have Visual Studio, use that instead of the VC Toolkit and WinDbg.
-
-For other compilers, you should always use the corresponding debugger: gdb
-(see above |debug-gcc|) for the Cygwin and MinGW compilers.
-
-
- *debug-vs2005*
-3.2 Debugging Vim crashes with Visual Studio 2005/Visual C++ 2005 Express ~
-
-First launch vim.exe or gvim.exe and then launch Visual Studio. (If you don't
-have Visual Studio, follow the instructions at |get-ms-debuggers| to obtain a
-free copy of Visual C++ 2005 Express Edition.)
-
-On the Tools menu, click Attach to Process. Choose the Vim process.
-
-In Vim, reproduce the crash. A dialog will appear in Visual Studio, telling
-you about the unhandled exception in the Vim process. Click Break to break
-into the process.
-
-Visual Studio will pop up another dialog, telling you that no symbols are
-loaded and that the source code cannot be displayed. Click OK.
-
-Several windows will open. Right-click in the Call Stack window. Choose Load
-Symbols. The Find Symbols dialog will open, looking for (g)vim.pdb. Navigate
-to the directory where you have the PDB file and click Open.
-
-At this point, you should have a full call stack with vim function names and
-line numbers. Double-click one of the lines and the Find Source dialog will
-appear. Navigate to the directory where the Vim source is (if you have it.)
-
-If you don't know how to debug this any further, follow the instructions
-at ":help bug-report". Paste the call stack into the bug report.
-
-If you have a non-free version of Visual Studio, you can save a minidump via
-the Debug menu and send it with the bug report. A minidump is a small file
-(<100KB), which contains information about the state of your process.
-Visual C++ 2005 Express Edition cannot save minidumps and it cannot be
-installed as a just-in-time debugger. Use WinDbg, |debug-windbg|, if you
-need to save minidumps or you want a just-in-time (postmortem) debugger.
-
- *debug-windbg*
-3.3 Debugging Vim crashes with WinDbg ~
-
-See |get-ms-debuggers| to obtain a copy of WinDbg.
-
-As with the Visual Studio IDE, you can attach WinDbg to a running Vim process.
-You can also have your system automatically invoke WinDbg as a postmortem
-debugger. To set WinDbg as your postmortem debugger, run "windbg -I".
-
-To attach WinDbg to a running Vim process, launch WinDbg. On the File menu,
-choose Attach to a Process. Select the Vim process and click OK.
-
-At this point, choose Symbol File Path on the File menu, and add the folder
-containing your Vim PDB to the sympath. If you have Vim source available,
-use Source File Path on the File menu. You can now open source files in
-WinDbg and set breakpoints, if you like. Reproduce your crash. WinDbg should
-open the source file at the point of the crash. Using the View menu, you can
-examine the call stack, local variables, watch windows, and so on.
-
-If WinDbg is your postmortem debugger, you do not need to attach WinDbg to
-your Vim process. Simply reproduce the crash and WinDbg will launch
-automatically. As above, set the Symbol File Path and the Source File Path.
-
-To save a minidump, type the following at the WinDbg command line: >
- .dump vim.dmp
-<
- *debug-minidump*
-3.4 Opening a Minidump ~
-
-If you have a minidump file, you can open it in Visual Studio or in WinDbg.
-
-In Visual Studio 2005: on the File menu, choose Open, then Project/Solution.
-Navigate to the .dmp file and open it. Now press F5 to invoke the debugger.
-Follow the instructions in |debug-vs2005| to set the Symbol File Path.
-
-In WinDbg: choose Open Crash Dump on the File menu. Follow the instructions
-in |debug-windbg| to set the Symbol File Path.
-
- *get-ms-debuggers*
-3.5 Obtaining Microsoft Debugging Tools ~
-
-Visual Studio 2017 Community Edition can be downloaded for free from:
- https://visualstudio.microsoft.com/downloads/
-
- vim:tw=78:ts=8:noet:ft=help:norl:
diff --git a/runtime/doc/dev_arch.txt b/runtime/doc/dev_arch.txt
@@ -6,34 +6,67 @@
How to develop Nvim, explanation of modules and subsystems *dev-arch*
-The top of each major module has (or should have) an overview in a comment at
-the top of its file. The purpose of this document is to give:
+Module-specific details are documented at the top of each module
+(`terminal.c`, `undo.c`, …). The top of each major module has (or should have)
+an overview in a comment at the top of its file.
+
+The purpose of this document is to give:
1. an overview of how it all fits together
2. how-to guides for common tasks such as:
- - deprecating public functions
- - adding a new public (API) function
- - adding a new public (UI) event
-3. TODO: move src/nvim/README.md into this doc.
+ - (TODO) deprecating public functions
+ - (TODO) adding a new public (API) function or (UI) event
Type |gO| to see the table of contents.
==============================================================================
+Filename conventions
+
+The source filenames use extensions to hint about their purpose.
+
+- `*.c`, `*.generated.c` - full C files, with all includes, etc.
+- `*.c.h` - parametrized C files, contain all necessary includes, but require
+ defining macros before actually using. Example: `typval_encode.c.h`
+- `*.h` - full headers, with all includes. Does *not* apply to `*.generated.h`.
+- `*.h.generated.h` - exported functions’ declarations.
+- `*.c.generated.h` - static functions’ declarations.
+
+==============================================================================
Data structures
+- StringBuilder
+- kvec or garray.c for dynamic lists / vectors (use StringBuilder for strings)
+
Use `kvec.h` for most lists. When you absolutely need a linked list, use
`lib/queue_defs.h` which defines an "intrusive" linked list.
+Buffer text is stored as a tree of line segments, defined in `src/nvim/memline.c`.
+The central idea is found in `ml_find_line`.
+
+Many of the editor concepts are defined as Lua data files:
+
+- Events (autocmds): src/nvim/auevents.lua
+- Ex (cmdline) commands: src/nvim/ex_cmds.lua
+- Options: src/nvim/options.lua
+- Vimscript functions: src/nvim/eval.lua
+- v: variables: src/nvim/vvars.lua
+
==============================================================================
-Events
+Events *dev-events*
-All new events must be implemented using `aucmd_defer()` (and where possible,
-old events should be migrated to this), so that they are processed in
-a predictable manner, which avoids crashes and race conditions. See
+The events historically called "autocmds", referred to here as "editor events"
+or simply "events", are high-level events for use by plugins, user config, and
+the Nvim editor. (There is an unrelated, low-level concept defined by the
+`event/defs.h#Event` struct, which is just a bag of data passed along the
+internal |event-loop|.)
+
+All new editor events must be implemented using `aucmd_defer()` (and where
+possible, old events should be migrated to this), so that they are processed
+in a predictable manner, which avoids crashes and race conditions. See
`do_markset_autocmd` for an example.
==============================================================================
-UI events
+UI events *dev-ui-events*
The long-term vision is that UI events are just another type of "editor event"
(formerly known as "autocmds"). There is no real reason that we have separate
@@ -92,5 +125,268 @@ expression evaluation.
==============================================================================
+The event-loop *event-loop*
+
+The internal, low-level, libuv event-loop (|luv-event-loop|) is used to
+schedule arbitrary work in a predictable way. One such obvious use-case for
+scheduling is deferred editor-events (autocmds). Another example is
+|job-control|.
+
+ASYNC EVENT SUPPORT
+
+One of the features Nvim added is the support for handling arbitrary
+asynchronous events, which can include:
+
+- RPC requests
+- job control callbacks
+- timers
+
+Nvim implements this functionality by entering another event loop while
+waiting for characters, so instead of: >py
+
+ def state_enter(state_callback, data):
+ do
+ key = readkey() # read a key from the user
+ while state_callback(data, key) # invoke the callback for the current state
+<
+
+The Nvim program loop is more like: >py
+
+ def state_enter(state_callback, data):
+ do
+ event = read_next_event() # read an event from the operating system
+ while state_callback(data, event) # invoke the callback for the current state
+<
+
+where `event` is something the operating system delivers to us, including (but
+not limited to) user input. The `read_next_event()` part is internally
+implemented by libuv, the platform layer used by Nvim.
+
+Since Nvim inherited its code from Vim, the states are not prepared to receive
+"arbitrary events", so we use a special key to represent those (When a state
+receives an "arbitrary event", it normally doesn't do anything other than
+update the screen).
+
+MAIN LOOP
+
+The `Loop` structure (which describes `main_loop`) abstracts multiple queues
+into one loop: >
+
+ uv_loop_t uv;
+ MultiQueue *events;
+ MultiQueue *thread_events;
+ MultiQueue *fast_events;
+
+`loop_poll_events` checks `Loop.uv` and `Loop.fast_events` whenever Nvim is
+idle, and also at `os_breakcheck` intervals.
+
+MultiQueue is cool because you can attach throw-away "child queues" trivially.
+For example `do_os_system()` does this (for every spawned process!) to
+automatically route events onto the `main_loop`: >
+
+ Process *proc = &uvproc.process;
+ MultiQueue *events = multiqueue_new_child(main_loop.events);
+ proc->events = events;
+
+
+NVIM LIFECYCLE
+
+How Nvim processes input.
+
+Consider a typical Vim-like editing session:
+
+01. Vim displays the welcome screen
+02. User types: `:`
+03. Vim enters command-line mode
+04. User types: `edit README.txt<CR>`
+05. Vim opens the file and returns to normal mode
+06. User types: `G`
+07. Vim navigates to the end of the file
+09. User types: `5`
+10. Vim enters count-pending mode
+11. User types: `d`
+12. Vim enters operator-pending mode
+13. User types: `w`
+14. Vim deletes 5 words
+15. User types: `g`
+16. Vim enters the "g command mode"
+17. User types: `g`
+18. Vim goes to the beginning of the file
+19. User types: `i`
+20. Vim enters insert mode
+21. User types: `word<ESC>`
+22. Vim inserts "word" at the beginning and returns to normal mode
+
+Note that we split user actions into sequences of inputs that change the state
+of the editor. While there's no documentation about a "g command mode" (step
+16), internally it is implemented similarly to "operator-pending mode".
+
+From this we can see that Vim has the behavior of an input-driven state machine
+(more specifically, a pushdown automaton since it requires a stack for
+transitioning back from states). Assuming each state has a callback responsible
+for handling keys, this pseudocode represents the main program loop: >py
+
+ def state_enter(state_callback, data):
+ do
+ key = readkey() # read a key from the user
+ while state_callback(data, key) # invoke the callback for the current state
+<
+
+That is, each state is entered by calling `state_enter` and passing a
+state-specific callback and data. Here is a high-level pseudocode for a program
+that implements something like the workflow described above: >py
+
+ def main()
+ state_enter(normal_state, {}):
+
+ def normal_state(data, key):
+ if key == ':':
+ state_enter(command_line_state, {})
+ elif key == 'i':
+ state_enter(insert_state, {})
+ elif key == 'd':
+ state_enter(delete_operator_state, {})
+ elif key == 'g':
+ state_enter(g_command_state, {})
+ elif is_number(key):
+ state_enter(get_operator_count_state, {'count': key})
+ elif key == 'G'
+ jump_to_eof()
+ return true
+
+ def command_line_state(data, key):
+ if key == '<cr>':
+ if data['input']:
+ execute_ex_command(data['input'])
+ return false
+ elif key == '<esc>'
+ return false
+
+ if not data['input']:
+ data['input'] = ''
+
+ data['input'] += key
+ return true
+
+ def delete_operator_state(data, key):
+ count = data['count'] or 1
+ if key == 'w':
+ delete_word(count)
+ elif key == '$':
+ delete_to_eol(count)
+ return false # return to normal mode
+
+ def g_command_state(data, key):
+ if key == 'g':
+ go_top()
+ elif key == 'v':
+ reselect()
+ return false # return to normal mode
+
+ def get_operator_count_state(data, key):
+ if is_number(key):
+ data['count'] += key
+ return true
+ unshift_key(key) # return key to the input buffer
+ state_enter(delete_operator_state, data)
+ return false
+
+ def insert_state(data, key):
+ if key == '<esc>':
+ return false # exit insert mode
+ self_insert(key)
+ return true
+<
+
+The above gives an idea of how Nvim is organized internally. Some states like
+the `g_command_state` or `get_operator_count_state` do not have a dedicated
+`state_enter` callback, but are implicitly embedded into other states (this
+will change later as we continue the refactoring effort). To start reading the
+actual code, here's the recommended order:
+
+1. `state_enter()` function (state.c). This is the actual program loop,
+ note that a `VimState` structure is used, which contains function pointers
+ for the callback and state data.
+2. `main()` function (main.c). After all startup, `normal_enter` is called
+ at the end of function to enter normal mode.
+3. `normal_enter()` function (normal.c) is a small wrapper for setting
+ up the NormalState structure and calling `state_enter`.
+4. `normal_check()` function (normal.c) is called before each iteration of
+ normal mode.
+5. `normal_execute()` function (normal.c) is called when a key is read in normal
+ mode.
+
+The basic structure described for normal mode in 3, 4 and 5 is used for other
+modes managed by the `state_enter` loop:
+
+- command-line mode: `command_line_{enter,check,execute}()`(`ex_getln.c`)
+- insert mode: `insert_{enter,check,execute}()`(`edit.c`)
+- terminal mode: `terminal_{enter,execute}()`(`terminal.c`)
+
+IMPORTANT VARIABLES
+
+The current mode is stored in `State`. The values it can have are `MODE_NORMAL`,
+`MODE_INSERT`, `MODE_CMDLINE`, and a few others.
+
+The current window is `curwin`. The current buffer is `curbuf`. These point
+to structures with the cursor position in the window, option values, the file
+name, etc.
+
+All the global variables are declared in `globals.h`.
+
+THE MAIN EVENT-LOOP
+
+The main loop is implemented in state_enter. The basic idea is that Vim waits
+for the user to type a character and processes it until another character is
+needed. Thus there are several places where Vim waits for a character to be
+typed. The `vgetc()` function is used for this. It also handles mapping.
+
+What we consider the "Nvim event loop" is actually a wrapper around `uv_run` to
+handle both the `fast_events` queue and possibly (a suitable subset of) deferred
+events. Therefore "raw" `vim.uv.run()` is often not enough to "yield" from Lua
+plugins; instead they can call `vim.wait(0)`.
+
+Updating the screen is mostly postponed until a command or a sequence of
+commands has finished. The work is done by `update_screen()`, which calls
+`win_update()` for every window, which calls `win_line()` for every line.
+See the start of [drawscreen.c](drawscreen.c) for more explanations.
+
+COMMAND-LINE MODE
+
+When typing a `:`, `normal_cmd()` will call `getcmdline()` to obtain a line with
+an Ex command. `getcmdline()` calls a loop that will handle each typed
+character. It returns when hitting `<CR>` or `<Esc>` or some other character that
+ends the command line mode.
+
+EX COMMANDS
+
+Ex commands are handled by the function `do_cmdline()`. It does the generic
+parsing of the `:` command line and calls `do_one_cmd()` for each separate
+command. It also takes care of while loops.
+
+`do_one_cmd()` parses the range and generic arguments and puts them in the
+exarg_t and passes it to the function that handles the command.
+
+The `:` commands are listed in [ex_cmds.lua](ex_cmds.lua).
+
+NORMAL MODE COMMANDS
+
+The Normal mode commands are handled by the `normal_cmd()` function. It also
+handles the optional count and an extra character for some commands. These
+are passed in a `cmdarg_T` to the function that handles the command.
+
+There is a table `nv_cmds` in [normal.c](normal.c) which
+lists the first character of every
+command. The second entry of each item is the name of the function that
+handles the command.
+
+INSERT MODE COMMANDS
+
+When doing an `i` or `a` command, `normal_cmd()` will call the `edit()` function.
+It contains a loop that waits for the next character and handles it. It
+returns when leaving Insert mode.
+
+
+==============================================================================
vim:tw=78:ts=8:sw=4:et:ft=help:norl:
diff --git a/runtime/doc/dev_tools.txt b/runtime/doc/dev_tools.txt
@@ -6,15 +6,221 @@
Tools and techniques for developing Nvim *dev-tools*
-The following advice is helpful when working on or debugging issues with Nvim
-itself.
-
-TODO: merge |debug.txt| into here.
+This is for developing or debugging Nvim itself.
Type |gO| to see the table of contents.
==============================================================================
-Backtraces *dev-tools-backtrace*
+Logs *dev-tools-logs*
+
+Low-level log messages sink to `$NVIM_LOG_FILE`.
+
+UI events are logged at DEBUG level. >
+
+ rm -rf build/
+ make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG"
+
+Use `LOG_CALLSTACK()` (Linux only) to log the current stacktrace. To log to an
+alternate file (e.g. stderr) use `LOG_CALLSTACK_TO_FILE(FILE*)`. Requires
+`-no-pie` ([ref](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860394#15)): >
+
+ rm -rf build/
+ make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG -DCMAKE_C_FLAGS=-no-pie"
+
+Many log messages have a shared prefix, such as "UI" or "RPC". Use the shell to
+filter the log, e.g. at DEBUG level you might want to exclude UI messages: >
+
+ tail -F ~/.local/state/nvim/log | cat -v | stdbuf -o0 grep -v UI | stdbuf -o0 tee -a log
+
+
+==============================================================================
+Reproducible build
+
+To make a reproducible build of Nvim, set cmake variable `LUA_GEN_PRG` to
+a LuaJIT binary built with `LUAJIT_SECURITY_PRN=0`. See commit
+cb757f2663e6950e655c6306d713338dfa66b18d.
+
+
+==============================================================================
+Debug TUI *dev-tools-tui*
+
+TUI INSPECT
+
+Use the Ghostty https://ghostty.org/ inspector tool to observe and query the
+output and events from any terminal application such as Nvim.
+
+TERMINFO LOGGING
+
+At 'verbose' level 3, Nvim logs its internal terminfo state, so you can see
+exactly what terminfo values it is using on the current system. >
+
+ nvim -V3log
+
+TUI DEBUGGING WITH GDB LLDB
+
+Launching the Nvim TUI involves two processes, one for main editor state and one
+for rendering the TUI. Both of these processes use the nvim binary, so somewhat
+confusingly setting a breakpoint in either will generally succeed but may not be
+hit depending on which process the breakpoints were set in.
+
+To debug the main process, you can debug the nvim binary with the `--headless`
+flag which does not launch the TUI and will allow you to set breakpoints in code
+not related to TUI rendering like so: >
+
+ lldb -- ./build/bin/nvim --headless --listen ~/.cache/nvim/debug-server.pipe
+
+While in lldb, enter `run`. You can then attach to the headless process in a
+new terminal window to interact with the editor like so: >
+
+ ./build/bin/nvim --remote-ui --server ~/.cache/nvim/debug-server.pipe
+
+Conversely for debugging TUI rendering, you can start a headless process and
+debug the remote-ui process multiple times without losing editor state.
+
+For details on using nvim-dap and automatically debugging the child (main)
+process, see [here](https://zignar.net/2023/02/17/debugging-neovim-with-neovim-and-nvim-dap/)
+
+TUI REDRAW
+
+For debugging Nvim TUI redraw behavior it is sometimes useful to slow down its
+redraws. Set the 'writedelay' and 'redrawdebug' options to see where and when
+the UI is painted. >
+
+ :set writedelay=50 rdb=compositor
+
+Note: Nvim uses an internal screenbuffer to only send minimal updates even if a large
+region is repainted internally. To also highlight excess internal redraws, use >
+
+ :set writedelay=50 rdb=compositor,nodelta
+
+TUI TRACE
+
+In the rare case that you want to collect a trace of terminal output, the
+ancient `script` command is still the "state of the art". The libvterm
+`vterm-dump` utility formats the result for human-readability.
+
+Record a Nvim terminal session and format it with `vterm-dump`: >sh
+
+ script foo
+ ./build/bin/nvim -u NONE
+ # Exit the script session with CTRL-d
+
+ # Use `vterm-dump` utility to format the result.
+ ./.deps/usr/bin/vterm-dump foo > bar
+
+Then you can compare `bar` with another session, to debug TUI behavior.
+
+TERMINAL REFERENCE
+
+- `man terminfo`
+- https://github.com/vim/vim/blob/0124320c97b0fbbb44613f42fc1c34fee6181fc8/src/libvterm/doc/seqs.txt
+- https://invisible-island.net/xterm/ctlseqs/ctlseqs.html
+
+==============================================================================
+Debug Performance *dev-tools-perf*
+
+PROFILING (EASY)
+
+For debugging performance bottlenecks in any code, there is a simple (and very
+effective) approach:
+
+1. Run the slow code in a loop.
+2. Break execution ~5 times and save the stacktrace.
+3. The cause of the bottleneck will (almost always) appear in most of the stacktraces.
+
+
+PROFILING (FANCY)
+
+For more advanced profiling, consider `perf` + `flamegraph`.
+
+
+USDT PROFILING (POWERFUL)
+
+Or you can use USDT probes via `NVIM_PROBE` ([#12036](https://github.com/neovim/neovim/pull/12036)).
+
+> USDT is basically a way to define stable probe points in userland binaries.
+> The benefit of bcc is the ability to define logic to go along with the probe
+> points.
+
+Tools:
+- bpftrace provides an awk-like language to the kernel bytecode, BPF.
+- BCC provides a subset of C. Provides more complex logic than bpftrace, but takes a bit more effort.
+
+Example using bpftrace to track slow vim functions, and print out any files
+that were opened during the trace. At the end, it prints a histogram of
+function timing: >
+
+ #!/usr/bin/env bpftrace
+
+ BEGIN {
+ @depth = -1;
+ }
+
+ tracepoint:sched:sched_process_fork /@pidmap[args->parent_pid]/ {
+ @pidmap[args->child_pid] = 1;
+ }
+
+ tracepoint:sched:sched_process_exit /@pidmap[args->pid]/ {
+ delete(@pidmap[args->pid]);
+ }
+
+ usdt:build/bin/nvim:neovim:eval__call_func__entry {
+ @pidmap[pid] = 1;
+ @depth++;
+ @funcentry[@depth] = nsecs;
+ }
+
+ usdt:build/bin/nvim:neovim:eval__call_func__return {
+ $func = str(arg0);
+ $msecs = (nsecs - @funcentry[@depth]) / 1000000;
+
+ @time_histo = hist($msecs);
+
+ if ($msecs >= 1000) {
+ printf("%u ms for %s\n", $msecs, $func);
+ print(@files);
+ }
+
+ clear(@files);
+ delete(@funcentry[@depth]);
+ @depth--;
+ }
+
+ tracepoint:syscalls:sys_enter_open,
+ tracepoint:syscalls:sys_enter_openat {
+ if (@pidmap[pid] == 1 && @depth >= 0) {
+ @files[str(args->filename)] = count();
+ }
+ }
+
+ END {
+ clear(@depth);
+ }
+
+ $ sudo bpftrace funcslower.bt
+ 1527 ms for Slower
+ @files[/usr/lib/libstdc++.so.6]: 2
+ @files[/etc/fish/config.fish]: 2
+ <snip>
+
+ ^C
+ @time_histo:
+ [0] 71430 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+ [1] 346 | |
+ [2, 4) 208 | |
+ [4, 8) 91 | |
+ [8, 16) 22 | |
+ [16, 32) 85 | |
+ [32, 64) 7 | |
+ [64, 128) 0 | |
+ [128, 256) 0 | |
+ [256, 512) 6 | |
+ [512, 1K) 1 | |
+ [1K, 2K) 5 | |
+<
+
+==============================================================================
+Backtraces *dev-tools-backtrace*
LINUX
@@ -83,6 +289,53 @@ https://developer.apple.com/library/archive/technotes/tn2124/_index.html#//apple
but note that some of the things on this page are out of date (such as enabling
core dumps with `/etc/launchd.conf`).
+
+WINDOWS
+
+If the Windows version of Nvim crashes in a reproducible manner, you can take
+some steps to provide a useful bug report.
+
+You must obtain the debugger symbols (PDB) file for the Nvim executable: nvim.pdb.
+The PDB should be available from the same place that you obtained the
+executable (TODO: not currently provided by Nvim CI releases). Be sure to use
+the PDB that matches the EXE (same build).
+
+If you built the executable yourself with the Microsoft Visual C++ compiler,
+then the PDB was built with the EXE.
+
+If you have Visual Studio, use that instead of the VC Toolkit and WinDbg.
+
+For other compilers, always use the corresponding debugger: gdb or lldb.
+
+Debugging Nvim crashes with Visual Studio 2005 ~
+
+First launch nvim.exe and then launch Visual Studio. (If you don't have
+Visual Studio, get it from https://visualstudio.microsoft.com/downloads/).
+
+On the Tools menu, click Attach to Process. Choose the Nvim process.
+
+In Nvim, reproduce the crash. A dialog will appear in Visual Studio, telling
+you about the unhandled exception in the Nvim process. Click Break to break
+into the process.
+
+Visual Studio will pop up another dialog, telling you that no symbols are
+loaded and that the source code cannot be displayed. Click OK.
+
+Several windows will open. Right-click in the Call Stack window. Choose Load
+Symbols. The Find Symbols dialog will open, looking for (g)vim.pdb. Navigate
+to the directory where you have the PDB file and click Open.
+
+At this point, you should have a full call stack with vim function names and
+line numbers. Double-click one of the lines and the Find Source dialog will
+navigate to the directory where the Nvim source is (if you have it.)
+
+If you don't know how to debug this any further, follow the instructions
+at ":help bug-report". Paste the call stack into the bug report.
+
+From Visual Studio you can also try saving a minidump via the Debug menu and
+send it with the bug report. A minidump is a small file (<100KB), which
+contains information about the state of your process.
+
==============================================================================
Gdb *dev-tools-gdb*
@@ -194,5 +447,42 @@ example:
(gdb) target remote localhost:6666
(gdb) br main
<
+==============================================================================
+Debugging crashes or memory leaks *dev-tools-asan*
+
+BUILD WITH ASAN
+
+Building Nvim with Clang sanitizers (Address Sanitizer: ASan, Undefined
+Behavior Sanitizer: UBSan, Memory Sanitizer: MSan, Thread Sanitizer: TSan) is
+a good way to catch undefined behavior, leaks and other errors as soon as they
+happen. It's significantly faster than Valgrind.
+
+Requires clang 3.4 or later, and `llvm-symbolizer` must be in `$PATH`: >
+
+ clang --version
+
+Build Nvim with sanitizer instrumentation (choose one): >
+
+ CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_ASAN_UBSAN=ON"
+ CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_MSAN=ON"
+ CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_TSAN=ON"
+
+Create a directory to store logs: >
+
+ mkdir -p "$HOME/logs"
+
+Configure the sanitizer(s) via these environment variables: >
+
+ # Change to detect_leaks=1 to detect memory leaks (slower, noisier).
+ export ASAN_OPTIONS="detect_leaks=0:log_path=$HOME/logs/asan"
+ # Show backtraces in the logs.
+ export MSAN_OPTIONS="log_path=${HOME}/logs/msan"
+ export TSAN_OPTIONS="log_path=${HOME}/logs/tsan"
+
+Logs will be written to `${HOME}/logs/*san.PID` then.
+
+For more information: https://github.com/google/sanitizers/wiki/SanitizerCommonFlags
+
+
vim:tw=78:ts=8:et:ft=help:norl:
diff --git a/src/gen/gen_help_html.lua b/src/gen/gen_help_html.lua
@@ -91,6 +91,7 @@ local new_layout = {
local redirects = {
['api-ui-events'] = 'ui',
['credits'] = 'backers',
+ ['dev_tools'] = 'debug',
['plugins'] = 'editorconfig',
['terminal'] = 'nvim_terminal_emulator',
['tui'] = 'term',
diff --git a/src/nvim/README.md b/src/nvim/README.md
@@ -1,526 +1,5 @@
-Nvim core
-=========
+## Moved to:
-Module-specific details are documented at the top of each module (`terminal.c`, `undo.c`, …).
-
-See `:help dev` for guidelines.
-
-Filename conventions
---------------------
-
-The source files use extensions to hint about their purpose.
-
-- `*.c`, `*.generated.c` - full C files, with all includes, etc.
-- `*.c.h` - parametrized C files, contain all necessary includes, but require
- defining macros before actually using. Example: `typval_encode.c.h`
-- `*.h` - full headers, with all includes. Does *not* apply to `*.generated.h`.
-- `*.h.generated.h` - exported functions’ declarations.
-- `*.c.generated.h` - static functions’ declarations.
-
-Common structures
------------------
-
-- StringBuilder
-- kvec or garray.c for dynamic lists / vectors (use StringBuilder for strings)
-
-Logs
-----
-
-Low-level log messages sink to `$NVIM_LOG_FILE`.
-
-UI events are logged at DEBUG level.
-
- rm -rf build/
- make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG"
-
-Use `LOG_CALLSTACK()` (Linux only) to log the current stacktrace. To log to an
-alternate file (e.g. stderr) use `LOG_CALLSTACK_TO_FILE(FILE*)`. Requires
-`-no-pie` ([ref](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860394#15)):
-
- rm -rf build/
- make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG -DCMAKE_C_FLAGS=-no-pie"
-
-Many log messages have a shared prefix, such as "UI" or "RPC". Use the shell to
-filter the log, e.g. at DEBUG level you might want to exclude UI messages:
-
- tail -F ~/.local/state/nvim/log | cat -v | stdbuf -o0 grep -v UI | stdbuf -o0 tee -a log
-
-Build with ASAN
----------------
-
-Building Nvim with Clang sanitizers (Address Sanitizer: ASan, Undefined
-Behavior Sanitizer: UBSan, Memory Sanitizer: MSan, Thread Sanitizer: TSan) is
-a good way to catch undefined behavior, leaks and other errors as soon as they
-happen. It's significantly faster than Valgrind.
-
-Requires clang 3.4 or later, and `llvm-symbolizer` must be in `$PATH`:
-
- clang --version
-
-Build Nvim with sanitizer instrumentation (choose one):
-
- CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_ASAN_UBSAN=ON"
- CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_MSAN=ON"
- CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_TSAN=ON"
-
-Create a directory to store logs:
-
- mkdir -p "$HOME/logs"
-
-Configure the sanitizer(s) via these environment variables:
-
- # Change to detect_leaks=1 to detect memory leaks (slower, noisier).
- export ASAN_OPTIONS="detect_leaks=0:log_path=$HOME/logs/asan"
- # Show backtraces in the logs.
- export MSAN_OPTIONS="log_path=${HOME}/logs/msan"
- export TSAN_OPTIONS="log_path=${HOME}/logs/tsan"
-
-Logs will be written to `${HOME}/logs/*san.PID` then.
-
-For more information: https://github.com/google/sanitizers/wiki/SanitizerCommonFlags
-
-Reproducible build
-------------------
-
-To make a reproducible build of Nvim, set cmake variable `LUA_GEN_PRG` to
-a LuaJIT binary built with `LUAJIT_SECURITY_PRN=0`. See commit
-cb757f2663e6950e655c6306d713338dfa66b18d.
-
-Debug: Performance
-------------------
-
-### Profiling (easy)
-
-For debugging performance bottlenecks in any code, there is a simple (and very
-effective) approach:
-
-1. Run the slow code in a loop.
-2. Break execution ~5 times and save the stacktrace.
-3. The cause of the bottleneck will (almost always) appear in most of the stacktraces.
-
-### Profiling (fancy)
-
-For more advanced profiling, consider `perf` + `flamegraph`.
-
-### USDT profiling (powerful)
-
-Or you can use USDT probes via `NVIM_PROBE` ([#12036](https://github.com/neovim/neovim/pull/12036)).
-
-> USDT is basically a way to define stable probe points in userland binaries.
-> The benefit of bcc is the ability to define logic to go along with the probe
-> points.
-
-Tools:
-- bpftrace provides an awk-like language to the kernel bytecode, BPF.
-- BCC provides a subset of C. Provides more complex logic than bpftrace, but takes a bit more effort.
-
-Example using bpftrace to track slow vim functions, and print out any files
-that were opened during the trace. At the end, it prints a histogram of
-function timing:
-
- #!/usr/bin/env bpftrace
-
- BEGIN {
- @depth = -1;
- }
-
- tracepoint:sched:sched_process_fork /@pidmap[args->parent_pid]/ {
- @pidmap[args->child_pid] = 1;
- }
-
- tracepoint:sched:sched_process_exit /@pidmap[args->pid]/ {
- delete(@pidmap[args->pid]);
- }
-
- usdt:build/bin/nvim:neovim:eval__call_func__entry {
- @pidmap[pid] = 1;
- @depth++;
- @funcentry[@depth] = nsecs;
- }
-
- usdt:build/bin/nvim:neovim:eval__call_func__return {
- $func = str(arg0);
- $msecs = (nsecs - @funcentry[@depth]) / 1000000;
-
- @time_histo = hist($msecs);
-
- if ($msecs >= 1000) {
- printf("%u ms for %s\n", $msecs, $func);
- print(@files);
- }
-
- clear(@files);
- delete(@funcentry[@depth]);
- @depth--;
- }
-
- tracepoint:syscalls:sys_enter_open,
- tracepoint:syscalls:sys_enter_openat {
- if (@pidmap[pid] == 1 && @depth >= 0) {
- @files[str(args->filename)] = count();
- }
- }
-
- END {
- clear(@depth);
- }
-
- $ sudo bpftrace funcslower.bt
- 1527 ms for Slower
- @files[/usr/lib/libstdc++.so.6]: 2
- @files[/etc/fish/config.fish]: 2
- <snip>
-
- ^C
- @time_histo:
- [0] 71430 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
- [1] 346 | |
- [2, 4) 208 | |
- [4, 8) 91 | |
- [8, 16) 22 | |
- [16, 32) 85 | |
- [32, 64) 7 | |
- [64, 128) 0 | |
- [128, 256) 0 | |
- [256, 512) 6 | |
- [512, 1K) 1 | |
- [1K, 2K) 5 | |
-
-Debug: TUI
-----------
-
-### TUI troubleshoot
-
-Nvim logs its internal terminfo state at 'verbose' level 3. This makes it
-possible to see exactly what terminfo values Nvim is using on any system.
-
- nvim -V3log
-
-### TUI Debugging with gdb/lldb
-
-Launching the nvim TUI involves two processes, one for main editor state and one
-for rendering the TUI. Both of these processes use the nvim binary, so somewhat
-confusingly setting a breakpoint in either will generally succeed but may not be
-hit depending on which process the breakpoints were set in.
-
-To debug the main process, you can debug the nvim binary with the `--headless`
-flag which does not launch the TUI and will allow you to set breakpoints in code
-not related to TUI rendering like so:
-
- lldb -- ./build/bin/nvim --headless --listen ~/.cache/nvim/debug-server.pipe
-
-While in lldb, enter `run`. You can then attach to the headless process in a
-new terminal window to interact with the editor like so:
-
- ./build/bin/nvim --remote-ui --server ~/.cache/nvim/debug-server.pipe
-
-Conversely for debugging TUI rendering, you can start a headless process and
-debug the remote-ui process multiple times without losing editor state.
-
-For details on using nvim-dap and automatically debugging the child (main)
-process, see
-[here](https://zignar.net/2023/02/17/debugging-neovim-with-neovim-and-nvim-dap/)
-
-### TUI trace
-
-The ancient `script` command is still the "state of the art" for tracing
-terminal behavior. The libvterm `vterm-dump` utility formats the result for
-human-readability.
-
-Record a Nvim terminal session and format it with `vterm-dump`:
-
- script foo
- ./build/bin/nvim -u NONE
- # Exit the script session with CTRL-d
-
- # Use `vterm-dump` utility to format the result.
- ./.deps/usr/bin/vterm-dump foo > bar
-
-Then you can compare `bar` with another session, to debug TUI behavior.
-
-### TUI redraw
-
-Set the 'writedelay' and 'redrawdebug' options to see where and when the UI is painted.
-
- :set writedelay=50 rdb=compositor
-
-Note: neovim uses an internal screenbuffer to only send minimal updates even if a large
-region is repainted internally. To also highlight excess internal redraws, use
-
- :set writedelay=50 rdb=compositor,nodelta
-
-### Terminal reference
-
-- `man terminfo`
-- http://bazaar.launchpad.net/~libvterm/libvterm/trunk/view/head:/doc/seqs.txt
-- http://invisible-island.net/xterm/ctlseqs/ctlseqs.html
-
-Data structures
----------------
-
-Buffer text is stored as a tree of line segments, defined in [memline.c](https://github.com/neovim/neovim/blob/v0.9.5/src/nvim/memline.c#L8-L35).
-The central idea is found in [ml_find_line](https://github.com/neovim/neovim/blob/v0.9.5/src/nvim/memline.c#L2800).
-
-Nvim lifecycle
---------------
-
-Following describes how Nvim processes input.
-
-Consider a typical Vim-like editing session:
-
-01. Vim displays the welcome screen
-02. User types: `:`
-03. Vim enters command-line mode
-04. User types: `edit README.txt<CR>`
-05. Vim opens the file and returns to normal mode
-06. User types: `G`
-07. Vim navigates to the end of the file
-09. User types: `5`
-10. Vim enters count-pending mode
-11. User types: `d`
-12. Vim enters operator-pending mode
-13. User types: `w`
-14. Vim deletes 5 words
-15. User types: `g`
-16. Vim enters the "g command mode"
-17. User types: `g`
-18. Vim goes to the beginning of the file
-19. User types: `i`
-20. Vim enters insert mode
-21. User types: `word<ESC>`
-22. Vim inserts "word" at the beginning and returns to normal mode
-
-Note that we split user actions into sequences of inputs that change the state
-of the editor. While there's no documentation about a "g command mode" (step
-16), internally it is implemented similarly to "operator-pending mode".
-
-From this we can see that Vim has the behavior of an input-driven state machine
-(more specifically, a pushdown automaton since it requires a stack for
-transitioning back from states). Assuming each state has a callback responsible
-for handling keys, this pseudocode represents the main program loop:
-
-```py
-def state_enter(state_callback, data):
- do
- key = readkey() # read a key from the user
- while state_callback(data, key) # invoke the callback for the current state
-```
-
-That is, each state is entered by calling `state_enter` and passing a
-state-specific callback and data. Here is a high-level pseudocode for a program
-that implements something like the workflow described above:
-
-```py
-def main()
- state_enter(normal_state, {}):
-
-def normal_state(data, key):
- if key == ':':
- state_enter(command_line_state, {})
- elif key == 'i':
- state_enter(insert_state, {})
- elif key == 'd':
- state_enter(delete_operator_state, {})
- elif key == 'g':
- state_enter(g_command_state, {})
- elif is_number(key):
- state_enter(get_operator_count_state, {'count': key})
- elif key == 'G'
- jump_to_eof()
- return true
-
-def command_line_state(data, key):
- if key == '<cr>':
- if data['input']:
- execute_ex_command(data['input'])
- return false
- elif key == '<esc>'
- return false
-
- if not data['input']:
- data['input'] = ''
-
- data['input'] += key
- return true
-
-def delete_operator_state(data, key):
- count = data['count'] or 1
- if key == 'w':
- delete_word(count)
- elif key == '$':
- delete_to_eol(count)
- return false # return to normal mode
-
-def g_command_state(data, key):
- if key == 'g':
- go_top()
- elif key == 'v':
- reselect()
- return false # return to normal mode
-
-def get_operator_count_state(data, key):
- if is_number(key):
- data['count'] += key
- return true
- unshift_key(key) # return key to the input buffer
- state_enter(delete_operator_state, data)
- return false
-
-def insert_state(data, key):
- if key == '<esc>':
- return false # exit insert mode
- self_insert(key)
- return true
-```
-
-The above gives an idea of how Nvim is organized internally. Some states like
-the `g_command_state` or `get_operator_count_state` do not have a dedicated
-`state_enter` callback, but are implicitly embedded into other states (this
-will change later as we continue the refactoring effort). To start reading the
-actual code, here's the recommended order:
-
-1. `state_enter()` function (state.c). This is the actual program loop,
- note that a `VimState` structure is used, which contains function pointers
- for the callback and state data.
-2. `main()` function (main.c). After all startup, `normal_enter` is called
- at the end of function to enter normal mode.
-3. `normal_enter()` function (normal.c) is a small wrapper for setting
- up the NormalState structure and calling `state_enter`.
-4. `normal_check()` function (normal.c) is called before each iteration of
- normal mode.
-5. `normal_execute()` function (normal.c) is called when a key is read in normal
- mode.
-
-The basic structure described for normal mode in 3, 4 and 5 is used for other
-modes managed by the `state_enter` loop:
-
-- command-line mode: `command_line_{enter,check,execute}()`(`ex_getln.c`)
-- insert mode: `insert_{enter,check,execute}()`(`edit.c`)
-- terminal mode: `terminal_{enter,execute}()`(`terminal.c`)
-
-## Important variables
-
-The current mode is stored in `State`. The values it can have are `MODE_NORMAL`,
-`MODE_INSERT`, `MODE_CMDLINE`, and a few others.
-
-The current window is `curwin`. The current buffer is `curbuf`. These point
-to structures with the cursor position in the window, option values, the file
-name, etc.
-
-All the global variables are declared in `globals.h`.
-
-### The main event-loop
-
-The main loop is implemented in state_enter. The basic idea is that Vim waits
-for the user to type a character and processes it until another character is
-needed. Thus there are several places where Vim waits for a character to be
-typed. The `vgetc()` function is used for this. It also handles mapping.
-
-What we consider the "Nvim event loop" is actually a wrapper around `uv_run` to
-handle both the `fast_events` queue and possibly (a suitable subset of) deferred
-events. Therefore "raw" `vim.uv.run()` is often not enough to "yield" from Lua
-plugins; instead they can call `vim.wait(0)`.
-
-Updating the screen is mostly postponed until a command or a sequence of
-commands has finished. The work is done by `update_screen()`, which calls
-`win_update()` for every window, which calls `win_line()` for every line.
-See the start of [drawscreen.c](drawscreen.c) for more explanations.
-
-### Command-line mode
-
-When typing a `:`, `normal_cmd()` will call `getcmdline()` to obtain a line with
-an Ex command. `getcmdline()` calls a loop that will handle each typed
-character. It returns when hitting `<CR>` or `<Esc>` or some other character that
-ends the command line mode.
-
-### Ex commands
-
-Ex commands are handled by the function `do_cmdline()`. It does the generic
-parsing of the `:` command line and calls `do_one_cmd()` for each separate
-command. It also takes care of while loops.
-
-`do_one_cmd()` parses the range and generic arguments and puts them in the
-exarg_t and passes it to the function that handles the command.
-
-The `:` commands are listed in [ex_cmds.lua](ex_cmds.lua).
-
-### Normal mode commands
-
-The Normal mode commands are handled by the `normal_cmd()` function. It also
-handles the optional count and an extra character for some commands. These
-are passed in a `cmdarg_T` to the function that handles the command.
-
-There is a table `nv_cmds` in [normal.c](normal.c) which
-lists the first character of every
-command. The second entry of each item is the name of the function that
-handles the command.
-
-### Insert mode commands
-
-When doing an `i` or `a` command, `normal_cmd()` will call the `edit()` function.
-It contains a loop that waits for the next character and handles it. It
-returns when leaving Insert mode.
-
-### Options
-
-There is a list with all option names in [options.lua](options.lua).
-
-Async event support
--------------------
-
-One of the features Nvim added is the support for handling arbitrary
-asynchronous events, which can include:
-
-- RPC requests
-- job control callbacks
-- timers
-
-Nvim implements this functionality by entering another event loop while
-waiting for characters, so instead of:
-
-```py
-def state_enter(state_callback, data):
- do
- key = readkey() # read a key from the user
- while state_callback(data, key) # invoke the callback for the current state
-```
-
-Nvim program loop is more like:
-
-```py
-def state_enter(state_callback, data):
- do
- event = read_next_event() # read an event from the operating system
- while state_callback(data, event) # invoke the callback for the current state
-```
-
-where `event` is something the operating system delivers to us, including (but
-not limited to) user input. The `read_next_event()` part is internally
-implemented by libuv, the platform layer used by Nvim.
-
-Since Nvim inherited its code from Vim, the states are not prepared to receive
-"arbitrary events", so we use a special key to represent those (When a state
-receives an "arbitrary event", it normally doesn't do anything other than
-update the screen).
-
-Main loop
----------
-
-The `Loop` structure (which describes `main_loop`) abstracts multiple queues
-into one loop:
-
- uv_loop_t uv;
- MultiQueue *events;
- MultiQueue *thread_events;
- MultiQueue *fast_events;
-
-`loop_poll_events` checks `Loop.uv` and `Loop.fast_events` whenever Nvim is
-idle, and also at `os_breakcheck` intervals.
-
-MultiQueue is cool because you can attach throw-away "child queues" trivially.
-For example `do_os_system()` does this (for every spawned process!) to
-automatically route events onto the `main_loop`:
-
- Process *proc = &uvproc.process;
- MultiQueue *events = multiqueue_new_child(main_loop.events);
- proc->events = events;
+- [dev_arch.txt](../../runtime/doc/dev_arch.txt)
+- [dev_tools.txt](../../runtime/doc/dev_tools.txt)
+- [develop.txt](../../runtime/doc/develop.txt)
