docs(dev): reorganize and clean up technical documentation

docs/Developer Guide/Developer Guide/Dependencies/CKEditor.md

@@ -0,0 +1,6 @@
+# CKEditor
+*   We migrated away from the legacy CKEditor builds using Webpack and instead use the prebuilt npm binaries.
+*   The role of the `packages/ckeditor5` is to gather the CKEditor for consumption by the client, which includes plugin definitions.
+*   The internal Trilium plugins (e.g. cut to note, include note) are present in `packages/ckeditor5/src/plugins`.
+*   External CKEditor plugins that needed adjustments are present in `packages/ckeditor5-*`.
+    *   To integrate a new plugin, see <a class="reference-link" href="CKEditor/Plugin%20migration%20guide.md">Plugin migration guide</a>.

docs/Developer Guide/Developer Guide/Dependencies/CKEditor/Differences from upstream.md

@@ -0,0 +1,25 @@
+# Differences from upstream
+*   Embeds [`~~isaul32/ckeditor5-math~~`](https://github.com/isaul32/ckeditor5-math)  <a class="reference-link" href="ckeditor5-math.md">ckeditor5-math</a>, which is a third-party plugin for adding math support. CKEditor itself also has a [math plugin](https://ckeditor.com/docs/ckeditor5/latest/features/math-equations.html) with MathType and ChemType but it's premium-only.
+*   Zadam left a TODO in `findandreplaceUI`: `// FIXME: keyboard shortcut doesn't work:` [`https://github.com/ckeditor/ckeditor5/issues/10645`](https://github.com/ckeditor/ckeditor5/issues/10645)
+*   `packages\ckeditor5-build-balloon-block\src\mention_customization.js` introduces note insertion via `@` character.
+
+| Affected file | Affected method | Changed in | Reason for change |
+| --- | --- | --- | --- |
+| `packages/ckeditor5-mention/src/mentionui.ts` | `createRegExp()` | `6db05043be24bacf9bd51ea46408232b01a1b232` (added back) | Allows triggering the autocomplete for labels and attributes in the attribute editor. |
+| `init()` | `55a63a1934efb9a520fcc2d69f3ce55ac22aca39` | Allows dismissing @-mention permanently after pressing ESC, otherwise it would automatically show up as soon as a space was entered. |     |
+
+## Checking the old repo
+
+Use the following command to identify commits from Zadam:
+
+```
+git log --oneline --author="adam" --all
+```
+
+It's best to run the command from zadam's fork of `trilium-ckeditor5` instead of the TriliumNext once since it might not contain all the unmerged branches.
+
+To show a filtered diff of a commit:
+
+```
+git show d42e772783 -- ':!*yarn.lock' ':!*packages/ckeditor5-build-balloon-block/build/*' ':!*package.json'
+```

docs/Developer Guide/Developer Guide/Dependencies/CKEditor/Plugin migration guide.md

@@ -0,0 +1,85 @@
+# Plugin migration guide
+This guide walks through the basic steps to take to integrate a CKEditor 5 plugin for use inside the Trilium monorepo, which allows:
+
+*   Making modifications to the implementation without having to maintain a new repo.
+*   Integrating an older plugin based on the legacy installation method so that it works well with the new one.
+
+> [!IMPORTANT]
+> This guide assumes that the CKEditor plugin is written in TypeScript. If it isn't, then you will have to port it to TypeScript to match the rest of the monorepo.
+
+## Step 1. Creating the project skeleton
+
+First, we are going to generate a project from scratch so that it picks up the latest template for building CKEditor plugins, whereas the plugin which is being integrated might be based on the legacy method.
+
+Outside the `Notes` repository, we are going to use the CKEditor generator to generate the new project structure for us. We are not doing it directly inside `Notes` repository since it's going to use a different package manager (Yarn/NPM vs `pnpm`) and it also creates its own Git repository.
+
+```
+npx ckeditor5-package-generator @triliumnext/ckeditor5-foo --use-npm --lang ts --installation-methods current
+```
+
+Of course, replace `foo` with the name of the plugin. Generally it's better to stick with the original name of the plugin which can be determined by looking at the prefix of file names (e.g. `mermaid` from `mermaidui` or `mermaidediting`).
+
+## Step 2. Copy the new project
+
+1.  Go to the newly created `ckeditor5-foo` directory.
+2.  Remove `node_modules` since we are going to use `pnpm` to handle it.
+3.  Remove `.git` from it.
+4.  Copy the folder into the `Notes` repo, as a subdirectory of `packages`.
+
+## Step 3. Updating dependencies
+
+In the newly copied package, go to `package.json` and edit:
+
+1.  In `devDependencies`, change `ckeditor5` from `latest` to the same version as the one described in `packages/ckeditor5/package.json` (fixed version, e.g. `43.2.0`).
+2.  In `peerDependencies`, change `ckeditor5` to the same version as from the previous step.
+3.  Similarly, update `vitest` dependencies to match the monorepo one.
+4.  Remove the `prepare` entry from the `scripts` section.
+5.  Change `build:dist` to simply `build`.
+6.  In `tsconfig.dist.json`, change `typings/types` to `../typings/types.d.ts` to be compatible with the latest TypeScript version.
+
+## Step 4. Install missing dependencies and build errors
+
+Run `pnpm build-dist` on the `Notes` root, and:
+
+1.  If there is an error about `Invalid module name in augmentation, module '@ckeditor/ckeditor5-core' cannot be found.`, simply replace `@ckeditor/ckeditor5-core` with `ckeditor5`.
+2.  Run the build command again and ensure there are no build errors.
+3.  Commit the changes.
+
+## Step 5. Using `git subtree` to pull in the original repo
+
+Instead of copying the files from the existing plugin we are actually going to carry over the history for traceability. To do so, we will use a temporary directory inside the repo:
+
+```
+git subtree add --prefix=_regroup/<name> https://[...]/repo.git <main_branch>
+```
+
+This will bring in all the commits of the upstream repo from the provided branch and rewrite them to be placed under the desired directory.
+
+## Step 6. Integrate the plugin
+
+1.  Start by copying each sub-plugin (except the main one such as `FooEditing` and `FooUI`).
+    1.  If they are written in JavaScript, port them to TypeScript.
+        1.  Remove any non-TypeScript type documentation.
+    2.  If they have non-standard imports to CKEditor, such as `'ckeditor5/src/core.js'`, rewrite them to simply `ckeditor`.
+2.  Install any necessary dependencies used by the source code (try going based on compilation errors rather than simply copying over all dependencies from `package.json`).
+3.  Keep the existing TypeScript files that were generated automatically and integrate the changes into them.
+4.  In `tsconfig.json` of the plugin, set `compilerOptions.composite` to `true`.
+5.  Add a workspace dependency to the new plugin in `packages/ckeditor5/package.json`.
+6.  In `packages/ckeditor5` look for `plugins.ts` and import the top-level plugin in `EXTERNAL_PLUGINS`.
+
+## Handling CSS
+
+Some plugins have custom CSS whereas some don't.
+
+1.  `import` the CSS in the `index.ts` of the plugin.
+2.  When building the plugin, `dist/index.css` will be updated.
+3.  In `plugins.ts` from `packages/ckeditor5`, add an import to the CSS.
+
+## Integrating from another monorepo
+
+This is a more complicated use-case if the upstream plugin belongs to a monorepo of another project (similar to how `trilium-ckeditor5` used to be).
+
+1.  Create a fresh Git clone of the upstream monorepo to obtain the plugin from.
+2.  Run `git filter-repo --path packages/ckeditor5-foo/` (the trailing slash is very important!).
+3.  Run `git subtree add` just like in the previous steps but point to the local Git directory instead (by appending `/.git` to the absolute path of the repository).
+4.  Follow same integration steps as normal.

docs/Developer Guide/Developer Guide/Dependencies/CKEditor/ckeditor5-math.md

@@ -0,0 +1,16 @@
+# ckeditor5-math
+<figure class="image image-style-align-right"><img src="ckeditor5-math_image.png"><figcaption><code>ckeditor5-math</code> in action.</figcaption></figure>
+
+A fork of [isaul32/ckeditor5-math](https://github.com/isaul32/ckeditor5-math), which is the CKEditor5 plugin which adds the math functionality. We keep our own version to be able to use it on the latest version of CKEditor, alongside some custom improvements.
+
+## Development environment
+
+*   Tested on Node.js 20.
+*   The package manager is yarn 1 (v1.22.22 is known to be working fine for it at the time of writing).
+
+Important commands:
+
+*   To check if the code has any formatting issues: `yarn lint`
+*   To start a live preview: `yarn start`
+*   To run the tests: `yarn test`
+    *   Note that this requires Chromium, on NixOS this can be achieved by running a `nix-shell -p chromium`, and running `CHROME_BIN=$(which chromium) yarn test` inside it.

docs/Developer Guide/Developer Guide/Dependencies/Per-dependency checks.md

@@ -0,0 +1,2 @@
+# Per-dependency checks
+<table class="ck-table-resized"><colgroup><col> <col> <col> <col> <col></colgroup><thead><tr><th>Dependency</th><th>Name in <code>library_loader</code></th><th>Things to check for a basic sanity check</th><th>&nbsp;</th><th>Protected by unit tests</th></tr></thead><tbody><tr><td><code>better-sqlite3</code></td><td>&nbsp;</td><td>See&nbsp;<a class="reference-link" href="Per-dependency%20checks/bettersqlite%20binaries.md">bettersqlite binaries</a>.</td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>jsdom</code></td><td>&nbsp;</td><td><ul><li>Note map</li><li>Clipper</li><li>Note similarity</li></ul></td><td>Protected by typings, should catch any potential changes in API.</td><td>Yes</td></tr><tr><td><code>async-mutex</code></td><td>&nbsp;</td><td><ul><li>Sync</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>axios</code></td><td>&nbsp;</td><td><ul><li>Can't be directly tested, as it's exposed only via the backend script API.</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>sax</code></td><td>&nbsp;</td><td><ul><li>EverNote imports</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><ul><li><code>ws</code></li><li><code>debounce</code></li></ul></td><td>&nbsp;</td><td><ul><li>Check any action is reported from server to client (e.g. delete a note).</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>ejs</code></td><td>&nbsp;</td><td><ul><li>Onboarding / first setup</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>dayjs</code></td><td>&nbsp;</td><td><ul><li>Day notes</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>semver</code></td><td>&nbsp;</td><td><ul><li>Application should start.</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>https-proxy-agent</code></td><td>&nbsp;</td><td>???</td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>sax</code></td><td>&nbsp;</td><td><ul><li>EverNote import</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>ini</code></td><td>&nbsp;</td><td><ul><li>Affects config, generally if the application starts then it should be OK.</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>jsplumb</code></td><td><code>RELATION_MAP</code></td><td><ul><li>Relation map note type</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>jquery.mark.es6</code></td><td><code>MARKJS</code></td><td><ul><li>In search, when highlighting the text that matched.</li><li>In search in HTML, which might not actually be used since it seems to have been replaced by CKEditor's own find &amp; replace dialog.</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>knockout.js</code></td><td>&nbsp;</td><td><ul><li>Used in rendering the login and main layout of the application.</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>normalize.min.css</code></td><td>&nbsp;</td><td><ul><li>Used in shared notes.</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>wheel-zoom.min.js</code></td><td><code>WHEEL_ZOOM</code></td><td><ul><li>When opening a image that is in attachment.</li><li>When opening a stand-alone image note.</li><li>When zooming in a mermaid chart.</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>fancytree</code></td><td>&nbsp;</td><td><ul><li>The note tree should be fully functional.</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>bootstrap</code></td><td>&nbsp;</td><td><ul><li>Check mostly the on-boarding pages, when there is no database.</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>electron-debug</code></td><td>&nbsp;</td><td><ul><li>Run electron using <code>npm run start-electron</code> and check that the debug hotkeys are still working (Ctrl+Shift+I on Windows/Linux, Cmd+Alt+I for dev tools, Cmd/Ctrl+R for reload).</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>electron-dl</code></td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>eslint</code></td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td></tr><tr><td><code>marked</code></td><td>&nbsp;</td><td><ul><li>Importing a markdown note.</li></ul></td><td>&nbsp;</td><td>Yes</td></tr><tr><td><code>force-graph</code></td><td>&nbsp;</td><td><ul><li>Note map</li></ul></td><td>&nbsp;</td><td>&nbsp;</td></tr></tbody></table>

docs/Developer Guide/Developer Guide/Dependencies/Per-dependency checks/bettersqlite binaries.md

@@ -0,0 +1,25 @@
+# bettersqlite binaries
+### The native node bindings
+
+`better-sqlite3` has native Node bindings. With updates of `better-sqlite3`, but also of Electron and Node.js versions, these bindings need to be updated.
+
+Note that Electron and Node.js versions need different versions of these bindings, since Electron usually packs a different version of Node.js.
+
+During development, `pnpm install` tries to build or reuse prebuilt natives for the current Node.js version. This makes `npm run start-server` work out of the box. Trying to run `npm run start-electron` with these versions generally causes an error such as this:
+
+```
+Uncaught Exception:
+Error: The module '/Users/elian/Projects/Notes/node_modules/better-sqlite3/build/Release/better_sqlite3.node'
+was compiled against a different Node.js version using
+NODE_MODULE_VERSION 108. This version of Node.js requires
+NODE_MODULE_VERSION 116. Please try re-compiling or re-installing
+the module (for instance, using `npm rebuild` or `npm install`).
+```
+
+### How the natives are handled
+
+To avoid issues between the `server` and the `desktop`, the `desktop` build gets its own copy of the `bettersqlite3` dependency in its `node_module`. This copy is then rebuilt automatically to match the Electron version.
+
+This process of rebuilding is handled by `scripts/electron-rebuild.mts` which runs automatically after `pnpm install` (via `postinstall`).
+
+If needed, the script can be run manually again via `pnpm postinstall`.