More information added

05_reversing/dynamic/setfunction_patching.md

@@ -4,7 +4,32 @@
 
 ## Overview
 
-`SetFunction()` is the official AmigaOS mechanism for **patching a library's JMP table** at runtime. It installs a custom function at a given LVO, replacing the original, and returns the old function pointer so a trampoline can be constructed.
+You want to know every file an application opens. Or every byte it writes. Or every memory allocation it makes — with sizes, flags, and call stacks. You could patch the binary. Or you could use the operating system's own hooking mechanism: **`SetFunction()`**.
+
+`SetFunction()` is AmigaOS's official API for **replacing a library's JMP table entry at runtime.** It atomically swaps the target address of a specific LVO, returning the original pointer so you can construct a trampoline. Every call through that LVO — from every task, in every process — now routes through your code. This is the foundation of Amiga reverse engineering tooling: file system monitors, API tracers, memory debuggers, and anti-piracy checks all begin with `SetFunction()`.
+
+```mermaid
+graph LR
+    subgraph "Before SetFunction"
+        APP1["App calls<br/>JSR -48(A6)"]
+        JMP["JMP table[-48]<br/>→ original Write"]
+        ORIG["dos.library<br/>Write_impl()"]
+    end
+    subgraph "After SetFunction"
+        APP2["App calls<br/>JSR -48(A6)"]
+        JMP2["JMP table[-48]<br/>→ my_write_hook"]
+        HOOK["my_write_hook()<br/>log, modify, block"]
+        TRAMP["Trampoline →<br/>original Write"]
+    end
+    APP1 --> JMP
+    JMP --> ORIG
+    APP2 --> JMP2
+    JMP2 --> HOOK
+    HOOK --> TRAMP
+    TRAMP --> ORIG
+```
+
+---
 
 ---
 
@@ -122,6 +147,174 @@ atexit(remove_hook);
 
 ---
 
+## Decision Guide — SetFunction vs Alternatives
+
+```mermaid
+graph TD
+    Q["Need to intercept<br/>library calls?"]
+    Q -->|"System-wide,<br/>all tasks"| SF["Use SetFunction()"]
+    Q -->|"Single task only"| ALT1["Consider patching<br/>the task's A6/A4"]
+    Q -->|"At load time,<br/>before execution"| ALT2["Binary patch<br/>or HUNK relocation"]
+    SF -->|"Need to call original?"| TRAMP["Write trampoline<br/>save orig pointer"]
+    SF -->|"Block/replace only"| BLOCK["Don't save orig<br/>simpler, no trampoline"]
+```
+
+| Approach | Scope | Invasiveness | Use Case |
+|---|---|---|---|
+| **SetFunction()** | System-wide | Low (official API) | API tracing, memory debugging, anti-piracy |
+| **Direct JMP table patch** | System-wide | Medium (bypasses API) | Pre-OS 2.0 compatibility |
+| **Task A6 replacement** | Single task | Medium | Per-application sandboxing |
+| **Binary patch (file)** | Single binary | High (modifies disk) | Permanent behavior change, crack intros |
+
+---
+
+## Named Antipatterns
+
+### 1. "The Leaky Hook"
+
+**What it looks like** — installing a hook but never removing it:
+
+```c
+void setup(void) {
+    Forbid();
+    orig = SetFunction(DOSBase, -48, my_write);
+    Permit();
+    // No atexit() cleanup — hook lives forever
+}
+```
+
+**Why it fails:** When the hooking program exits, `my_write` is unloaded from memory. But the JMP table still points to it. The next task that calls `Write()` jumps into freed memory → Guru Meditation.
+
+**Correct:** Always register cleanup:
+
+```c
+void cleanup(void) {
+    Forbid();
+    SetFunction(DOSBase, -48, orig);  // restore original
+    Permit();
+}
+// In main():
+atexit(cleanup);
+```
+
+### 2. "The Forbid-Free Patch"
+
+**What it looks like** — calling `SetFunction()` without `Forbid()`:
+
+```c
+// BROKEN — task switch during SetFunction may corrupt list
+orig = SetFunction(DOSBase, -48, my_write);
+```
+
+**Why it fails:** `SetFunction()` modifies the library's `lib_OpenCnt` and may trigger expunge logic. If a task switch occurs during this modification, another task may see an inconsistent state. The result: corrupted open counts, premature expunge, or lost patches.
+
+**Correct:** Always wrap in `Forbid()`/`Permit()`.
+
+### 3. "The Register Stomper"
+
+**What it looks like** — a hook that corrupts registers before calling the original:
+
+```asm
+_my_write:
+    MOVEM.L  D0-D2/A0-A1, -(SP)   ; save only D0-D2/A0-A1
+    JSR      _log_args
+    MOVEM.L  (SP)+, D0-D2/A0-A1
+    MOVEA.L  _orig_write, A0
+    JMP      (A0)                  ; D3-D7/A2-A6 may contain garbage!
+```
+
+**Why it fails:** The original `Write()` expects `D1`=file, `D2`=buffer, `D3`=length. If your logging code modified D3 and you didn't save/restore it, the original function sees a corrupted length — potentially writing gigabytes or zero bytes. Even worse: the caller may rely on other registers (D4-D7, A2-A5) being preserved per the AmigaOS ABI, and your hook trashed them.
+
+**Correct:** Save and restore ALL registers the original function might read or the caller expects preserved. The safest approach is `MOVEM.L D0-D7/A0-A6`.
+
+---
+
+## Use-Case Cookbook
+
+### File Access Tracer — Log Every Open
+
+```c
+static APTR orig_Open;
+
+LONG __asm my_Open(register __d1 STRPTR name,
+                    register __d2 LONG mode) {
+    LONG result = ((LONG(*)(STRPTR,LONG))orig_Open)(name, mode);
+    if (result) {
+        kprintf("Open: %s mode=%ld → handle=%ld\n", name, mode, result);
+    }
+    return result;
+}
+
+void install_file_tracer(void) {
+    Forbid();
+    orig_Open = SetFunction(DOSBase, -30, my_Open);
+    Permit();
+}
+```
+
+### Write Blocker — Prevent All Disk Writes
+
+```c
+static APTR orig_Write;
+static BOOL write_blocked = TRUE;
+
+LONG __asm my_Write(register __d1 BPTR fh,
+                     register __d2 APTR buf,
+                     register __d3 LONG len) {
+    if (write_blocked) {
+        return 0;  // pretend success, write nothing
+    }
+    return ((LONG(*)(BPTR,APTR,LONG))orig_Write)(fh, buf, len);
+}
+```
+
+### Detect SetFunction Itself Being Called (Anti-Anti-Debug)
+
+Some software detects patching by checking if `SetFunction` returns the expected original address. Counter-patch by hooking `SetFunction` itself:
+
+```c
+static APTR orig_SetFunction;
+
+APTR __asm my_SetFunction(register __a1 struct Library *lib,
+                           register __a0 LONG lvo,
+                           register __d0 APTR newFunc) {
+    if (lib == DOSBase && lvo == -48) {
+        return orig_Write;  // lie: return our hook as "original"
+    }
+    return ((APTR(*)(struct Library*,LONG,APTR))orig_SetFunction)(lib, lvo, newFunc);
+}
+```
+
+---
+
+## Cross-Platform Comparison
+
+| Amiga Concept | Win32 Equivalent | Linux Equivalent | Notes |
+|---|---|---|---|
+| `SetFunction()` | `DetourAttach()` (Microsoft Detours) | `LD_PRELOAD` + `dlsym(RTLD_NEXT)` | Same idea: intercept library calls transparently |
+| JMP table modification | IAT hooking | PLT/GOT hooking | Amiga's JMP table is simpler — one 6-byte write vs multi-level indirection |
+| Trampoline pattern | Detour trampoline | `dlsym(RTLD_NEXT, "write")` | Same: call original after instrumentation |
+| `Forbid()`/`Permit()` | `SuspendThread` / `ResumeThread` (crude) | Signal blocking (crude) | Amiga's task-level atomicity is unique — no per-thread suspend needed |
+| System-wide by default | Per-process by default | Per-process by default | Amiga's flat address space means one hook covers everything |
+
+---
+
+## FAQ
+
+### Does SetFunction work on all library types?
+
+Yes — `SetFunction()` works on any library with a standard JMP table (exec, dos, graphics, intuition, third-party). It does NOT work on ROM-based resident modules that use a different dispatch mechanism (some Kickstart modules).
+
+### Can multiple hooks coexist on the same function?
+
+Yes — in a chain. Each hook saves the "original" pointer (which may itself be a previous hook's trampoline). Removal must happen in reverse order: last hooked = first removed. Removing hooks out of order breaks the chain.
+
+### Is SetFunction safe across CPU architectures?
+
+On 68000–68060, yes. However, 68040+ systems with data cache enabled may cache the old JMP table entry. Always call `CacheClearU()` after `SetFunction()` on 040/060 to flush the data cache and ensure the new target address is visible to the instruction fetch unit.
+
+---
+
 ## References
 
 - NDK39: `exec/execbase.h`