amiga-bootcamp/04_linking_and_libraries/compiler_stubs.md

[← Home](../README.md) · [Linking & Libraries](README.md)

# Compiler Library Call Stubs

## Overview

This document explains how each major AmigaOS compiler generates library calls and what the resulting machine code looks like — essential for recognising compiler signatures during reverse engineering.

---

## The AmigaOS Calling Convention

All library calls use the same register-based convention:
- **A6** = library base pointer
- Arguments in **A0–A3**, **D0–D7** as specified by the `.fd` file
- Return value in **D0** (or D0+D1 for 64-bit returns)
- **Callee** preserves: D2–D7, A2–A6
- **Caller** may destroy: D0, D1, A0, A1

The code generated by all compilers follows this — the difference is how they **set up A6** before the `JSR`.

---

## SAS/C 6.x

SAS/C uses **global library base variables** (`_DOSBase`, `_SysBase`, etc.) and generates direct `JSR LVO(A6)` calls. The standard pattern:

```asm
; Calling Write(file, buffer, length):
;   D1=file, D2=buffer, D3=length, A6=_DOSBase

MOVE.L  _DOSBase, A6        ; load library base from global
MOVE.L  file, D1            ; arg 1
MOVE.L  buffer, D2          ; arg 2
MOVE.L  length, D3          ; arg 3
JSR     -48(A6)             ; Write LVO = -48
```

### SAS/C Inline Pragmas

SAS/C uses `#pragma libcall` to define inline stubs:
```c
#pragma libcall DOSBase Write 30 32103
/* 30 = bias/6 = 8th function (LVO -48)
   32103 = register encoding: D3,D2,D1 → args 3,2,1 */
```

The pragma-generated stub wraps the JSR into a C-callable inline function.

### SAS/C Library Base Globals

SAS/C's `clib/dos_protos.h` + startup code declares:
```c
extern struct DosLibrary *DOSBase;
extern struct ExecBase    *SysBase;
```

These are initialised in `c.o` (the startup stub):
```asm
_c_start:
    MOVE.L  4.W, A6             ; SysBase from exception vector
    MOVE.L  A6, _SysBase        ; store in global
    LEA     _dosname, A1        ; "dos.library"
    MOVEQ   #0, D0              ; version 0
    JSR     -552(A6)            ; OpenLibrary
    MOVE.L  D0, _DOSBase
```

---

## GCC (m68k-amigaos / bebbo GCC)

GCC uses a different stub mechanism — **inline functions** from `inline/` headers or the `libnix` library:

```c
/* inline/exec.h (generated from exec_lib.fd) */
static __inline APTR
AllocMem(ULONG byteSize, ULONG requirements)
{
    register APTR _res __asm("d0");
    register struct ExecBase *const _SysBase __asm("a6") = SysBase;
    register ULONG _byteSize __asm("d0") = byteSize;
    register ULONG _requirements __asm("d1") = requirements;
    __asm volatile ("jsr a6@(-198:W)"
        : "=r"(_res)
        : "r"(_SysBase), "r"(_byteSize), "r"(_requirements)
        : "a0", "a1", "fp0", "fp1", "cc", "memory");
    return _res;
}
```

The `jsr a6@(-198:W)` emits a 4-byte instruction: `JSR -198(A6)` using 16-bit word displacement.

### GCC Code Pattern

```asm
; GCC calling AllocMem(1024, MEMF_CHIP):
MOVEA.L _SysBase, A6        ; GCC uses same global SysBase
MOVEQ   #$01, D1             ; MEMF_CHIP = 2... actually:
MOVE.L  #$00000400, D0       ; byteSize = 1024
MOVE.L  #$00000002, D1       ; MEMF_CHIP
JSR     -$C6(A6)             ; -198 decimal = AllocMem
MOVE.L  D0, _mybuf           ; save return value
```

### PC-Relative Addressing (GCC Default)

GCC with `-fpic` or `-fbaserel` generates **PC-relative** accesses:
```asm
LEA     _SysBase(PC), A0    ; load SysBase address PC-relative
MOVEA.L (A0), A6            ; dereference to get SysBase value
JSR     -$228(A6)           ; OpenLibrary LVO = -552
```

This reduces relocation entries and is the default for GCC on AmigaOS.

---

## VBCC (vbcc m68k-amigaos)

VBCC uses `inline.h` style stubs similar to GCC. The calling pattern is identical at the assembly level:

```asm
; VBCC compiled library call — indistinguishable from GCC at asm level
MOVEA.L (_SysBase), A6
MOVE.L  D2, -(SP)           ; VBCC may push/pop D2 as scratch
MOVE.L  arg, D0
JSR     -$C6(A6)
MOVE.L  (SP)+, D2
```

VBCC is slightly more aggressive about not clobbering D2-D7, matching the calling convention exactly.

---

## Recognising Compiler Signatures During RE

### SAS/C Signature

```asm
; Entry prologue:
LINK    A5, #-N              ; frame setup (A5 = frame pointer)
MOVEM.L D2-D7/A2-A4, -(SP)  ; save callee-saved regs
; ...
MOVEM.L (SP)+, D2-D7/A2-A4  ; restore
UNLK    A5                   ; teardown
RTS
```

SAS/C uses `A5` as frame pointer and `LINK/UNLK` for stack frames.

### GCC Signature

```asm
; Entry prologue (GCC without frame pointer):
MOVEM.L D2/A2, -(SP)        ; only saves what it uses
; ...
MOVEM.L (SP)+, D2/A2
RTS
```

GCC typically does **not** use LINK/UNLK unless required. It uses `A5` as an additional scratch register and often generates `JSR` through `A0` for indirect calls.

### VBCC Signature

VBCC generates very clean, minimal code — similar to GCC but with more consistent callee-save patterns and no GCC-specific idioms (e.g., no `__builtin_` calls).

---

## Library Call Pattern Summary

| Pattern | Compiler | Key Signature |
|---|---|---|
| `LINK A5, #-N` + `MOVEM D2-D7/A2-A4` | SAS/C | Frame pointer A5, full reglist |
| `JSR LVO(A6)` with global `_LibBase` | SAS/C / GCC | Both use global |
| `jsr a6@(-LVO:W)` (GAS syntax) | GCC inline | 16-bit short displacement |
| PC-relative `LEA _SysBase(PC)` | GCC -fpic | No absolute refs |
| No LINK, minimal MOVEM | GCC / VBCC | No frame pointer |

---

## References

- NDK39: `fd/`, `proto/`, `inline/`, `clib/` directories
- SAS/C 6.x Programmer's Guide — pragma libcall chapter
- GCC m68k-amigaos port documentation (bebbo/m68k-amigaos-toolchain)
- VBCC documentation — register calling convention
- *Amiga ROM Kernel Reference Manual: Libraries* — calling conventions appendix