File Systems, I/O & Storage

A file system is the data structure that organizes bytes on a storage device into files and directories. Without a file system, a disk is just a flat sequence of blocks with no meaning.

Everything is a file (Unix philosophy):
• Regular files, directories, symbolic links
• Block devices (/dev/sda — raw disk access)
• Character devices (/dev/tty — terminals, /dev/null, /dev/urandom)
• Named pipes (FIFOs)
• Sockets (yes, even network connections can be represented as files)
• /proc and /sys virtual filesystems (kernel data structures exposed as files)

The VFS (Virtual File System) layer:
The kernel's VFS is an abstraction layer that provides a unified API (open, read, write, close, mkdir, stat) to all file systems. Whether the underlying storage is ext4, NTFS, btrfs, NFS, or procfs — user programs always use the same syscalls. The VFS dispatches to the right implementation.

ext4 (the most common Linux filesystem):
• Disk is divided into blocks (4 KB each by default)
• Inodes: every file has an inode — a data structure storing metadata:
  - File type (regular, directory, symlink, device)
  - Permissions (rwxrwxrwx) and owner (UID, GID)
  - Size, timestamps (created, modified, accessed)
  - Pointers to data blocks
  The inode does NOT contain the filename! Filenames live in directories.
• Directories: a file whose content is a list of (filename → inode number) pairs
• Hard links: two directory entries pointing to the same inode. Deleting one doesn't delete the file — only when reference count reaches 0 does the inode get freed
• Symbolic links: a file whose content is a path string. Like a shortcut

The I/O path (reading a file):
① User calls read(fd, buf, size) — syscall
② Kernel checks page cache — is this data already in RAM?
③ If yes (cache hit) — copy from page cache to user buffer → done
④ If no (cache miss) — submit I/O request to block device driver
⑤ Driver sends command to disk (NVMe, SATA)
⑥ Disk hardware reads data → DMA into kernel memory (page cache)
⑦ Kernel copies from page cache to user buffer
⑧ Return to user space

Page cache: the kernel caches file contents in RAM. Subsequent reads of the same data skip the disk entirely. Writes also go to the page cache first (write buffering) — flushed to disk later by pdflush/kdmflush threads (or on fsync()).

File descriptors: every open file is represented by a file descriptor (small integer). fd 0 = stdin, 1 = stdout, 2 = stderr. open() returns a new fd. File descriptors are entries in the process's file descriptor table, which point to kernel file objects.

journaling in ext4: before modifying disk structures, ext4 writes a journal entry. If the system crashes mid-write, the journal is replayed on next mount to restore consistency. Prevents filesystem corruption.

1import os
2import stat
3import pathlib
4import mmap
5
6# ── File metadata (inode info) ─────────────────────────
7print("=== File System Info ===")
8
9path = pathlib.Path("/etc/hosts")
10st = os.stat(path)
11
12print(f"File:          {path}")
13print(f"Inode:         {st.st_ino}")
14print(f"Size:          {st.st_size} bytes")
15print(f"Hard links:    {st.st_nlink}")
16print(f"Owner UID:     {st.st_uid}")
17print(f"Permissions:   {oct(stat.S_IMODE(st.st_mode))}")
18print(f"Device:        {st.st_dev}")
19print(f"Modified:      {st.st_mtime}")
20print(f"Is file:       {stat.S_ISREG(st.st_mode)}")
21print(f"Is dir:        {stat.S_ISDIR(st.st_mode)}")
22
23# ── File descriptors ───────────────────────────────────
24print("
25=== File Descriptors ===")
26fd = os.open("/tmp/test_fd.txt", os.O_CREAT | os.O_WRONLY | os.O_TRUNC, 0o644)
27print(f"Opened fd:     {fd}")
28os.write(fd, b"Hello from raw fd!
29")
30os.fsync(fd)   # flush to disk (wait for physical write)
31os.close(fd)
32
33# Read it back
34fd = os.open("/tmp/test_fd.txt", os.O_RDONLY)
35data = os.read(fd, 1024)
36print(f"Read:          {data!r}")
37os.close(fd)
38
39# ── List open file descriptors of this process ─────────
40print("
41=== Open FDs for this process ===")
42fd_dir = f"/proc/{os.getpid()}/fd"
43try:
44    for fd_name in os.listdir(fd_dir):
45        try:
46            target = os.readlink(f"{fd_dir}/{fd_name}")
47            print(f"  fd {fd_name:3s} → {target}")
48        except:
49            pass
50except PermissionError:
51    print("  (need elevated permissions)")
52
53# ── Directory walking ──────────────────────────────────
54print("
55=== /proc/self/ contents ===")
56for entry in sorted(pathlib.Path("/proc/self").iterdir())[:15]:
57    kind = "DIR" if entry.is_dir() else "FILE"
58    print(f"  {kind:4s}  {entry.name}")
59
60# ── Reading /proc virtual filesystem ──────────────────
61print("
62=== Kernel data via /proc ===")
63with open("/proc/uptime") as f:
64    up_secs = float(f.read().split()[0])
65    print(f"Uptime:        {up_secs/3600:.1f} hours")
66
67with open("/proc/loadavg") as f:
68    load = f.read().split()
69    print(f"Load avg:      {load[0]} {load[1]} {load[2]} (1m 5m 15m)")