CS 480 M01 Lecture Notes 15 - Oct 26, 2022

Last Time / Previously

Ch 20. Storage

1. Adding a Disk - (Simple / Traditional) Disk Installation Procedure
2. Storage Hardware
3. Interfaces
4. Attachement and Low-level Management of Drives
5. Software Side of Storage
6. Disk Partitioning Disk device files - device: /dev/sda . partitions: /dev/sda1, /dev/sda2, ...
   Ways to divide up disks
   Traditional partitioning
   MBR partitioning
   GPT partitioning
   Linux partitioning parted (or gparted)
   
7. LVM
   • "lets you reallocate space dynamically from the greedy partition to the needy partition"
   • improveed / supercharged / abstracted version of disk partitioning:
     individual storage devices (physical volumes) grouped into volume groups whose blocks can be allocated to logical volumes (block device files) that hold filesystems
   • LVM allows to move , grow and shrink logical volumes on the fly, snapshots, mirroring or striping, ...
   • Linux LVM: lvm command plus many subcommands
   • LVM configuration
   • Volume Shapshots ... read the book
   • Filesystem Resizing

8. RAID: Redundant Arrays of Inexpensive (Independent) Disks

   • RAID Levels
   • Linear mode
   • Level 0
   • Level 1 / Mirroring
   • Level 1+0 (Stripes of Mirrors) and 0+1 (Mirrors of Stripes)
   • Level 5 stripes both data and parity info ( Redundancy, read performance, disk efficiency )
   • Level 6 like RAID 5 but two parity disks - Withstands complete failure of two disks
   • mdadm : Linus Software RAID

Ch 20. Storage Continued

9. Filesystems
Code that implements abstractions described in Ch 5, The Filesystem

10. Traditional FSs:
    Volume management and RAID are implemented separately from the filesystem itself
    Older FSs got corrupted often if power was lost -> fsck
    Modern FSs - include journaling :
    • Before FS operation occurs, the required modifications are first written to the journal
    • Once the journal update is complete, a “commit record” is written to mark the end of the entry
    • Finally the normal FS modified
    If a crash occurs during the update, the FS can replay the journal log to reconstruct a perfectly consistent FS - quick ~ second(s) !
    UNIX : Berkeley Fast FS => UFS; XFS (SGI's IRIX) , JFS (Journaled FS for IBM's AIX)
    Linux: ext2 (similar functionality as BFFS) => etx3 (journaled) => ext4 (raised size limits, improved performace, extents,.. ); ReiserFS, ...

    fsck - check & repair filesystems
    plus tune2fs (dumpe2fs) and dd
    

11. Next-Generation Filesystems: ZFS and BtrFS

    Vertically integrated approaches to storage management
    filesystem plus
    logical volume manager and a RAID controller
    • Copy-on-write - avoids overwriting data in place
      FS modifies the in-memory copy and then writes it to a previously vacant disk block
      plus parents as well
      root block updated at the end
      FS moves directly into a new consistent state
    • Error detection "take data integrity far more seriously than do traditional filesystems"
      - store checksum for every block - misreads are detected
    • Performance - Copy-on-write makes them slower (worst case up to 1/2) but comparable mostly
12. ZFS OpenSolaris introduced in 2005 => quickly to Solaris 10 and various BSD distros

13. BtrFS

    • Setup and Storage Conversion

      # mkfs.btrfs -L btrfs_demo -d raid1 /dev/sdd /dev/sde
      # mkdir /extra/btrfs
      # mount LABEL=btrfs_demo /extra/btrfs
      # df -h /extra/btrfs/
      # btrfs filesystem usage  /extra/btrfs
      # mkdir /extra/btrfs/usr_copy
      # cd /usr; tar cf - . | (cd /extra/btrfs/usr_copy; tar xfp -)
      # btrfs filesystem df  /extra/btrfs
      # btrfs filesystem usage  /extra/btrfs
      # btrfs device add /dev/sdf /extra/btrfs
      # btrfs filesystem usage  /extra/btrfs
      # btrfs balance start --full-balance /extra/btrfs
      # btrfs filesystem usage  /extra/btrfs
      # btrfs balance start -dconvert=raid5 -mconvert=raid5 /extra/btrfs
      # btrfs filesystem usage  /extra/btrfs
      

    • Volumes and Subvolumes

    • Volume Snapshots

    • Shalow Copies

14. Data backup strategy

udisks, udisksd, udisksctl, hwinfo
enumerate and perform operations on disks and storage devices
ex.: udisksctl info -b /dev/sdb

also note (for other hardware):
• lshw
• dmidecode - other hardware components, plus other useful info such as serial numbers and BIOS revision
• lspci info about PCI buses in the system and devices connected to them
• lscpu lscpu - info about CPUs - from /proc/cpuinfo
• lsmem, lsblk, lcipc, ...
• hwinfo --disk --only /dev/sdb

Ch 13: TCP/IP Networking

1. textbook
2. TCP/IP Protocol Suite by Forouzan, McGraw Hill 2003, ISBN 0-07-246060-1
3. TCP/IP Illustrated, Vol 1 by Stevens, Addison-Wesley 2000 , ISBN 0-201-63346-9

TCP/IP Networking

The networking system that underlies the Internet.
• IP - Internet Protocol - underlying delivery protocol - RFC 791 / STD 5
• TCP - Transmission Control Protocol - connection oriented transport protocol on top of IP (reliable delivery, flow control, congestion control, ..) - RFC 793 / STD 7
• UDP - User Datagram Protocol - packet oriented service (no connection, no congestion control, ...) - RFC 768 / STD 6

History

ARPANET by DARPA (Defense Advanced Research Project Agency) in 1969
NSFNET backbone (till 1994)
Collection of private networks (owned by ISPs) today

Today Managed by

• ICANN - Internet Corporation for Assigned Names and Numbers
  The one most in charge .... A not-for-profit corporation that was formed to assume responsibility for the IP address space allocation, protocol parameter assignment, domain name system management, and root server system management functions previously performed under U.S government.
  web : http://www.icann.org
• ISOC - Internet Society - membership organization representing users
  web : http://www.isoc.org
  open membership organization representing users and responsible for Internet's technical development and infrastructure standards. Parent of the Internet Engineering Task Force (IETF).
  • IETF - development and standardization of technical aspect, open forum
    web : http://www.ietf.org
• IGF - Internet Governance Forum - established by UN in 2006 for policy oriented Internet related discussions
  web : http://intgovforum.org

Network Standards and Documentation

Accessing and reading the Internet's authoritative technical documentation is crucial skill for sysadmins.
RFC - Request for Comments : standards, proposed changes, bulletins, ... (index)
FYI - For Your Information
BCP - Best Current Practice
STD - Standard
  at RFC Editor

OSI Model

Conceptual network model to standardize network communication of diverse systems through layers of protocols

Open System Interconnection (OSI) by International Standard Organization (ISO)
Peer-to-Peer Processes


End to End Delivery

TCP/IP Protocol Suite and Network Model

• Application - end-user application programs (browser <-> web server; port numbers)
• Transport - delivery of data to applications (reliability and flow control issues, TCP / UDP, also SCTP, DCCP, ....)
• Network - basic communication, addressing, and routing - global traffic ( from a host on CS network to a server at CNN.com using IP addresses)
• Link - network hardware and device drivers (local network and addressing, from cs lab machine to cs mail server per MAC addressing)
• Physical - physical medium itself
  
  

Packets & encapsulation:

Data on network travels in "packets": bursts of data with a max length per the link layer.
Packet has a header and a payload (data to be transferred).
Header has source, destination, plus usually checksums, protocol-specific information, and other handling instructions.
The name of the primitive data unit depends on the layer of the protocol:
• frame (Link - ethernet, ...)
• packet (network - IP)
• segment (transport - TCP, ..)

Ethernet

TCP, UDP, and IP Headers

Next : Maximum Transfer Unit (MTU)