101 on Tech Notes by ClaudeXu

Docker Essentials

Sun, 23 Aug 2026 15:35:08 +0800

Introduction

Some basic concepts:

Images - an ordered collection of root filesystem changes and corresponding execution parameters for use, acting as the basis for containers
Containers - a running instance of an image, acting as the operating environment
Docker Engine - a daemon process on the host working as the runtime for containers
Docker Desktop - a desktop toolkit including an embedded Docker Engine and other tools

Docker, Kubernetes, and VMs are all related to virtualization, but they have different common use cases:

Docker - a local dev environment to remove the “it works on my machine” issue
Kubernetes - for container orchestration, usually in a production environment
VMs - a more full-fledged environment, including a kernel and kernel modules

Work with Containers

Run Containers

docker run <image> pulls the image if needed and starts up a container. Notice that the image is identified in the format:

official images - repo:tag
personal images - user_ns/repo:tag

Example: run a container against the hello-world:latest image

$ docker run --rm hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Notes:

hello-world - an official image on Docker Hub
latest - avoid using latest because it may change when a newer version is published
Docker Hub - one of the main resources we pull images from

List Containers

To list all containers, including the stopped ones, use the following command:

docker container ls -as

# alternatively
docker ps -as

Explanation

a - display all containers, including the stopped ones
s - display the file size, for example 4.19MB (virtual 516MB)
- 4.19MB: the writable layer size, meaning how much data has been written inside this specific container
- 516MB: the total size of the container’s filesystem, including all the read-only image layers plus the writable layer

Inspect a Container

To inspect a container, use docker container inspect <container>

$ docker container inspect my-postgres-16
[
    {
        "Id": "5f61f69143c4058721441059b1a620fd18b2450fc9fb53b1e2d5a68e881c5388",
        "Created": "2026-09-04T11:28:45.999713761Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "postgres"
        ],
  ...
]

Go’s text/template is supported, so specific information can be queried

$ docker container inspect --format "{{json .Mounts}}" my-postgres-16 | jq .
[
  {
    "Type": "volume",
    "Name": "6d61519596fd358dd8e0b5b091da97174f5c75ed79bcd03a723c86e5ac17eced",
    "Source": "/var/lib/docker/volumes/6d61519596fd358dd8e0b5b091da97174f5c75ed79bcd03a723c86e5ac17eced/_data",
    "Destination": "/var/lib/postgresql/data",
    "Driver": "local",
    "Mode": "",
    "RW": true,
    "Propagation": ""
  }
]

$ docker container inspect --format "{{json .Config.Env}}" my-postgres-16 | jq .
[
  "NO_PROXY=*.test.example.com,.example.org,127.0.0.0/8",
  "no_proxy=*.test.example.com,.example.org,127.0.0.0/8",
  "POSTGRES_PASSWORD=p@ssw0rd",
  "HTTP_PROXY=http://172.17.0.1:10809",
  "http_proxy=http://172.17.0.1:10809",
  "HTTPS_PROXY=http://172.17.0.1:10809",
  "https_proxy=http://172.17.0.1:10809",
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/16/bin",
  "GOSU_VERSION=1.19",
  "LANG=en_US.utf8",
  "PG_MAJOR=16",
  "PG_VERSION=16.15-1.pgdg13+2",
  "PGDATA=/var/lib/postgresql/data"
]

To inspect a running container, another way is to get the tty of the running container with docker exec -it <container> <shell>

$ docker exec -it my-postgres-16 sh
# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.6
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Remove a Container

Remove a specific container with docker container remove <container_name>. For example:

$ docker container remove gifted_panini

Explanation:

gifted_panini is the name of the hello-world container

Images

What is an image?

App binaries and dependencies
Not a complete operating system, because it does NOT contain a kernel or kernel modules
Can be as small as a single file, like a static Go binary
Can be as large as an Ubuntu distro with apt, Apache, PHP, etc.

Browse images on Docker Hub. Typically, use official images and always identify a specific tag. Don’t use the latest tag since it may change to point to the newest one. Instead, indicate a concrete tag like 5.2.

To list all the images on the local computer, use the following command:

docker image ls

Image Layers

Images are built using a layered structure, as shown below:

To show the image layers, use the docker image history <image> command

$ docker image history nginx:1.31
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
4e5db4761e0f   5 weeks ago   CMD ["nginx" "-g" "daemon off;"]                0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   STOPSIGNAL SIGQUIT                              0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   EXPOSE map[80/tcp:{}]                           0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   ENTRYPOINT ["/docker-entrypoint.sh"]            0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   COPY 30-tune-worker-processes.sh /docker-ent…   4.62kB    buildkit.dockerfile.v0
<missing>      5 weeks ago   COPY 20-envsubst-on-templates.sh /docker-ent…   3.03kB    buildkit.dockerfile.v0
<missing>      5 weeks ago   COPY 15-local-resolvers.envsh /docker-entryp…   389B      buildkit.dockerfile.v0
<missing>      5 weeks ago   COPY 10-listen-on-ipv6-by-default.sh /docker…   2.12kB    buildkit.dockerfile.v0
<missing>      5 weeks ago   COPY docker-entrypoint.sh / # buildkit          1.62kB    buildkit.dockerfile.v0
<missing>      5 weeks ago   RUN /bin/sh -c set -x     && groupadd --syst…   82.7MB    buildkit.dockerfile.v0
<missing>      5 weeks ago   ENV DYNPKG_RELEASE=1~trixie                     0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   ENV PKG_RELEASE=1~trixie                        0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   ENV ACME_VERSION=0.4.1                          0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   ENV NJS_RELEASE=1~trixie                        0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   ENV NJS_VERSION=1.0.0                           0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   ENV NGINX_VERSION=1.31.3                        0B        buildkit.dockerfile.v0
<missing>      5 weeks ago   LABEL maintainer=NGINX Docker Maintainers <d…   0B        buildkit.dockerfile.v0
<missing>      6 weeks ago   # debian.sh --arch 'amd64' out/ 'trixie' '@1…   78.6MB    debuerreotype 0.17

Explanation

View the layers from bottom to top. The <missing> layer at the bottom, with a size of 78.6MB, is the first layer, created based on the base image.
The <missing> value indicates that a layer is based on an image or another layer — it does NOT mean something is actually missing. <missing> means Docker has no local image metadata (ID) associated with that specific point in the chain.
- If we separately have debian:trixie pulled locally, and one of nginx’s layers happens to match debian’s topmost layer exactly (same content hash), docker image history can actually resolve and show that real image ID instead of <missing> — because now there is a locally tracked image pointing at that layer.
4e5db4761e0f is the final/topmost layer, and by convention the ID of an image is the ID of its topmost layer.

$ docker image list -f reference='nginx:*'
IMAGE        ID             DISK USAGE   CONTENT SIZE   EXTRA
nginx:1.31   4e5db4761e0f        161MB             0B    U

Reuse Layers

With the layering structure above, layers can be cached and reused. For example, when we run docker pull nginx:1.31, the following happens:

Docker fetches the image manifest - a list of layer digests (content hashes) that make up the image.
For each layer digest, Docker checks if it has a layer with this exact digest in local storage.
- If yes, it skips downloading and reuses the layer.
- If no, it downloads that layer blob.
Once all layers are present locally, Docker creates the image record (config + tag, nginx:1.31) pointing at that chain of layers.

Build a Custom Image

Images are built on top of base images using the FROM command in a Dockerfile. To prove that, check the Dockerfile of the official image nginx:1.31.

#
# NOTE: THIS DOCKERFILE IS GENERATED VIA "update.sh"
#
# PLEASE DO NOT EDIT IT DIRECTLY.
#
FROM debian:trixie-slim

# other definition

One trick to view the base image is to run the cat /etc/os-release command directly inside the running container.

$ docker run --rm --entrypoint cat nginx:1.31 /etc/os-release
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.6
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Now let’s build our ‘mirror’ image. Below is the Dockerfile

FROM debian:trixie-slim

# install python
RUN apt-get update && apt-get install -y python3

# create program working dir
RUN mkdir -p /usr/src/app

# change working dir
WORKDIR /usr/src/app

# copy mirror program to container
COPY ./mirror.py ./

# run mirror program
CMD ["python3", "mirror.py"]

Notes:

Dockerfile is NOT a shell script file.
The -f option can be used to specify the file name if Dockerfile is not the name of the file.

Place the Dockerfile in the same directory as mirror.py. Below is the content of mirror.py.

import sys

def mirror():
    print("You: ", end='', flush=True)
    while (msg := sys.stdin.readline()):
        if msg == "": 
            break
        print(f"Mirror: {reverse_msg(msg.strip())}\n")
        print("You: ", end='', flush=True)
    

def reverse_msg(msg):
    return f"{msg[::-1]}"

if __name__ == "__main__":
    mirror()

Notes:

EOF causes an empty line, so the if statement checks against "".
On Windows, the EOF keystroke is Ctrl-Z (followed by Enter).

Build the image according to the Dockerfile with the command docker image build -t <tag> <dir_containing_Dockerfile>

$ export HTTP_PROXY=http://127.0.0.1:10809

$ export HTTPS_PROXY=http://127.0.0.1:10809

$ export NO_PROXY=localhost,127.0.0.1

$ docker image build -t mirror ./
[+] Building 104.5s (10/10) FINISHED                                                              docker:default
 => [internal] load build definition from Dockerfile                                                        0.5s
 => => transferring dockerfile: 336B                                                                        0.0s
 => [internal] load metadata for docker.io/library/debian:trixie-slim                                       4.8s
 => [internal] load .dockerignore                                                                           0.0s
 => => transferring context: 2B                                                                             0.0s
 => [1/5] FROM docker.io/library/debian:trixie-slim@sha256:d7e12182ce18b85b93007c1dedf31f2d29e01ccf3182cc  25.9s
 => => resolve docker.io/library/debian:trixie-slim@sha256:d7e12182ce18b85b93007c1dedf31f2d29e01ccf3182cc4  0.0s
 => => sha256:e426a54f50cc4cf82dd5cab8ba8426ed02c391840cb5a62dfd987542dbabea3b 451B / 451B                  0.0s
 => => sha256:6310eb16bf4251731feab01e8f633bf5e2d75a657ccad97f420b1f83cce457be 29.79MB / 29.79MB           24.9s
 => => sha256:d7e12182ce18b85b93007c1dedf31f2d29e01ccf3182cc4017c709b6259bc132 8.97kB / 8.97kB              0.0s
 => => sha256:abc9cb88a5587630d7f915f47b23b0668fe250fbfc6457aa4d52b534c1bbf73f 1.02kB / 1.02kB              0.0s
 => => extracting sha256:6310eb16bf4251731feab01e8f633bf5e2d75a657ccad97f420b1f83cce457be                   0.8s
 => [internal] load build context                                                                           0.0s
 => => transferring context: 378B                                                                           0.0s
 => [2/5] RUN apt-get update && apt-get install -y python3                                                 72.1s
 => [3/5] RUN mkdir -p /usr/src/app                                                                         0.3s
 => [4/5] WORKDIR /usr/src/app                                                                              0.0s
 => [5/5] COPY ./mirror.py ./                                                                               0.1s
 => exporting to image                                                                                      0.5s
 => => exporting layers                                                                                     0.5s
 => => writing image sha256:b1b4248eb439f90b513f16140a5341338b452dcf6e84ee5a85930aadc9b673dc                0.0s
 => => naming to docker.io/library/mirror                                                                   0.0s

Notes:

If a proxy is needed, set the environment variables.
- Neither the proxy settings in /etc/docker/daemon.json nor those in the systemd service work, because during docker image build, the daemon asks the CLI to fetch an anonymous pull token, and the CLI does NOT refer to daemon.json or the systemd service for proxy settings. Instead, it refers to environment variables.
docker image build -t mirror ./ builds the image from the Dockerfile in the current directory and tags it as mirror.

Try starting a container using the custom image

$ docker container run --rm -it mirror
You: Hello
Mirror: olleH

You: World
Mirror: dlroW

ENTRYPOINT and CMD

ENTRYPOINT and CMD are related but distinct Dockerfile instructions:

ENTRYPOINT: defines the fixed, main executable that always runs when the container starts. It’s meant to be the “this container’s whole purpose” command.
CMD: defines default arguments (or a default command, if no ENTRYPOINT is set) — meant to be easily overridden by whoever runs the container.

When both are present, CMD’s contents get appended as arguments to ENTRYPOINT. For example, given the following Dockerfile definition:

ENTRYPOINT ["hugo"]
CMD ["--help"]

Usage:

running docker run <image> → executes hugo --help
running docker run <image> server → the server you pass on the CLI replaces CMD entirely (not ENTRYPOINT), so it executes hugo server

Suppose a Dockerfile only has CMD ["nginx", "-g", "daemon off;"] and no ENTRYPOINT, then:

docker run <image> → runs nginx -g daemon off;
docker run <image> cat /etc/os-release → completely replaces CMD, runs cat /etc/os-release directly

To override the ENTRYPOINT defined in the Dockerfile, use the --entrypoint option of docker run

$ docker run --rm --entrypoint cat nginx:1.31 /etc/os-release

# alternatively we can override the ENTRYPOINT by entering a shell tty
$ docker run --rm -it --entrypoint sh ghcr.io/gohugoio/hugo:v0.145.0

Notice that the command to use is placed right after --entrypoint, unlike the normal docker run <container> <cmd> syntax.

Docker Networks

Basic concepts:

Each container is connected to the private virtual network bridge.
Each virtual network routes through a NAT firewall on the host IP.
All containers on a virtual network can talk to each other.
Best practice is to create a new virtual network for each app group. For example:
- network my_web_app for mysql and php/apache containers
- network my_api for mongo and nodejs containers
Containers can be attached to more than one virtual network.
Containers can skip virtual networks and use the host IP via --net=host.

Create a postgres container for demonstration purposes

$ docker container run -d --name my-postgres-16 -p 2345:5432 -e POSTGRES_PASSWORD=p@ssw0rd postgres:16
5f61f69143c4058721441059b1a620fd18b2450fc9fb53b1e2d5a68e881c5388

Notes:

-e POSTGRES_PASSWORD=p@ssw0rd is specific to postgres and sets the superuser password. Without it, the container cannot run.
2345:5432 means to route traffic on the host’s 2345 port to the container’s 5432 port.
- To reach the container’s 5432 port, on which postgres is listening, you can visit the host’s 2345 port.

Inspect the network with the command below:

$ docker container port my-postgres-16
5432/tcp -> 0.0.0.0:2345
5432/tcp -> [::]:2345

$ docker container inspect --format "{{json .NetworkSettings }}" my-postgres-16 | jq .
{
  "SandboxID": "4f58f408b32bb90873a30730a21b56e82077299eb315433ee938d585fd3eb40f",
  "SandboxKey": "/var/run/docker/netns/4f58f408b32b",
  "Ports": {
    "5432/tcp": [
      {
        "HostIp": "0.0.0.0",
        "HostPort": "2345"
      },
      {
        "HostIp": "::",
        "HostPort": "2345"
      }
    ]
  },
  "Networks": {
    "bridge": {
      "IPAMConfig": null,
      "Links": null,
      "Aliases": null,
      "DriverOpts": null,
      "GwPriority": 0,
      "NetworkID": "e32aa7bf189c397ee25ee6863ffbdd3f31c60021140d7dc1ac89901a3b584a8c",
      "EndpointID": "d3e042a233b600da6afa554a3e5490ffa75c8194065af680799e6bdd7559fb6e",
      "Gateway": "172.17.0.1",
      "IPAddress": "172.17.0.2",
      "MacAddress": "d2:89:fb:8d:df:68",
      "IPPrefixLen": 16,
      "IPv6Gateway": "",
      "GlobalIPv6Address": "",
      "GlobalIPv6PrefixLen": 0,
      "DNSNames": null
    }
  }
}

Notes:

docker container port only works with running containers.
In the form 5432/tcp -> 0.0.0.0:2345, the left side of the arrow is the container.
docker container inspect --format accepts Go’s text/template syntax.

Network Operations

To show networks, use docker network ls

$ docker network ls
NETWORK ID     NAME                                  DRIVER    SCOPE
fc4441078ba1   1-dimensional-data-modeling_default   bridge    local
e32aa7bf189c   bridge                                bridge    local
ecf83f26709e   host                                  host      local
0825c9fcbeb4   kind                                  bridge    local
f8770c7e8ed3   none                                  null      local
982a9344a9af   spark-cluster_default                 bridge    local
c27ddea3424b   spark-essentials_default              bridge    local

To inspect a network, use docker network inspect <network_id/network_name>

$ docker network inspect 1-dimensional-data-modeling_default
[
    {
        "Name": "1-dimensional-data-modeling_default",
        "Id": "fc4441078ba1d808255aad2438c33395dfff784b8b6c41e8489af3caf69745d0",
        "Created": "2026-07-18T09:47:39.10768521+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv4": true,
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.21.0.0/16",
                    "IPRange": "",
                    "Gateway": "172.21.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Options": {},
        "Labels": {
            "com.docker.compose.config-hash": "2cbb2e262bf5fe26ec10ac03a575a90f8f80dabc3cee805e9d21428064a6d98f",
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "1-dimensional-data-modeling",
            "com.docker.compose.version": "5.0.2"
        },
        "Containers": {
            "2bcd191a1a0b451b0576c0332c28320037cef44aae301c891f8b06fd0acf72f2": {
                "Name": "my-postgres-container",
                "EndpointID": "ac8ca61f1892634e29fff5b948b62bc7ab4194e8243b1ac7fb391ca6386ec0c9",
                "MacAddress": "4e:ce:98:d5:33:1c",
                "IPv4Address": "172.21.0.3/16",
                "IPv6Address": ""
            },
            "2fd442178af0c717c9cf74e3b6f29ccb983120197dd0a76abc7d4943fb9c397e": {
                "Name": "pgadmin",
                "EndpointID": "bb8a62d2546ef6ff283331dc03b77a1699772f388c054f844969057aa400034d",
                "MacAddress": "da:a2:5e:59:54:97",
                "IPv4Address": "172.21.0.2/16",
                "IPv6Address": ""
            }
        },
        "Status": {
            "IPAM": {
                "Subnets": {
                    "172.21.0.0/16": {
                        "IPsInUse": 5,
                        "DynamicIPsAvailable": 65531
                    }
                }
            }
        }
    }
]

To create a network, use docker network create [--driver <driver>] <network_name>

$ docker network create --driver bridge data_group
c094e7cbd0173e09a205b28e325e68c1ce9eb87a9f6acf913d5acaa959e04309

Different driver types:

bridge or docker0 - is the default network for containers. When we launch a new container with docker run, the container automatically connects to this bridge network. We cannot remove this default bridge network.

docker0 is the network interface used on the host machine, and the IP address usually assigned to the host is 172.17.0.1. However, it does NOT provide DNS resolution.

$ ifconfig docker0
  docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
    inet6 fe80::5016:68ff:fe8a:eeda  prefixlen 64  scopeid 0x20<link>
    ether 52:16:68:8a:ee:da  txqueuelen 0  (Ethernet)
    RX packets 17  bytes 476 (476.0 B)
    RX errors 0  dropped 0  overruns 0  frame 0
    TX packets 50  bytes 7572 (7.5 KB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

host - directly attached to the host’s physical network. It gains performance by skipping virtual networks but sacrifices the security of the container model.
none/null - not attached to anything. It removes eth0 and leaves you with only the localhost interface inside the container. (in the container, you can probably only do ping localhost)

To attach a network to a container, use docker network connect <network> <container>

$ docker network connect data_group my-postgres-16

$ docker network connect data_group my-postgres-container

$ docker network inspect --format "{{json .Containers}}" data_group | jq .
{
  "2bcd191a1a0b451b0576c0332c28320037cef44aae301c891f8b06fd0acf72f2": {
    "Name": "my-postgres-container",
    "EndpointID": "43157c47b41cdd162a7dd0f8635d41d75b3b2ba7f53c7b1d84904f3479b67266",
    "MacAddress": "da:9a:89:cd:6e:41",
    "IPv4Address": "172.22.0.3/16",
    "IPv6Address": ""
  },
  "5f61f69143c4058721441059b1a620fd18b2450fc9fb53b1e2d5a68e881c5388": {
    "Name": "my-postgres-16",
    "EndpointID": "451e71b01a4c62e3ddf4b530f62a900071b68a99742cbd7bfd059b7ff8fc4539",
    "MacAddress": "96:6a:e4:ff:d1:d9",
    "IPv4Address": "172.22.0.2/16",
    "IPv6Address": ""
  }
}

To detach a network from a container, use docker network disconnect <network> <container>

$ docker network disconnect data_group my-postgres-container

$ docker network inspect --format "{{json .Containers}}" data_group | jq .
{
  "5f61f69143c4058721441059b1a620fd18b2450fc9fb53b1e2d5a68e881c5388": {
    "Name": "my-postgres-16",
    "EndpointID": "451e71b01a4c62e3ddf4b530f62a900071b68a99742cbd7bfd059b7ff8fc4539",
    "MacAddress": "96:6a:e4:ff:d1:d9",
    "IPv4Address": "172.22.0.2/16",
    "IPv6Address": ""
  }
}

DNS Naming

Talking to containers using IPs is an anti-pattern. Instead, use DNS naming. The Docker daemon has a built-in DNS server, and Docker defaults the container’s hostname to its name.

$ docker image pull busybox:1.38
1.38: Pulling from library/busybox
b05093807bb0: Pull complete
Digest: sha256:dc2d74b28e4cf8984fa52af1f39bc7c3d9c73760b41a74d629f5d11b1ab28616
Status: Downloaded newer image for busybox:1.38
docker.io/library/busybox:1.38$ docker network connect data_group my-nginx

$ docker run --rm --network=data_group -it busybox:1.38 ping -c4 my-postgres-16
PING my-postgres-16 (172.22.0.2): 56 data bytes
64 bytes from 172.22.0.2: seq=0 ttl=64 time=0.088 ms
64 bytes from 172.22.0.2: seq=1 ttl=64 time=0.090 ms
64 bytes from 172.22.0.2: seq=2 ttl=64 time=0.125 ms
64 bytes from 172.22.0.2: seq=3 ttl=64 time=0.256 ms

--- my-postgres-16 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.088/0.139/0.256 ms

Notes

docker0 does NOT provide DNS capability, so it’s recommended to add containers to a user-defined network.
Containers cannot be reached by their hostnames from the host, but can be reached by IP from the host.
Many images are not shipped with the ping program, but busybox is.

Persistent Storage with Volumes

Containers are usually immutable and ephemeral, but normally data is mutable. Docker provides features to ensure separation of concerns with the two approaches below:

Data Volumes - binary files that reside on the host
Bind Mounts - a host path mounted to a container path

In a Dockerfile, VOLUME declares the mount points where containers expect external persistent data.

$ docker image inspect postgres:14 --format='{{json .Config.Volumes}}'
{"/var/lib/postgresql/data":{}}

Notes:

It’s common to use either a named volume or bind mounts to “inject” storage from the host into the container later, at run time.
VOLUME /some/path is a hint from the image author saying “this path holds data and should persist”, but we can arbitrarily “shadow” directories other than the one specified by VOLUME.
If you don’t explicitly mount a volume or bind mount to the VOLUME directory, Docker will automatically create an anonymous volume for it when the container starts.

Inspect the Volumes setting of postgres:16.

$ docker image inspect postgres:16 --format '{{json .Config.Volumes}}'  | jq
{
  "/var/lib/postgresql/data": {}
}

Inspect the Mounts of a postgres:16 container

$ docker container inspect my-postgres-16 --format '{{json .Mounts}}' | jq
[
  {
    "Type": "volume",
    "Name": "6d61519596fd358dd8e0b5b091da97174f5c75ed79bcd03a723c86e5ac17eced",
    "Source": "/var/lib/docker/volumes/6d61519596fd358dd8e0b5b091da97174f5c75ed79bcd03a723c86e5ac17eced/_data",
    "Destination": "/var/lib/postgresql/data",
    "Driver": "local",
    "Mode": "",
    "RW": true,
    "Propagation": ""
  }
]

Data Volumes

List all data volumes with docker volume list

$ docker volume list
DRIVER    VOLUME NAME
local     1-dimensional-data-modeling_pgadmin-data
local     1-dimensional-data-modeling_postgres-data
local     1dbece87659b1ea45fd59805e39156d8c57b1f05781e55032335aa6e35e07d07
local     3eb2dc4d2b3d9fbebedbcf052017854f012fcb58c16b82aea584331c5ee7dac7
local     6d61519596fd358dd8e0b5b091da97174f5c75ed79bcd03a723c86e5ac17eced

Inspect a specific volume with docker volume inspect <volume_name>

$ docker volume inspect 6d61519596fd358dd8e0b5b091da97174f5c75ed79bcd03a723c86e5ac17eced
[
    {
        "CreatedAt": "2026-09-04T19:28:46+08:00",
        "Driver": "local",
        "Labels": {
            "com.docker.volume.anonymous": ""
        },
        "Mountpoint": "/var/lib/docker/volumes/6d61519596fd358dd8e0b5b091da97174f5c75ed79bcd03a723c86e5ac17eced/_data",
        "Name": "6d61519596fd358dd8e0b5b091da97174f5c75ed79bcd03a723c86e5ac17eced",
        "Options": null,
        "Scope": "local"
    }
]

The automatically generated anonymous volumes have dummy names that are not user-friendly. In practice, when creating a volume, it’s recommended to provide a meaningful name.

$ docker container run -d --name your-postgres-16 -v your-pg16:/var/lib/postgresql/data -e POSTGRES_PASSWORD=helloworld postgres:16
3bf304a4bebf29ceae0fc539dfb375083d438835e362b963eca44180e7c5315a

$ docker volume ls
DRIVER    VOLUME NAME
...
local     your-pg16

Notes:

The -v option lets you create a volume and mount it to the specified path.
Anonymous volumes are basically an artifact of the early days of Docker, before volumes could be named. You would normally specify the volume name when creating the container.
One best practice is to name the volume the same as your project, so that it intuitively shows what the volume is for.

Removing the container will NOT delete the volume automatically, because you may later create another container to continue using the data in the volume.

Collision Pitfall

Technically, it’s possible to share a volume across different containers for reading, and that makes sense for a web server and an ftp server. However, for writing, containers may collide with each other. An edge case is that collision can occur even within a single container. Suppose we have the following mount “map” for a drupal container:

volumes:
  - drupal_postgres_cms:/var/www/html/modules
  - drupal_postgres_cms:/var/www/html/profiles
  - drupal_postgres_cms:/var/www/html/sites
  - drupal_postgres_cms:/var/www/html/themes

Docker will not create separate sub-directories for ‘modules’, ‘profiles’, ‘sites’ and ’themes’. By default, mounting a named volume attaches the volume’s entire root at that container path. That means ‘/var/www/html/modules’ shows you the whole volume root, and all of the other three show you the exact same whole volume root.

Using a subpath can resolve the collision above, but an easier approach is to use different volumes.

Bind Mounts

With bind mounts, a file or a directory can be mapped from the host to the container, which will shadow the corresponding path inside the container.

$ docker container run -v /some/path/or/file/in/host:/path/or/file/in/container <image>

Notes

The leading slash / in /some/path/or/file/in/host indicates that it is a bind mount, instead of a named volume.

This technique is widely used in local development. One practical example is bind mounting an nginx configuration file from host to container

$ docker run --name my-custom-nginx-container -v /host/path/nginx.conf:/etc/nginx/nginx.conf:ro -d nginx

Data Volumes vs. Bind Mounts

Feature	Named Data Volume	Bind Mount
Primary Use Case	Production, database storage, persistence	Local development, source code sync
Host Directory Control	Managed by Docker	Specified explicitly by user
Cross-Platform Compatibility	High	Low (path dependent)
Performance on macOS/Win	High	Slower

Docker Compose

In practice, a product or solution may involve several containers, e.g. one for web hosting, another for ftp service, and a third for the database. Instead of creating and running those containers with separate docker container run commands, Docker Compose can better configure the collaboration among those containers. The core of Docker Compose is a YAML file, usually named docker-compose.yml, which contains configuration for:

containers
networks
volumes

version: '3.1' # v1 is used if not specified. Recommend v2 minimum

services: # this actually means to treat different containers as different service providers. Think of one providing the db service, another providing the web service, and a third...
  <servicename>: # a meaningful name. This is also the **DNS name** inside the network
    image: # optional if you use build:
    command: # optional, **replaces the default CMD** specified by the image
    environment: # optional, same as -e in 'docker run'
    volumes: # optional, same as -v in 'docker run'
  <servicename2>: # another service ...


volumes: # optional, same as 'docker volume create'

networks: # optional, same as 'docker network create'

Here is a concrete example of docker-compose.yml

version: '2'

services:
  
  wordpress:
    image: wordpress
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_PASSWORD: example
    volumes:
      - ./wordpress-data:/var/www/html # the leading dot means the current directory
  
  mysql:
    image: mariadb
    environment:
      MYSQL_ROOT_PASSWORD: example
    volumes:
      - ./mysql-data:/var/lib/mysql

With Docker Compose, the two most commonly used commands are:

docker compose up - set up networks and volumes, and then start all containers
docker compose down - stop all containers and remove networks; by default, volumes and images are kept

Example - Drupal CMS

Suppose we’d like to use Drupal CMS with Docker Compose. Two components are needed:

Drupal instance
Postgres DB instance

Here is the docker-compose.yml file

services:

  drupal:
    image: drupal:11.3.16-php8.5-apache
    ports:
      - 8080:80
    depends_on:
      - postgres
    volumes:
      - drupal_storage_modules:/var/www/html/modules
      - drupal_storage_profiles:/var/www/html/profiles
      - drupal_storage_sites:/var/www/html/sites
      - drupal_storage_themes:/var/www/html/themes
    networks: 
      - network

  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: drupal
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pwd_523
    volumes:
      - postgres_storage:/var/lib/postgresql/data
    networks: 
      - network

volumes:
  drupal_storage_modules:
  drupal_storage_profiles:
  drupal_storage_sites:
  drupal_storage_themes:
  postgres_storage:

networks:
  network:
    driver: bridge

Notes:

Place the docker-compose.yml file under a your_cms directory. By default, the directory name becomes the project name, and all resources — including containers, networks, and volumes — are generated under that namespace on the host, so the directory name appears as the leading characters of those generated resources.
Containers still talk to each other using the service name on the network, not including the “namespace” characters.

Navigate to the directory and execute docker compose up, or docker compose up -d for detached mode.

$ docker compose up
[+] up 8/8
 ✔ Network your_cms_network                Created                    0.4s
 ✔ Volume your_cms_drupal_storage_modules  Created                    0.1s
 ✔ Volume your_cms_drupal_storage_profiles Created                    0.1s
 ✔ Volume your_cms_drupal_storage_sites    Created                    0.2s
 ✔ Volume your_cms_drupal_storage_themes   Created                    0.3s
 ✔ Volume your_cms_postgres_storage        Created                    0.3s
 ✔ Container your_cms-postgres-1           Created                    1.6s
 ✔ Container your_cms-drupal-1             Created                    1.1s
Attaching to drupal-1, postgres-1
...

To stop the containers, execute docker compose down

$ docker compose down
[+] down 3/3
 ✔ Container your_cms-drupal-1   Removed                 3.0s
 ✔ Container your_cms-postgres-1 Removed                 3.3s
 ✔ Network your_cms_network      Removed                 0.9s

Compose with Build

In some cases, instead of using someone else’s image, we may want to build our own image. Below is the docker-compose.yml file

version: '2'

services:
  proxy:
    build:
      context: . # to find the docker file under the current working dir
      dockerfile: nginx.Dockerfile # explicitly define the Dockerfile
    image: nginx-custom # give the built image a custom name/tag
    ports:
      - 80:80
  web:
    image: httpd

nginx.Dockerfile file

FROM nginx:1.11

COPY nginx.conf /etc/nginx/conf.d/default.conf

nginx.conf file

server {

  listen 80;

  location / {

    proxy_pass         http://web;
    proxy_redirect     off;
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Host $server_name;

  }
}

Notes

It’s a great way to handle complex builds that involve lots of variables or build args.
If the image is not found in the cache, docker compose up will build the image.
docker compose build can rebuild the image.

Vifm: The Vim Way to Manage Files

Sun, 23 Aug 2026 15:35:08 +0800

Introduction

Vifm is a TUI dual-pane file manager. You can dramatically improve your productivity with Vifm, and it will feel quite natural if you are familiar with Vim bindings.

Motions

Vifm uses Vim keybindings:

h - go one level up in the directory tree
l - enter the directory highlighted by the cursor
j and k - move up and down in the pane to select items
H, M, and L - go to the top, middle, and bottom of the content list
m and ' - mark a location with a label and later jump to the marked location
- :marks command - show all the marks
  - In the Marks view, use dd to delete bookmarks

Window Operations

Tab key - switch between the two panes
Hold Ctrl and press w, then s - switch the layout to a horizontal split
Hold Ctrl and press w, then v - switch the layout to a vertical split
Hold Ctrl and press w, then o - switch to a one-pane layout. To switch back to a two-pane layout, use either C-w s or C-w v.
- Notice that in the one-pane layout, there are still two open panes, but only one of them is displayed. Tab still works to switch between the two panes.
C-w H or C-w L - switch the current pane to the left or right.

File Operations

File Management

cw - ‘change word’ means to rename the file in vifm
- cW - with capital W, change the base name of the file
- A - append to the file name, similar to A in vim
dd - delete (cut) the current item
- u - undo to restore the deleted item
- p - put (paste) the deleted (cut) item in the current position
yy - yank (copy) the item
- p - put (paste) the yanked (copied) item
Ctrl + g - show detailed metadata information including path, name, size, type, permissions, mime type, mtime, atime, etc. about a file or directory

File Selection

There are multiple ways to select items:

t - toggle item selection
:select <item_name> - select the desired items, such as select a.txt, select *txt, and select */
- Notice that the selection is cumulative
- select - select the current item / show the number of selected items
- Esc - deselect all
v - enter visual mode for selection
- Enter - end the selection
- Esc - deselect all
:invert s - invert the selection

Delete and Restore

dd can delete (cut) files, and under the hood, the files are moved to the ‘vifm trash’. To view the location of the trash, use the :trashes command.

┌── Non-empty trash directories ────────┐
│ /home/chsu/.local/share/vifm/Trash    |

To list the trashed items, use the :lstrash command.

┌── Original paths of files in trash ──────────┐
│ /home/chsu/Downloads/my_backups.zip          |

To restore deleted items, enter the corresponding trash directory, toggle the files to restore with t, and finally use the :restore command.

Filters

There are three basic file filters:

dot files filter: displays or hides hidden files and directories
local filter: keeps matching files and directories in the current directory
permanent filter: sets a global filter to filter out matching files and directories

Dot Files Filter

Key bindings:

za - toggle the visibility of dot files
zo - show dot files
zm - hide dot files

Notice that the dot files filter does not affect the special . and .. directories, whose appearance is controlled by set dotdirs.

rootparent - show “../” in root directory of file system
nonrootparent - show “../” in non-root directories of file system
treeleafsparent - show “../” in empty directories of tree view

By default, set dotdirs returns dotdirs=nonrootparent,treeleafsparent.

Local Filter

=<regex> creates a local filter:

it keeps the items matching the filter
it automatically resets when the directory is changed

Permanent Filter

The permanent filter is split into two parts:

zf - add the selected files to the permanent filter implicitly
:filter <regex> - set the filter explicitly to filter out or hide matching files

Notice that the permanent filter behaves like a global option: even if we navigate to another directory, the permanent filter affects that directory as well.

Key bindings:

:filter or zO - clear the filters, including both explicit and implicit filters
:filter! or :invert - invert the filter
- So, if we’d like to use :filter as the command to ‘keep’ specific items in one step, use :filter! <pattern_to_keep_item>

:filter? - view the current filter

Filter -- Flags -- Value
Local
Explicit    ---->    .*txt
Implicit
Press ENTER or type command to continue

The filter information below shows:

  Filter -- Flags -- Value
Local
Explicit    ---->    file2
Implicit    I        ^file1$|^dump\.txt$
Press ENTER or type command to continue

file2 is set explicitly by :filter file2.
file1 and dump.txt are set implicitly by zf.
Setting an explicit filter with :filter <regex> clears the implicit filter.

Sorting

There are two approaches to sorting:

:sort - set a simple global sorting order
set sort=<sort keys> - set a complex global sorting order

Notice that the sorting order affects all directories globally.

`:sort` for Simple Sorting Order

The :sort command shows the available sorting criteria to choose from.

![Sorting criterion]

C-n and C-p Emacs key bindings can be used.
Alternatively, simply use the shortcut letter.

Related comands

:invert o - invert the sorting order
:invert? o - display the sorting order, either descending or ascending

`set sort` for Complex Sorting Order

The syntax is :set sort=<sort keys>, and the available keys are:

   [+-]ext     - extension of files and directories
   [+-]fileext - extension of files only
   [+-]name    - name (including extension)
   [+-]iname   - name (including extension, ignores case)
   [+-]type    - file type (dir/reg/exe/link/char/block/sock/fifo)
   [+-]dir     - directory grouping (directory < file)
   [+-]gid     - group id (*nix only)
   [+-]gname   - group name (*nix only)
   [+-]mode    - file mode (file type + permissions) in octal (*nix only)
   [+-]perms   - permissions string (*nix only)
   [+-]uid     - owner id (*nix only)
   [+-]uname   - owner name (*nix only)
   [+-]nlinks  - number of hard links (*nix only)
   [+-]inode   - inode number (*nix only)
   [+-]size    - size
   [+-]nitems  - number of items in a directory (zero for files)
   [+-]groups  - groups extracted via regexps from vifm-'sortgroups'
   [+-]target  - symbolic link target (empty for other file types)
   [+-]atime   - time accessed (e.g., read, executed)
   [+-]ctime   - time changed (changes in metadata, like mode)
   [+-]mtime   - time modified (when file contents is changed)

+ - ascending order
- - descending order

Notes:

Multiple sorting keys can be set simultaneously, e.g. :set sort=+size,-name.
:set sort displays the current sort setting.

Configuration

The configuration file is located at:

Linux: ~/.config/vifm/vifmrc
Windows: $env.appdata/vifm/vifmrc

Bookmarks

The corresponding configuration section is shown below:

" ------------------------------------------------------------------------------

" :mark mark /full/directory/path [filename]

" mark b ~/bin/
mark h ~/
mark s ~/Sandbox/
mark l ~/Downloads/
mark o ~/Documents/
mark t ~/Temp/

In the configuration above, several directories are configured, so the '<mark> key binding navigates to the corresponding directory immediately.

File Handler

To open a file with the desired program, a file handler needs to be configured. The configuration contains many pre-configured file handlers for different file types, but those programs may not be installed on the OS. For example, to open PDF files with the zathura program:

" Pdf
filextype {*.pdf},<application/pdf> zathura %c %i &, apvlv %c, xpdf %c

In vifm, move the cursor to a PDF file and press Enter to open it with zathura. Notice that zathura needs to be installed on the OS.

File Previewer

Similarly, a file previewer needs to be configured so that file contents can be previewed with e and w.

fileviewer {*.7z},<application/x-7z-compressed> 7z l %c

Key bindings:

e - view the contents, instead of editing the file
- e on a directory shows the tree structure of the directory. Notice: do NOT preview a directory containing many files/subdirectories. It may freeze the program.
- Enter opens a file instead of viewing its contents in vifm
w - view file/directory contents in the other pane
- Shift + Tab - switch the cursor to the other pane to scroll through the preview contents
  - A single Tab shifts the ‘focus’ to the other file pane. Remember that even if there is only one pane displayed, there are still two open panes under the hood.
- w again to exit preview mode

Metadata Columns

Metadata columns such as size, atime, etc. can be configured to display in the UI with the command below:

:set viewcolumns=<column_setting>

<column_setting> has the following format: [width][{name}].

[width] - can be either a number or - representing automatic sizing
[{name}] - metadata columns to display; the supported columns are listed below
- {name} - filename
- {ext} - file extension
- {size} - file size
- {perms} - permissions (like -rwxr-xr-x)
- {mtime} - modification time
- {atime} - access time
- {ctime} - change time
- {dir} - directory indicator (/ for dirs)
- {type} - file type indicator (/, @, |, etc.)
- {target} - symlink target
- {nlinks} - number of hard links
- {inode} - inode number
- {uid} - user ID
- {gid} - group ID

Some examples

-{size} - automatic width for displaying the size of files
6{name} - a width of 6 characters for displaying the name. If the file/directory name exceeds 6 characters, it is not truncated; the 6 is simply ignored
-{name},-{size},-{mtime} - set multiple columns simultaneously

Interaction with Shell

:!<external_command> allows you to run external commands in vifm. The external command is executed in a shell. For example, create a README file with the command below:

:!touch README

echo $SHELL shows which shell program vifm uses.

Macros

Command macros are expanded to values at runtime. Two commonly used macros are shown below:

%c - the current file under the cursor
%f - all selected files

To illustrate macros, use 7z with %f to create a ZIP file:

t - select the files and directories to be zipped
!7z a <desired_zip_file_name> %f - compress the selected files and directories into a ZIP file

htop Essentials

Sun, 16 Aug 2026 15:35:08 +0800

Basic Usage

Start the program with htop. When running inside a desktop terminal emulator, htop supports mouse interaction — you can scroll, click to sort columns, and use the bottom menu.

Kill a Process

Use the arrow keys to move through the process list. By default, Vim motion keys are not working.

To kill a process, press F9 or k and then choose the signal to send.

Like the kill command in shell:

15 SIGTERM - recommended to try first
9 SIGKILL - forcibly terminates the process

SIGKILL may leave behind artifacts such as stale terminal output because the process is terminated without cleanup. Example: terminate htop process with SIGKILL, and there will be leftover contents in terminal window.

Find the Desired Process

Press u to show the list of users, and then narrow down the process list by selecting one.

Press F3 or / to search the process by name. The search is incremental and highlights matches.

Shortcuts

Some commonly used shortcus:

PgUp, PgDn, Home & End - move the cursor as expected
\ - filter the process list by keyword
t - toggle tree view
/ - search by process name
Shift + p - sort by CPU usage
Shift + m - sort by memory usage
H - toggle userland threads
- threads appear under their parent process
- MEM% is a process-level meter, so all threads under the same process show the same value
q - quit htop

UI Setup

Press F2 to open Setup, where you can customize what htop displays. Some useful options include:

Tree view - show processes in parent/children relationships
CPU temporature - display per‑core or overall temperature if supported by your system
CPU usage layout - control how multiple CPU cores are grouped and shown. Example: on a 32-core CPU, selecting CPUs (1-4/8): first half in 4 shorter columns
- - first half - CPU cores from number 0 to number 15
  - in 4 shorter columns - 0-15 cores are shown in 4 columns, so 4 CPU cores per column

fd - the Cmdl Version of Everything

Fri, 14 Aug 2026 15:35:08 +0800

Intro

fd searches files according to name, size, type, etc. Don’t get confused with grep which searches the contents of the file.

Alias

The executable is named fdfind in some old versions, and it’s recommended to set fd as an alias.

alias fd='fdfind'

Usage

Regex is used by default

chsu@claudehsu:~/Sandbox$ fd '.*\.rs$'
RustLab/RustHello/hello.rs
RustLab/rust_hi/src/main.rs

Notice

-i option can be used for case insensitive search.
Files in .gitignore are by default ignored, meaning fd respects .gitignore. Use -I to include those ignored files.
Hidden files are ignored. Use -H to include those hidden files.

Search by File Full Path

With -p option, the regex matches against the file full path, instead of the file name.

chsu@claudehsu:~/Sandbox$ fd -p -i '.*pythonlab.*\.py$'
PythonLab/Scripts/lab.py

Search by File Size

Search by file size using:

+<size>: greater than <size>, such as +1m means greater than 1 MB
-<size>: smaller than <size>

The command below searches .py files whose size is greater than 800KB.

chsu@claudehsu:~/Sandbox$ fd -l -S +800k '.*\.py$'
-rw-rw-r-- 2 chsu chsu 841K Nov  6  2025 ./AILab/Env/lib/python3.14/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/_debug_adapter/pydevd_schema.py
-rw-rw-r-- 2 chsu chsu 841K Jul  7 00:15 ./DuckDBLab/env/lib/python3.14/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/_debug_adapter/pydevd_schema.py

Search by File Type

-t option can be used to specify the file type

-t, --type <filetype>
        Filter the search by type:
          'f' or 'file':         regular files
          'd' or 'dir' or 'directory':    directories
          'l' or 'symlink':      symbolic links
          's' or 'socket':       socket
          'p' or 'pipe':         named pipe (FIFO)
          'b' or 'block-device': block device
          'c' or 'char-device':  character device
          'x' or 'executable':   executables
          'e' or 'empty':        empty files or directories

The command below searches the current directory to find all directories containing the letter ’t'.

chsu@claudehsu:~/Sandbox$ fd -d 1 -t d '.*t.*'
DataLab
PythonLab
RustLab

Explanation

-d specifies the max depth, so -d 1 means to search the current directory only.
-t d specifies to only search for directory entries.

AI 101

Sun, 18 Jan 2026 21:35:08 +0800

What is AI?

Some tasks are easy for computers to perform, such as:

Identifying the maximum number in a dataset
Calculating the shortest route from your office to the subway station

For these types of problems, we can provide clear, step-by-step instructions that computers execute reliably. However, other tasks are much harder to solve with traditional programming:

Is this person happy?
Is that a cat in this image?

While these questions might seem straightforward for humans to answer, they’re surprisingly difficult to program explicitly. Artificial Intelligence (AI) refers to systems that can perform tasks typically requiring human intelligence — such as recognizing patterns, understanding language, and making decisions based on complex data.

Rather than following rigid instructions, AI systems learn from examples and experience, allowing them to handle tasks that are difficult or impossible to solve with conventional programming alone.

What is Machine Learning?

Machine learning is a subset of AI where computers learn patterns from data rather than following explicit instructions. A machine learning system analyzes examples, identifies patterns, and uses what it has learned to make predictions or decisions about new, similar data.

The key difference from traditional programming:

Traditional Programming: You provide explicit rules → the computer follows them → produces output
- Analogy: Following a recipe step-by-step to cook a meal
Machine Learning: You provide examples (data) → the computer finds patterns → creates a model that can handle new situations
- Analogy: Tasting a dish at a restaurant and figuring out the ingredients and techniques (recipe) to recreate it at home

For instance, instead of writing rules like “if the email contains the word ‘prize’ and asks for personal information, it’s spam,” a machine learning system learns what spam looks like by analyzing thousands of examples of spam and legitimate emails, then identifies patterns we might never have thought to program explicitly.

AI, Machine Learning & Deep Learning

You’ve probably heard these terms used interchangeably, but they’re actually nested concepts:

Artificial Intelligence (AI) is the broadest concept — any technique that enables computers to mimic human intelligence
Machine Learning (ML) is a subset of AI—systems that learn from data without being explicitly programmed
Deep Learning (DL) is a subset of Machine Learning — uses neural networks with multiple layers to learn complex patterns

Example to illustrate the difference:

AI: A chess-playing computer (could use rule-based logic or learning)
Machine Learning: A spam filter that improves by analyzing thousands of emails
Deep Learning: Image recognition that identifies cats in photos by learning from millions of images through neural networks

AI vs. Data Science

While AI and Data Science both work with data and often use similar tools, they have different primary goals:

Aspect	AI	Data Science
Primary Goal	Enable machines to perform intelligent tasks (perception, reasoning, decision-making)	Extract insights and knowledge from data to inform decisions
Focus	Automating intelligent behavior	Understanding patterns and answering questions
Key Techniques	Machine learning, neural networks, computer vision, natural language processing	Statistics, data analysis, machine learning, visualization
Typical Output	A system that performs tasks (chatbot, recommendation engine, autonomous vehicle)	Insights, reports, predictions, dashboards
Example Applications	Virtual assistants (Siri, Alexa), self-driving cars, facial recognition	Customer churn analysis, sales forecasting, A/B testing, market research

The overlap: Data scientists often use AI/ML techniques, and AI systems require data science skills to prepare and analyze data. Many professionals work at the intersection of both fields.

Common AI Terms Explained

As AI evolves rapidly, new terminology emerges constantly. Here are some current buzzwords simplified:

LLM (Large Language Model) - AI systems trained on massive amounts of text that can understand and generate human - like language. Examples: ChatGPT, Claude, GPT-4. Think of them as extremely well-read assistants that can write, answer questions, and help with various language tasks.
Agent - An AI system that can take actions autonomously to achieve goals, not just respond to questions. Unlike a basic chatbot that only answers when asked, an agent can plan multiple steps, use tools, and complete complex tasks with minimal human supervision. Example: an AI that researches a topic, writes a report, and emails it to you.
MCP (Model Context Protocol) - A standard way for AI systems to connect with external tools and data sources. It’s like giving your AI assistant the ability to check your calendar, access your files, or pull data from various apps—all through a unified interface.
Skill - A specialized capability or function that an AI system can perform, often by connecting to specific tools or following particular workflows. Think of skills as apps for your AI—each one teaches it how to do something specific like analyzing spreadsheets, generating images, or searching databases.
RAG (Retrieval-Augmented Generation) - A technique where AI retrieves relevant information from a knowledge base before generating a response, making answers more accurate and up-to-date. It’s like giving the AI a library to reference instead of relying only on what it memorized during training.
Fine-tuning - Customizing a pre-trained AI model for specific tasks or domains by training it further on specialized data. Like teaching a general doctor to become a heart surgeon — the foundation is there, but you add expertise in a particular area.

MCP vs. Skill: What’s the Difference?

These terms are related but serve different purposes:

Aspect	MCP (Model Context Protocol)	Skill
What it is	The connection standard or “language” that lets AI communicate with external systems	The actual capability or know-how that AI uses to accomplish specific tasks
Analogy	The USB port on your computer	The USB device you plug in (printer, camera, keyboard)
Technical role	Infrastructure/protocol layer	Application/capability layer
Example	The protocol that allows Claude to connect to Google Drive	The “document editing skill” that knows how to read, analyze, and modify Word files
Focus	How AI connects to tools	What the AI can do with those tools

In practice: An AI system might use MCP to connect to various services (email, calendar, databases), and then use different skills to actually work with the data from those services — like a “meeting scheduler” skill or “data analysis” skill. MCP is the bridge; skills are what you accomplish after crossing it.

Machine Learning 101

Sun, 18 Jan 2026 21:35:08 +0800

What is Machine Learning?

Refer to this link

ML is NOT the solution to every problem. If simple rules work, use them instead.

Main types of ML

Supervised learning - you have data and labels. The algorithm learns by predicting labels and correcting mistakes.
- Classification - categorizing samples (binary for 2 options, multi-class for 3+)
- Regression - predicting numbers (e.g., “how many users will subscribe?”)
Unsupervised learning - you have data but no labels. The algorithm finds patterns, and you interpret them. Example: clustering customers into “summer buyers” and “winter buyers” for targeted promotions.
Transfer learning - reusing a trained model for a new task (e.g., adapting a car recognition model to identify dog breeds). Valuable because training models from scratch is expensive.
Reinforcement learning - the algorithm learns by trial and error within defined rules, earning rewards or penalties. Example: teaching an AI to play chess by updating its score based on moves.

Matching your problem

Supervised learning - you know inputs and outputs
Unsupervised learning - you have inputs but uncertain outputs
Transfer learning - your problem resembles an existing one

Typical Machine Learning Flow

Problem → Data → Evaluation → Features → Modeling → Experiments

Problem - What are we trying to solve?
Data - What data do we have?
Evaluation - What does success look like?
Features - Which variables should we feed into the model?
Modeling - Which model fits our problem best?
Experiments - What else can we try to improve results?

Evaluation

Evaluation defines what success looks like. Common metrics include:

Accuracy - How often is the model correct overall?
$Accuracy = \frac {TP + TN} {TP + TN + FP + FN}$
Precision - When the model predicts positive, how often is it right?
$Precision = \frac {TP} {TP + FP}$
Recall - Of all actual positives, how many did the model catch?
$Recall = \frac {TP} {TP + FN}$

When to prioritize each:

High precision - Use when false positives are costly. Example: spam filters (don’t want important emails marked as spam).
- The model acts more deliberately, so when it reports positive, it is very likely to be actual positive.
High recall - Use when false negatives are costly. Example: cancer detection (missing a case is worse than a false alarm).
- The modle becomes more sensitive. It may reports False Positive, but it tries its best to catch all ‘potential’ positives.

Modeling

Modeling is the core of the ML workflow, consisting of three stages:

Training - train the model on data
Validation - tune the model’s parameters
Test - verify the model’s performance

The dataset is split accordingly:

Set	Split
Training	70% – 80%
Validation	10% – 15%
Test	10% – 15%

A key goal is generalization — a good model performs well on data it has never seen before, producing similar results across all three sets.

Watch out for these two failure modes:

Data Set	Underfitting	Overfitting
Training	62%	95%
Test	50%	60%

Underfitting - poor accuracy across the board; the model is too simple for the problem.
Overfitting - high training accuracy but poor test accuracy; the model memorized the training data instead of learning the underlying pattern.

Commonly Used Tools

Core Python Libraries:

NumPy - numerical computing with arrays and matrices
Pandas - data manipulation and analysis
Matplotlib/Seaborn - data visualization

ML Frameworks:

Scikit-learn - classical ML algorithms (regression, classification, clustering)
TensorFlow - Google’s deep learning framework
PyTorch - Facebook’s deep learning framework, popular in research
Keras - high-level neural network API (now multi-backend with TensorFlow, PyTorch and JAX)

Development Tools:

Jupyter Notebook - interactive coding environment
Google Colab - free cloud-based Jupyter notebooks with GPU access
Anaconda/Miniconda - Python distribution with package management

Optional but Popular:

MLflow - experiment tracking and model management
Hugging Face - pre-trained models for NLP tasks

Scala Quick Guide

Sun, 07 Dec 2025 21:35:08 +0800

Fundamental

Primitive Types

Primitive types in Scala are like the ones in Java, but they are capitalized

Int: 32-bit integers
Long: 32-bit integers
Float: 32-bit floating point numbers
Double: 64-bit floating point numbers
Char: 16-bit unicode characters
Short: 16-bit integers
Byte: 8-bit integers
Boolean: boolean values true and false

scala> val x = 23
val x: Int = 23

Expression

Scala supports functional programming quite well. Many items are expressions in Scala, though in many other languages they are not treated as expressions. For example the if...then...else expression

val x = if true then 10 else 20

A block is denoted by a pair of curly braces {}. A block is an expression, and it uses the last expression as its value.

val x = {
  val y = 10
  y + 5
}
// the block is evaluated to 15, so x is 15

In Scala 3, braces are optional, so below code snippet is valid.

scala> val x2 =
     |  val y2 = 20
     |  y2 * 3
     |
val x2: Int = 60

To declare a variable, use var. However, following functional programming style, mutation is encouraged, and use val in immutation case.

val x = 1
val y: Int = 2
var sum = 0

Evaluation

To evaluate an expression

Take the leftmost operator.
Evaluate its operands, left to right for multiple operands case.
Apply the operator to the operands.

There are two evaluation strategies regarding function invocation

Applicative order(call by value): reduce function arguments to values before rewriting function application
- by default used in Scala
- the expression evaluates at most once
Normal order(call by name): apply function directly without reducing arguments first
- use => to indicate normal order before parameters in function signature
- no need to do unnecessary evaluation

The two evaluation strategies reduce to the same final result as long as

The reduce expression consists of pure function.
Both evaluations terminate.

Below is an example showing how ‘impure’ function evaluates to different results in applicative order and normal order.

import scala.util.Random

def squareA(x: Int): Int = x * x
def squareN(x: => Int): Int = x * x

@main
def main = {
  val rnd = new Random()
  val resultA = squareA(rnd.nextInt(10)) 
  println(s"Got a square(A) value: $resultA")
  // below evaluates to `rnd.nextInt(10) * rnd.nextInt(10)`
  val resultN = squareN(rnd.nextInt(10)) 
  println(s"Got a square(N) value: $resultN")
}

Normal order evaluation does NOT do unnecessary evaluation.

// define infinite loop
def loop: Boolean = loop

// define normal order function
def constOne(x: Int, y: => Boolean) = 1

// y follows normal order evaluation, so it is NOT evaluated at all.
constOne(2, loop) // won't trigger infinite loop

Scala uses short-circuit evaluation for && and ||. As a result, true || loop terminates.

Recursion

Recursion is widely used in FP, but it prevents the compiler from inferring the function’s return type, requiring an explicit type declaration in the function signature.

def fact(n: Int): Int =
  require(n >= 0, "parameter should NOT be negative")
  if n == 0 then 1 else n * fact(n-1)

Tail Recursion

Tail recursion is supported in Scala. The @tailrec annotation is more like a reminder, because even if the annotation is not present, the Scala compiler will still try to make recursion tail-recursive whenever possible

def fact(n: Int): Int =
  require(n >= 0, "parameter should NOT be negative")

  @tailrec
  def factTR(curr: Int, accu: Int): Int = 
    if curr == 0 then accu else factTR(curr-1, accu*curr)
  factTR(n, 1)

Higher Order Function

Functions are treated as first-class citizens in Scala. That means functions can be parameters and return values of functions. Functions that take other functions as parameters or return functions as return values are called higher order functions(HOF).

But why does that matter? HOF provides a flexible way to compose functions.

def sumOfSquare(lowerBound: Int, upperBound: Int): Int =
  require(lowerBound <= upperBound, "Lower bound should NOT exceed upper bound.")
  if lowerBound == upperBound then lowerBound*lowerBound else lowerBound*lowerBound + sumOfSquare(lowerBound+1, upperBound)

def sumOfDouble(lowerBound: Int, upperBound: Int): Int =
  require(lowerBound <= upperBound, "Lower bound should NOT exceed upper bound.")
  if lowerBound == upperBound then upperBound*2 else lowerBound*2 + sumOfDouble(lowerBound+1, upperBound)

The behavior of above two functions is very similar to each other. We can ’extract’ the different part, and introduce a function to ‘abstract’ the behavior like below

def sumOfFn(lowerBound: Int, upperBound: Int, fn: Int => Int): Int =
  require(lowerBound <= upperBound, "Lower bound should NOT exceed upper bound.")
  if lowerBound == upperBound then fn(lowerBound) else fn(lowerBound) + sumOfFn(lowerBound+1, upperBound, fn)

# to get sum of square from 3 to 6
sumOfFn(3, 6, x => x*x)

There is still hard code in above implementation, the + operation. If we extract that part, and introduce a function, we actually implement the map-reduce operation.

def mapReduce(lowerBound: Int, upperBound: Int, map: Int => Int, reduce: (Int, Int) => Int): Int =
  require(lowerBound <= upperBound, "Lower bound should NOT exceed upper bound.")
  if lowerBound == upperBound then map(lowerBound)
  else reduce(map(lowerBound), mapReduce(lowerBound+1, upperBound, map, reduce))

# sum of square from 3 to 6
mapReduce(3, 6, x => x*x, (a, b) => a+b)

Anonymous Functions

In above sumOfFn and mapReduce cases, anonymous functions are passed as arguments to the functions.

val doubleFn = (x: Int) => x * 2

Note

To explicitly declare the return type of function like (x: Int): Int => x*2 is NOT legal.
Usually the return type of the function can be inferred, but if we’d like to explicitly denote the return type, we declare it on the identifier like val cube: Int => Int = x => x*x*x

Function Currying

When invoking above mapreduce function, we need to provide all four parameters. Think about below cases

sum of square from 3 to 5
sum of double from 2 to 6
sum of cube from 5 to 7

In above cases, the reduce function is sum, and we need to repeatedly provide it in each invocation. It would be better if we can make an ‘intermediate’ function that is fixed to sum as reduce logic. To do that we utilize function currying.

// mapReduce via function currying
def mapReduce(reduce: (Int, Int) => Int) (map: Int => Int) (lowerBound: Int, upperBound: Int): Int =
  require(lowerBound <= upperBound, "Lower bound should NOT exceed upper bound.")
  if lowerBound == upperBound then map(lowerBound)
  else reduce(map(lowerBound), mapReduce(reduce)(map)(lowerBound+1, upperBound))

// As usual we can provide all the parameters
mapReduce((a: Int, b: Int)=>a+b) ((x:Int)=>x*x) (3,5)

The mapReduce function has type ((Int, Int) => Int) => (Int => Int) => (Int, Int) => Int, which forms a chain of functions.

scala> mapReduce
val res0: ((Int, Int) => Int) => (Int => Int) => (Int, Int) => Int = Lambda/0x0000026581544c10@5b74902c

That makes partial application available. Partial application returns an ‘intermediate’ function. Note

Function types associate to the right, so Int => Int => Int means Int => (Int => Int).
Function invocation assoticates to the left, so myFun(1)(2) means (myFun(1))(2).

// partial application with `reduce` function provided
scala> val sumOfFn = mapReduce((a: Int, b: Int) => a+b)
val sumOfFn: (Int => Int) => (Int, Int) => Int = Lambda/0x0000026581570410@1397b141

scala> sumOfFn((x: Int) => x*x)(3,5)
val res1: Int = 50

scala> sumOfFn((x: Int) => x*x*x)(3,5)
val res2: Int = 216

// further partial applicaiton with `map` function provided
scala> val sumOfQuad = sumOfFn((x: Int) => x*4)
val sumOfQuad: (Int, Int) => Int = Lambda/0x0000026581572000@2bc8caa7

scala> sumOfQuad(3, 6)
val res3: Int = 72

scala> sumOfQuad(5, 9)
val res4: Int = 140

Object Oriented Programming with Scala

Scala can be used in functional programming style, but it also supports object oriented programming well.

Simple Introduction to Classes

To define a class

class Robot(val robotName: String, height: Int):
  val robotHeight = height
  def introSelf(target: Robot) : String =
    s"Hello ${target.robotName}, I am ${this.robotName}"
    
@main
def main =
  val arale = new Robot("Arale", 139)
  val wallE = new Robot("wallE", 100)
  println(arale.introSelf(wallE))
  println(s"WallE's height is ${wallE.robotHeight}")

Note

Don’t forget the colon : at the end of the ‘class definition’ line.
val robotName: String makes robotName a public field and available from outside of the class.
- Without val in front of robotName, robotName would not be treated as a filed, it’s only visible inside of the class, and target.robotName would cause syntax error.
robotHeight is defined as a value, which represents one attribute of the class.
this keyword represents the current object of the class.

Data Abstraction with Classes

Classes can be used to describe data and behavior. For example, below class represents a rational number

// define a class without explicit class body
class Rational(val numerator: Int, val denominator: Int)

// try it in Scala REPL
scala> val r1 = Rational(15, 55)
val r1: Rational = Rational@a4dcede

scala> r1.numerator
val res10: Int = 15

scala> r1.denominator
val res11: Int = 55

Explanation

val before the class parameters (numerator and denominator) make those parameters available to the outside of the class. Otherwise, they can only be visited inside of the class.

Reduction of fraction is a common operation to rational numbers.

class Rational(val numerator: Int, val denominator: Int):
  // greatest common divisor
  private def gcd(a: Int, b: Int): Int =
    if b == 0 then a else gcd(b, a%b)
  def numer = numerator/gcd(numerator, denominator)
  def denom = denominator/gcd(numerator, denominator)
  override def toString = s"$numerator/$denominator[$numer/$denom]"

// try in REPL
scala> val r1 = Rational(15, 55)
val r1: Rational = 15/55[3/11]

In above implementation, gcd is repeatedly called each time numer or denom is called. To prevent that, refine the implementation as

// refine Rational implementation to remove duplicate call on gcd
class Rational(val numerator: Int, val denominator: Int):
  private def gcd(a: Int, b: Int): Int =
    if b == 0 then a else gcd(b, a%b)
  private val divisor = gcd(numerator, denominator)
  def numer = numerator/divisor
  def denom = denominator/divisor
  override def toString = s"$numerator/$denominator[$numer/$denom]"

// try in REPL
scala> val r2 = Rational(27, 90)
val r2: Rational = 27/90[3/10]

From the caller perspective, it may not realize the refinement at all, because the way how it interacts with Rational remains the same. The ability to choose different implementations of the data without affecting clients is called data abstraction.

Extension Methods

Extension methods allow use toe expand the behavior of a class. Some typical cases are

Operations that feel like they “should” be part of a type but aren’t.
Domain-specific operations on thrid-party or standard library types, when we cannot modify the original class but want to add domain-specific behavior.

For example, in above Rational case, suppose Rational class is defined as a standard library class. We cannot modify the “original” standard libraray code, but we can utilize extention methods to implement operations like abs, min and etc., so that we “expand” the feature of Rational.

class Rational(val numerator: Int, val denominator: Int):
  private def gcd(a: Int, b: Int): Int =
    if b == 0 then a else gcd(b, a%b)
  private val divisor = gcd(numerator, denominator)
  def numer = numerator/divisor
  def denom = denominator/divisor
  override def toString = s"[$numer/$denom]"

extension (r: Rational)
  def abs: Rational = Rational(r.numer.abs, r.denom.abs)
end extension

// run in REPL
scala> val myNumber = new Rational(-2, 5)
val myNumber: Rational = [-2/5]

scala> myNumber.abs
val res0: Rational = [2/5]

Note

Unlike class, there is no trailing colon on extension line.
r in extension(r: Rational) is known as extension parameter.
Define parameterless function abs without a pair of empty parentheses to indicate no side effect.
Extentions can only add new members, not override existing ones.

Different people may define extensions in different modules so that they can have their own versions of enhancement that don’t run into others. Extensions of a class become available if they are

defined or imported in the current scope
listed in the companion object

Infix Notation

Operators like +, - and etc. are identifiers, and they can be used as function names.

class Rational(val numerator: Int, val denominator: Int):
  private def gcd(a: Int, b: Int): Int =
    if b == 0 then a else gcd(b, a%b)
  private val divisor = gcd(numerator, denominator)
  def numer = numerator/divisor
  def denom = denominator/divisor
  override def toString = s"[$numer/$denom]"
  def +(o: Rational): Rational =
    new Rational(numer*o.denom+denom*o.numer, denom*o.denom)

extension (r: Rational)
  def abs: Rational = Rational(r.numer.abs, r.denom.abs)
  def -(o: Rational) = 
    new Rational(r.numer*o.denom-r.denom*o.numer, r.denom*o.denom)
end extension

With above + method and - method defined, they can be used as infix notation

// run in REPL
scala> val r1 = new Rational(2,3)
val r1: Rational = [2/3]

scala> val r2 = new Rational(1,5)
val r2: Rational = [1/5]

scala> r1 + r2
val res0: Rational = [13/15]

scala> r1 - r2
val res1: Rational = [7/15]

Explanation

Those infix notation functions can be defined in either classes or extensions.
If the identifier is not symbolic, such as min, the leading infex modifer is mandatory.

extension (r: Rational)
  def abs: Rational = Rational(r.numer.abs, r.denom.abs)
  def -(o: Rational) = 
    new Rational(r.numer*o.denom-r.denom*o.numer, r.denom*o.denom)
  infix def min(o: Rational) =
    if r.numer * o.denom <= o.numer * r. denom
    then r
    else o
end extension

Abstraction and Class Hierarchies

Programming to abstractions rather than concrete implementations is a cornerstone principle behind most design patterns. In Scala, we can define abstract classes to follow that principle. Suppose we need a type to represent a set of integer numbers. We can have below abstraction.

abstract class IntSet:
  def include(x: Int): IntSet
  def contains(x: Int): Boolean

One implementation for above IntSet is binary trees. For instance, if we have the IntSet {1, 2, 4, 5}, below is the corresponding binary tree

Obviously two types of nodes are needed.

Empty node
NonEmpty node

class Empty() extends IntSet:
  def include(x :Int): IntSet = NonEmpty(x, Empty(), Empty())
  def contains(x: Int): Boolean = false
  override def toString = "<E>"

class NonEmpty(elem:Int, left: IntSet, right: IntSet) extends IntSet:
  def include(x: Int): IntSet =
    if x > elem then NonEmpty(elem, left, right.include(x))
    else if x < elem then NonEmpty(elem, left.include(x), right)
    else this
  def contains(x: Int): Boolean =
    if x > elem then right.contains(x)
    else if x < elem then left.contains(x)
    else true
  override def toString = s"($elem, ${left.toString}, ${right.toString})"

Explanation

extends keyword denotes the subclass-superclass relationship.
- A class defined without extends actually extends Objct class, which is java.lang.Object.
- The direct or indirect superclass of a class is called base class. For instance, IntSet and Object are base classes of Empty class.
override is a modifer, instead of an annotation in Scala. It denotes to override the exsiting method defined in base class.

// run in REPL
scala> val intSetA = Empty().include(1).include(4).include(5).include(2)
val intSetA: IntSet = (1, <E>, (4, (2, <E>, <E>), (5, <E>, <E>)))

scala> val intSetB = Empty().include(4).include(1).include(5).include(2)
val intSetB: IntSet = (4, (1, <E>, (2, <E>, <E>)), (5, <E>, <E>))

`object` for Singleton

In above IntSet implementation, Empty is defined as a class, meaning multiple instances of Empty will be created, and that’s a waste. To fix that defect, use object type to implement singleton.

object Empty extends IntSet:
  def include(x :Int): IntSet = NonEmpty(x, Empty, Empty)
  def contains(x: Int): Boolean = false
  override def toString = "<E>"
end Empty // optional end marker

Note

Empty now is object type and we cannot create instance of it anymore. To reference it, simply use Empty.
Singleton objects are values, so Empty evaluates to itself.

Companion object

When a class type and an object type are defined in the same source file with the same name, they are called companions. Note

The companion object must be defined in the same source file as its companion class.
If they are defined in separate files, they are just a regular class and a regular objct that happen to have the same name.

A companion object of a class plays a role similar to static class definitions in Java.

object for Program Entry Point

To build a standalone application/program, create an object type with main method

object Hello:
  def main(args: Array[String]): Unit =
    println("Hello from Scala!")

A common best practice is to name the source file after the object it defines, so name the source file Hello.scala

To execute the program

C:\sandbox\ScalaLab\Scripts> scala run Hello.scala
Compiling project (Scala 3.7.4, JVM (21))
Compiled project (Scala 3.7.4, JVM (21))
Hello from Scala!

Alternatively, we can compile the source to class files using scala compile, and then run the program. Notice in Scala 3, powered by Scala CLI, a unifed scala command with subcommands like compile, run, repl, test, etc. are introduced. The old scalac and scala still exist for backward compatibility.

Using scala compile, class files are generated under .scala-build/ directory by default.

Dynamic Method Binding

Method invocation depends on the runtime type of the object.

class Robot(val name: String):
  def greet(target: String): String =
    s"Hi $target, this is Robot[$name]."

class CleaningRobot(name: String) extends Robot(name):
  override def greet(target: String): String =
    s"Greeting from CleaningRobot[$name]: Hello $target."

@main
def main() =
  val wallE: Robot = new CleaningRobot("Wall-E")
  println(wallE.greet("EVE"))

Explanation

wallE is declared as Robot type, but assigned to an instance of CleaningRobot.
The method invocation depends on the runtime type, so the greet defined in CleaningRobot is invoked.

Organize Classes with Packages

Classes and objects are organized in packages. To place a class/object type inside a package, use package keyword at the top of the source file. For example

package id.my.cxu.examples

object Hello:
  def main(args: Array[String]): Unit =
    println("Hello from Scala!")
    
  def sayHello(target: String) = s"Hello, $target"

Now compile the source file.

C:\sandbox\ScalaLab\Scripts> scala compile Hello.scala
Compiling project (Scala 3.7.4, JVM (21))
Compiled project (Scala 3.7.4, JVM (21))

C:\sandbox\ScalaLab\Scripts> fd -H .*Hello
.scala-build/.bloop/Scripts_86c27f223b-f4d7975d63/bloop-internal-classes/main-3F_GeVW0Sx2WzkmaZblELw==/id/my/cxu/examples/Hello$.class
.scala-build/.bloop/Scripts_86c27f223b-f4d7975d63/bloop-internal-classes/main-3F_GeVW0Sx2WzkmaZblELw==/id/my/cxu/examples/Hello.class
.scala-build/.bloop/Scripts_86c27f223b-f4d7975d63/bloop-internal-classes/main-3F_GeVW0Sx2WzkmaZblELw==/id/my/cxu/examples/Hello.tasty
.scala-build/Scripts_86c27f223b-f4d7975d63/classes/main/id/my/cxu/examples/Hello$.class
.scala-build/Scripts_86c27f223b-f4d7975d63/classes/main/id/my/cxu/examples/Hello.class
.scala-build/Scripts_86c27f223b-f4d7975d63/classes/main/id/my/cxu/examples/Hello.tasty
Hello.scala

Explanation

One common best practice is to construct the same directory structure for srouce file. So in above case, the Hello.scala is supposed to be put under id/my/cxu/examples, but that’s not mandatory.
In above example, the ‘qualified name’ of the Hello is id.my.cxu.examples.Hello. The compilation operation generates the directories according to the package.

To reference to the type, use the qualified name including both package name and class name.

@main
def main() =
  val greeting = id.my.cxu.examples.Hello.sayHello("Wall-E")
  println(greeting)

Execute above script file (namely lab.scala) with -cp to indicate the path of the dependent class file.

C:\sandbox\ScalaLab\Scripts> scala run -cp .scala-build\Scripts_86c27f223b-f4d7975d63\classes\main lab.scala
Compiling project (Scala 3.7.4, JVM (21))
Compiled project (Scala 3.7.4, JVM (21))
Hello, Wall-E

Arguably, a better way to run the script is to provide both the entry point and the dependent files together. However, in above case, there are two ’entry points’. To prove that point, use the command scala run --main-class-ls <scala_files> to list the available entry points of the scala files.

C:\sandbox\ScalaLab\Scripts> scala run --main-class-ls Hello.scala lab.scala
Warning: setting C:\Sandbox\ScalaLab\Scripts as the project root directory for this run.
Compiling project (Scala 3.7.4, JVM (21))
Compiled project (Scala 3.7.4, JVM (21))
id.my.cxu.examples.Hello main

Explanation: the command returns two entry points

id.my.cxu.examples.Hello
- Now we need to use the qualified name, instead of simply Hello to refer to the Hello object.
main

The desired entry point in this case is main, so we execute the command scala run -M main Hello.scala lab.scala

C:\sandbox\ScalaLab\Scripts> scala run -M main Hello.scala lab.scala
Warning: setting C:\Sandbox\ScalaLab\Scripts as the project root directory for this run.
Hello, Wall-E

Explanation

-M option denotes the ’entry point’/‘main class’ to run. In above script case, namely that’s main.

Import Classes

Using the qualified name of a class seems combersome, and we can ‘import’ the class to simplify the reference form. There are two types import

named import, such as import id.my.cxu.examples.Hello to just import the Hello object
wildcard import, such as import id.my.cxu.examples._ to import everything in the id.my.cxu.examples package.
- Notice the wildcard is underscore, NOT asterisk.

We can import types from either of the following:

a package
- conceptually, a package is a ’large’ object that contains many members in several individual source files.
an object

So refine above script file as

@main
def main() =
  import id.my.cxu.examples.Hello
  val greeting = Hello.sayHello("Arale")
  println(greeting)

Notice some items are automatically imported in every Scala program.

members of package scala
members of package java.lang
members of the singleton object scala.Prefe

That’s the reason why some types and functions seem available ’natually’, for example

Int (defined inscala.Int)
Boolean (defined in scala.Boolean)
Object (defined in java.lang.Object)
require (defined in scala.Predef.require)
assert (defined in scala.Predef.assert)

Traits

A class may have several natural supertypes, and that is implemented with traits in Scala. A class can inherit from

at most one class
arbitrary many traits

A trait is declared like an abstract class

trait Movable:
  def move(): Unit

trait Soundable:
  def sound: String

class Robot extends Movable, Soundable:
  def move() = println("Robot is moving")
  def sound = "Robot is talking"

@main
def main =
  val arale = new Robot()
  arale.move()
  println(arale.sound)

Explanation

Class Robot extends two traits in above example.

Class Hierarchy

Explanation

scala.AnyVal is the super type of all the primitive types.
scala.AnyRef is the synonym of java.lang.Object, and it’s the super type of the user defined class.
Root type of the type system is scala.Any.
- Methods like ==, !=, equals, hashCode, toString are defined there.
Bottom type of the type system is scala.Nothing.
- Nothing is the subtype of every other type.
- Nothing is a type, and there is no value of type Nothing, so we cannot assign Nothing to an identifier.
- Use case of Nothing type: as an element type of empty collections.
Notice two types of arrows
- Solid line arrows: conform to the super class
- Dashed line arrows: implicit conversion

A Tricky Type Question

Given below expression, what’s the type of it?

if true then 1 else false

In Scala 2, we get the common supertype of Int and Boolean, so we end up getting type AnyVal. However in Scala 3, we get Int | Boolean, the union type, which represents a value that can be either an Int or Boolean.

By design, the compiler does NOT look into the condition, although it’s even fixed as true.

Polymorphism

Polymorphism is one of the pillars of OOP. To help understand and remember the word

poly: it means many
morphism: it means forms

In programming, polymorphism means

the identify of a certain instance can be viewed as many different types
the function can be applied to arguments of many types

class Robot(val name: String, val model: String):
  def move(): Unit = println("Robot is moving...")

class HumanoidRobot(name: String, model: String) extends Robot(name, model):
  def talk(): Unit = println(s"HumanoidRobot[$name] is talking...")

class Arale(robotID: String) extends HumanoidRobot("Arale-"+robotID, "Arale"):
  override def move(): Unit = println(s"$name is running...")

@main
def main() =
  val a01: Robot = new Arale("a01")
  a01.move()
  val a02: HumanoidRobot = new Arale("a02")
  a02.talk()
  a02.move()


// running above program and we get
scala run PolyLab.scala
Compiling project (Scala 3.7.4, JVM (21))
Compiled project (Scala 3.7.4, JVM (21))
Arale-a01 is running...
HumanoidRobot[Arale-a02] is talking...
Arale-a02 is running...

Explanation

a01 is declared as type Robot, but in fact it is an instance of Arale. Due to dynammic method binding, the effective method for a01.move() invocation is the one defined in Arale.
a02 is an instance of Arale, and it can be recognized as an instance of HumanoindRobot as well, so a02.talk() works.

Generics

Suppose we need to implement the linked list data structure. Below is a simple design

Nil to represent the empty list
Cons to represent a cell containing an element, and the remainder (tail) of the list

For a list like List(1, 2, 3), above design looks like

The implementation can be

import java.{util => ju}

trait IntList:
  def isEmpty: Boolean
  def head: Int
  def tail: IntList

class Cons(val head: Int, val tail: IntList) extends IntList:
  def isEmpty: Boolean = false
  override def toString = s"$head | ${tail.toString}"

object Nil extends IntList:
  def isEmpty = true
  def head = throw new ju.NoSuchElementException()
  def tail = throw new ju.NoSuchElementException()
  override def toString = "<Nil>"

@main
def main() =
  val myList = Cons(1, Cons(2, Cons(3, Nil)))
  println(myList)


// execute and we get
scala run IntList.scala
Compiling project (Scala 3.7.4, JVM (21))
Compiled project (Scala 3.7.4, JVM (21))
1 | 2 | 3 | <Nil>

Explanation

val head: Int and val tail: IntList of Cons fulfill the head and tail methods defined in IntList.
To create multiple instances of Nil does NOT make a lot of sense, so define it as object type.

It’s easy to create types like BooleanList, StringList, etc. In fact, the only difference among those types is the type of the inner stored elements. To ‘generalize the type’, we can use Type Parameters.

import java.{util => ju}

trait MyList[+T]:
  def isEmpty: Boolean
  def head: T
  def tail: MyList[T]

class Cons[T](val head: T, val tail: MyList[T]) extends MyList[T]:
  def isEmpty: Boolean = false
  override def toString = s"$head | ${tail.toString}"

object Nil extends MyList[Nothing]:
  def isEmpty = true
  def head = throw new ju.NoSuchElementException()
  def tail = throw new ju.NoSuchElementException()
  override def toString = "<Nil>"

@main
def main() =
  val myNumList: MyList[Int] = Cons(1, Cons(2, Cons(3, Nil)))
  println(myNumList)
  val myStrList : MyList[String] = Cons("Hello", Cons("From", Cons("Scala", Nil)))
  println(myStrList)

Explanation

The plus sign in MyList[+T] makes MyList covariant, which means an instance of MyList[DerivedType] can be considered as an instance of MyList[BaseType].
object Nil extends MyList[Nothing] makes Nil to be an sub-type of MyList[Nothing], which in turn makes Nil an sub-type of any MyList[+T].
- Technically, object Nil extends MyList also works, but explicitly denoting Nothing makes the type relationship crystal clear.

Objects for OOP

In pure object-oriented programming language, every value is an object. To understand the point, let’s follow this idea to create an ideal version of Boolean type and values.

abstract class IdealBoolean:
  def ifThenElse[T](t: => T, e: => T): T
  
  def &&(x: => IdealBoolean): IdealBoolean =
    ifThenElse(x, IdealFalse)
    
  def ||(x: => IdealBoolean): IdealBoolean = 
    ifThenElse(IdealTrue, x)
    
  def unary_! : IdealBoolean =
    ifThenElse(IdealFalse, IdealTrue)
    
  def ==(x: => IdealBoolean): IdealBoolean =
    ifThenElse(x, x.unary_!)
    
  def !=(x: => IdealBoolean): IdealBoolean =
    ifThenElse(x.unary_!, x)

object IdealTrue extends IdealBoolean:
  def ifThenElse[T](t: => T, e: => T): T = t
  
object IdealFalse extends IdealBoolean:
  def ifThenElse[T](t: => T, e: => T): T = e

Explanation

IdealBoolean is an abstract class defining the behavior of the type.
There are only two possible values for IdealBoolean type, IdealTrue and IdealFalse. It doesn’t make a lot of sense to create different instances of IdealTrue and IdealFalse, so they are defined as object

Functions as Objects

In Scala, a function can be treated as an object. The function type A => B is just an abbreviation of the type scala.Function[A, B], just as below

package scala

trait Function1[A, B]:
  def apply(x: A): B

Explanation

Functions are objects with apply methods.
The name of the trait in above example is Function1, and there are traits named as Function2, Function3, etc., for functions which have more parameters.

Let’s create equivalent integer sum functions using anonymous function and Function2 trait

@main
def main() =
  val intSumFn: (Int, Int) => Int = (a, b) => a + b
  val intSumFn2: Function2[Int, Int, Int] = new Function2[Int, Int, Int]:
      def apply(a: Int, b: Int): Int = a + b

  val cmpr = intSumFn(2,3) == intSumFn2(2,3)
  println(cmpr) // true
  println(intSumFn == intSumFn2) // false

Explanation

Funciton2 is an trait, so it’s impossible to create a ‘direct’ instance. As a result, ‘anonymous class’ is used to create an ‘indirect’ instance of Function2.
Although logically intSumFn and intSumFn2 are the same thing, they are not equivalent.

With above concept in mind, we now have two ways to invoke a function

regular function call such as fn(args)
using apply like fn.apply(args)

@main
def main() =
  val intSumFn: (Int, Int) => Int = (a, b) => a + b
  println(intSumFn(5, 91))
  println(intSumFn.apply(5, 91))

  def intSumFn3(a: Int, b:Int): Int = a + b
  println(intSumFn3(5, 23))
  println(intSumFn3.apply(5, 23))

Exaplantion

intSumFn is an anonymous function and intSumFn3 is a regular function. We can use either regular function call or the apply function of the function object.

OO Decomposition vs Pattern Matching

Suppose we need to build a dummy interpretor and for demo purpose, let’s consider below entities with integer nubmers for now

Expr: the abstract type to represent expressions, and for now only consider integer numbers
Number: to represent numbers, and for demo purpose only consider integer numbers for now
Sum: to represent addition operation, and for now only consider integer numbers

OO Decomposition

One solution to build the interpretor is like below, via OO decomposition

trait Expr:
  def eval: Int

class Number(n: Int) extends Expr:
  def eval: Int = n

class Sum(left: Expr, right: Expr) extends Expr:
  def eval: Int = left.eval + right.eval

@main
def main() =
  val e1 = Number(2)
  val e2 = Sum(Number(2), Number(5))
  val r = Sum(e1, e2).eval
  print(r)

Explanation

Expr is defined as a trait and it is abstract type.
Number contains both data(n) and operation(eval).
Similar to Number, Sum contains both data(left and right) and operation(eval)

But why is it called OO decomposition, given the fact that data is not separated from opertions?

“Decomposition” here means to decompose a complex system into cohesive units with clear responsibilities. So in OO territory, it’s the idea of breaking a system down into objects, each representing a concept in the problem domain. An object encapsulates both state(data) and behavior(operations).

Above implementation decompose the problem domain into Expr trait, Number class and Sum class. The responsiblity is clear, so to some extent that’s a good design. Suppose we need to add new entities/data such as Prod to represent multiplication, it’s still quite easy.

class Prod(left: Expr, right: Expr) extends Expr:
  def eval: Int = left.eval * right.eval

However if we need to add a new behavior, such as a display method, now that’s a problem, because every type needs to be updated.

trait Expr:
  def eval: Int
  def display: String

class Number(n: Int) extends Expr:
  def eval: Int = n
  def display: String = n.toString

class Sum(left: Expr, right: Expr) extends Expr:
  def eval: Int = left.eval + right.eval
  def display: String = s"${left.display} + ${right.display}"

@main
def main() =
  val e1 = new Number(2)
  val e2 = new Sum(new Number(2), new Number(5))
  val e = new Sum(e1, e2)
  print(s"${e.display} = ${e.eval}")

Note: using OO decomposition, it’s

easy to add new kinds of entities/data
hard to add new operations

FP Decomposition - Pattern Matching

Another approach to implement the interpretor is to utilize pattern matching.

trait Expr
case class Number(n: Int) extends Expr
case class Sum(left: Expr, right: Expr) extends Expr

def eval(expr: Expr): Int = expr match
  case Number(n: Int) => n
  case Sum(left: Expr, right: Expr) => eval(left) + eval(right)

@main
def main() =
  val e1 = Number(91)
  val e2 = Sum(Number(2), Number(3))
  val e = Sum(e1, e2)
  println(s"${eval(e)}")

Explanation

We use case classes to represent numbers and operations, because classes have below ‘automatic’ features
- immutability by default: all constuctor parameters are automatically val fields (public and immutable).
- proper equals() and hashCode(): automatically get proper euqals() and hashCode() to compare based on field values, not object identity.
- pattern matching support: can be used in pattern matching with automatically generated unapply method.
- companion object: automatically get a companion object with an apply method, so new keyword is not needed in instantiation.
- copy method: automatically get a copy() method that creates modified copies while keeping other fields the same to support working with immutable data.
- toString implementation: automatically get a readable toString implementation showing class name and field values.
Pattern matching in above example
- match is preceded by a selector expression, in above case expr.
- A MatchError is thrown if no pattern matches the value of the selector expression.
- Patterns are constructured from
  - constructors, e.g. Number(n)
  - variables, e.g. n, left
  - wildcard patterns, e.g. _ to match anything but don’t care about the value
  - constants, e.g. 1, true
  - type test, e.g. n: Int
- some ruels and conventions of patterns
  - Variables always begin with a lowercase letter.
  - Names of constants begin with a Capital letter, with the exception of the reserved words like null, true, false.
  - The same variable name can only appear once in a pattern, so Sum(x, x) is NOT a valid pattern.

Now think about below bullets

Move eval function into Expr trait to from a eval method.
What if we’d like to add a display behavior?
What if we’d like to add a Prod calculation?

trait Expr:
  
  def eval: Int = this match
    case Number(n: Int) => n
    case Sum(left: Expr, right: Expr) => left.eval + right.eval
    
  def display: String = this match
    case Number(n) => n.toString
    case Sum(l, r) => s"${l.display} + ${r.display}"
    
case class Number(n: Int) extends Expr
case class Sum(left: Expr, right: Expr) extends Expr


@main
def main() =
  val e1 = Number(91)
  val e2 = Sum(Number(2), Number(3))
  val e = Sum(e1, e2)
  println(s"${e.display} = ${e.eval}")

Explanation

eval is transformed from a function to a method of Expr trait. Instead of accepting the expr parameter for pattern matching, it use the current instance, namely this, directly.
display is added to Expr trait as another method. It also utilizes pattern matching technique.

It’s not that difficult to implement the first two bullets. However, to add additional data/entities, in this case Prod, all the pattern matching logic needs to be updated to ‘include’ that new data.

trait Expr:
  
  def eval: Int = this match
    case Number(n: Int) => n
    case Sum(left: Expr, right: Expr) => left.eval + right.eval
    case Prod(left: Expr, right: Expr) => left.eval * right.eval
    
  def display: String = this match
    case Number(n) => n.toString
    case Sum(l, r) => s"${l.display} + ${r.display}"
    case Prod(l:Sum, r:Sum) => s"(${l.display}) * (${r.display})"
    case Prod(l, r:Sum) => s"${l.display} * (${r.display})"
    case Prod(l:Sum, r) => s"(${l.display}) * ${r.display}"
    case Prod(l, r) => s"${l.display} * ${r.display}"
    
case class Number(n: Int) extends Expr
case class Sum(left: Expr, right: Expr) extends Expr
case class Prod(left: Expr, right: Expr) extends Expr


@main
def main() =
  val e1 = Number(91)
  val e2 = Sum(Number(2), Number(3))
  val e = Prod(e1, e2)
  println(s"${e.display} = ${e.eval}")

Note: using FP decomposition, pattern matching, it’s

easy to add new operations
hard to add new kinds of entities/data

Back to the implementation. The 4 adjacent Prod patterns look a little awkard. To improve the implementation, we can do a further pattern matching like below

trait Expr:
  
  def eval: Int = this match
    case Number(n: Int) => n
    case Sum(left: Expr, right: Expr) => left.eval + right.eval
    case Prod(left: Expr, right: Expr) => left.eval * right.eval
    
  def display: String = this match
    case Number(n) => n.toString
    case Sum(l, r) => s"${l.display} + ${r.display}"
    case Prod(l, r) => s"${{addParenthesesIfNecessary(l)}} * ${{addParenthesesIfNecessary(r)}}"
    
case class Number(n: Int) extends Expr
case class Sum(left: Expr, right: Expr) extends Expr
case class Prod(left: Expr, right: Expr) extends Expr

def addParenthesesIfNecessary(e: Expr): String = e match
  case Sum(l, r) => s"(${l.display} + ${r.display})"
  case _ => e.display

@main
def main() =
  val e1 = Number(91)
  val e2 = Sum(Number(2), Number(3))
  val e = Prod(e2, e1)
  println(s"${e.display} = ${e.eval}")

Expalnation

addParenthesesIfNecessary adds parentheses to Sum expression.
_ pattern matches to any value.

Types to Support Pattern Matching

A list looks like an array, but there are two important fundamental differences

Lists are immutable
Lists are recursive

It’s easy to understand ‘immutable’ mentioned above, but what does ‘recursive’ mean? A non-empty list consists of an element and a sub list. The cons operator : separates the head(element) and the tail(sub list) of a list. For example

@main
def main() =
  val l1 = List("US", "CN", "RU")
  val l2 = "US" :: "CN" :: "RU" :: Nil
  println(l1 == l2) // true
  println(l1 eq l2) // false
  println(Nil == List()) //true

Explanation

l1 and l2 are two equivalent lists, but they are indeed two different objects.
== tests the value equality and eq tests the reference equality.
Nil is simply an empty list. When using :: to construct lists, do not forget to put Nil at the end of the list to indicate there is no more sub list.

Below are the three commonly used methods of lists

head: to get the first element of the list
tail: to get the sub list
isEmpty: to return true if the list is empty, and otherwise false

List can be effectively used in pattern matching with understanding the underlying structure. Below are some examples

1 :: 2 :: xs: the list starting with element 1 and 2
x: Nil: the list with one element
List(x): same as above x :: Nil
List(): the empty list, same as Nil
List(2 :: xs): the list containing another list as its element, and the inner list starts with element 2

Utilize pattern matching to implement the insertion sort of lists

def insertionSort(l: List[Int]): List[Int] = l match
  case Nil => Nil
  case h :: Nil => l
  case h :: t =>
    val sortedTail = insertionSort(t)
    if h > sortedTail.head then sortedTail.head :: insertionSort(h :: sortedTail.tail)
    else h :: sortedTail
    
@main
def main() =
  val l = List(5, 8, 2, 9, 7, 3)
  val sortedL = insertionSort(l)
  println(sortedL)

Exaplanation

insertionSort is a recursive function so the return type should be declared explicitly.
Lists are immutable so the sorted list is a new list.

Python Quick Guide

Sat, 18 Oct 2025 21:35:08 +0800

Get Help

>>> help(str)
Help on class str in module builtins:

>>> help(str.find)
Help on method_descriptor:

find(...)
    S.find(sub[, start[, end]]) -> int

    Return the lowest index in S where substring sub is found,
    such that sub is contained within S[start:end].  Optional
    arguments start and end are interpreted as in slice notation.

    Return -1 on failure.

Collection

Lists, Tuples, Sets and Dictionaries

Summary

Lists: containers for holding multiple elements in order
Tuples: similar to lists, but immutable
Sets: containers to hold multiple elements when membership rather than order or position is important
Dictionaries: key-value pairs

List highlights

# a list can hold elements of different types
>>> x = [1, 2, 3, "abc", [4, 5]]

# slicing is a widely used operation
# [start index: end index: step]
>>> x[3:1:-1]
['abc', 3]
# perform in-place modification with slicing
>>> x = [1, 2, 3, "abc", [4, 5]]
>>> x[3:] = [4]
>>> x
[1, 2, 3, 4]
# in-place 'filtering'
>>> x[:] = [e for e in x if e % 2 == 0]
>>> x
[2, 4]


# in-place sort vs. returning a sorted list
# in-place sort
>>> countries = ["China", "USA", "Australia"]
>>> countries.sort(key=lambda x: len(x))
>>> countries
['USA', 'China', 'Australia']
# sorted built-in function returns a sorted list
>>> countries = ["China", "USA", "Australia"]
>>> sorted(countries, key=lambda x: len(x))
['USA', 'China', 'Australia']

# shallow copy vs. deep copy
>>> l1 = [["x"], "y"]
# shallow copy via slicing
>>> l1_sc = l1[:]
# deep copy
>>> import copy
>>> l1_dc = copy.deepcopy(l1)

Tuple highlights

# `,` is needed for single element tuple
>>> type((1,))
<class 'tuple'>
>>> type((1))
<class 'int'>

# tuple may be immutable, but NOT hashable
>>> x = (1,2,[3])
>>> type(x)
<class 'tuple'>
# tuple itself is immutable, but its content may be mutable
>>> x[2].extend([4,5])
>>> x
(1, 2, [3, 4, 5])

# swap variable values with tuple and packing/unpacking
>>> x = 5
>>> y = 23
>>> x,y = y,x # y,x is packed into a tuple and then unpacked for assignment
>>> x
23
>>> y
5

Set highlights

# items in a set must be immutable and hashable
>>> set((1,2,3))
{1, 2, 3}
>>> set((1,2,[3]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

# duplicate items are removed when adding to set
>>> s = {1,2,3,4,5,2,3}
>>> s
{1, 2, 3, 4, 5}
>>> s.add(5)
>>> s
{1, 2, 3, 4, 5}

# a set itself is mutable and unhashable
# to put a set inside another set, use frozenset
>>> {1,2,3,{4,5}}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'
>>> {1,2,3,frozenset({4,5})}
{frozenset({4, 5}), 1, 2, 3}

Dictionary highlights

# widely used `items` function
>>> for k,v in {"China": 5, "USA": 3}.items():
...     print(f"{k} --> {v}")
...
China --> 5
USA --> 3

# to delete an entry, use `del`
>>> d = {"China":5, "USA":3}
>>> del d["USA"]
>>> d
{'China': 5}

# provide default value when the key does NOT exist in the dict
# `dict.get(key, dflt_val)`
# `dict.setdefault(key, dflt_val)`
>>> d = {"China":5, "USA":3}
>>> d.get("Japan", 5)
5
>>> d.setdefault("Korea", 5)
5
>>> d["Korea"]
5

Dictionaries can be used as caches to avoid recalculation

cal_cache = {}
def calc(param):
    if param not in cal_cache:
        # calculate and then store the result into cache
        result = calculate(param)
        cal_cache[param] = result
    return cal_cache[param]

Comprehension

Don’t use a loop if a comprehension can do it more cleanly.

# list comprehension
>>> [e*e for e in [1,2,3]]
[1, 4, 9]

# set comprehension
>>> {e*e for e in {1,2,3}}
{1, 4, 9}

# dict comprehension
>>> {k.upper() : v*2 for k, v in {"a":1, "b":2}.items()}
{'A': 2, 'B': 4}

Strings

Strings can be treated as sequences of characters, so operations such as slicing can be performed on them.

>>> "Hello"[-1::-1]
'olleH'

Numeric and Unicode escape sequences can be used to represent characters in strings.

>>> "\x6D"
'm'
>>> "\u2713"
'✓'
>>> '\u4F60\u597D'
'你好'

Strings are immutable, so methods return new strings, although they may look like they update the string contents in place.

>>> "hello, world".title()
'Hello, World'

>>> "C++++".replace("++","+")
'C++'

The string module defines some useful constants.

>>> import string
>>> string.digits
'0123456789'
>>> string.hexdigits
'0123456789abcdefABCDEF'
>>> string.whitespace
' \t\n\r\x0b\x0c'
>>> string.ascii_lowercase
'abcdefghijklmnopqrstuvwxyz'

Formal string representation vs. Informal string representation

repr: formal string representation of a Python object. The returned string representation can be used to rebuilt the original object, just like serialization/deserialization. It’s great for debugging programs.
str: informal string representation of a Python object. It’s intended to be read by humans. str applied to any built-in Python object always call repr

>>> repr([1,2,3])
'[1, 2, 3]'
>>> str([1,2,3])
'[1, 2, 3]'

String interpolation is available since version 3.6. It’s called f-string.

>>> value = 523
>>> f"The value: {value}"
'The value: 523'

# function can be called
>>> lang = "go"
>>> f"The next one: {lang.upper()}"
'The next one: GO'

Bytes

String vs. Bytes

A string object is an immutable sequence of Unicode characters.
A bytes object is a sequence of integers with values from 0 to 256, mainly for dealing with binary data.

Two confusing items

unicode: a set of characters
utf-8: an encoding standard, which is used to present unicode. With different encodings, unicode will be presented with different values.

>>> c = "\u2713"
>>> c
'✓'

# try different encoding
>>> c.encode(encoding='utf-16')
b"\xff\xfe\x13'"
>>> c.encode(encoding='utf-8')
b'\xe2\x9c\x93'

# encoded value back to string, and by default utf-8 is the encoding/decoding standard.
>>> b'\xe2\x9c\x93'.decode()
'✓'

Control Flow

The ’ladder’ structure is as shown below

if condition1:
   body1
elif condition2:
   body2
elif condition3:
   body3
...
elif condition(n-1):
   body(n-1)
else:
   body(n)

pass can be used if an empty body of if or else is neede.

if cond:
    pass
else:
    # do something else

A dictionary can be used to ease the ’ladder’ structure.

def take_action_a():
    # do something

def take_action_b():
    # do something else

def take_action_c():
    # do another thing

func_dict = {'a': take_action_a,
             'b': take_action_b,
             'c': take_action_c}

# populate the desired function key, and here simple assign 'a' for demo purpose
desired_func_key = 'a' 
func_dict[desired_func_key]()

`for` Loop

for loop is different from the one in ‘C family’ programming languages. In Python, for iterates over the values returned by any iterable object, so it’s more like an iterator, instead of a loop structure.

>>> for elt in [1,2,3,4,5]:
...     if elt % 2 == 0:
...         print(elt)
...
2
4

Unpacking is supported by for.

>>> for idx, val in enumerate(["A", "B", "C"]):
...     print(f"{idx}: {val}")
...
0: A
1: B
2: C

`range`, Generator and Memory Usage

When dealing with a list holding a large number of elements, we may encounter the memory usage issue. Compare the memory consumption of a list and a range.

>>> import sys

>>> sys.getsizeof(list(range(1000000)))
8000056

>>> sys.getsizeof(range(1000000))
48

So using range or generator can reduce the strain on memory.

>>> x = list(range(1_000_000))
# using generator expression, we don't have to 'duplicate' the size of `x`
>>> g = (elt * elt for elt in x)
>>> import sys
>>> sys.getsizeof(g)
104

Boolean Values for Conditions

In Python

0 or empty values are False.
Any other values are True.

Some practical terms

Values like 0.0 and 0+0j are False.
Empty String "" is False.
Empty list [] is False.
Empty dictionary {} is False.
The special value None is always False.

Some objects, such as file objects and code objects don’t have a sensible definition of 0 or empty element, so they should NOT be used in a Boolean context.

Some boolean related operators

in and not in to test the membership
is and is not to test the identity
and, or, and not to combine boolean values

Operators

`==`/`!=` vs. `is`/`is not`

Equality vs. Identity

==/!=: to test the equality
is/is not: to test the identity

>>> l1 = [1,2,3]
>>> l2 = [1,2,3]
>>> l1 == l2
True
>>> l1 is l2
False

`and` and `or` Used in Non-Boolean Context

and and or can be used in non-boolean context to ‘pick’ the object.

and: pick the first false object or the last object
or: pick the first true object or the last object

>>> "a" and "" and "c"
''
>>> "a" and "b" and "c"
'c'

>>> "a" or "" or "c"
'a'
>>> "" or "" or "c"
'c'

Alternative to Ternary Operator `? :`

Some programming languages provide the ternary operator ? : such as the JavaScript code snippet below

name = 1 ? "Yang" : "Yin"
console.log(name)

However, there is NO such ternary operator in Python. Python chooses a more readable style

>>> name = "Yang" if 1 else "Yin"
>>> print(name)
Yang

Functions

The basic function definition is shown below

>>> def double(x):
...     return x * 2
...

>>> double(5)
10

# function without parameters
>>> def subroutine():
...     print("This is subroutine")
...
>>> subroutine()
This is subroutine

# function without explicit return
# in this case `None` is returned
>>> def no_explicit_return():
...     print("No explicit_return")
...
>>> r = no_explicit_return()
No explicit_return
>>> r is None
True

Parameters

Three available options for function parameters

Positional parameters
Named parameters
Variable numbers of parameters

Named parameters help remove the ambiguity in some cases

>>> def power(base, exponential):
...     if exponential == 0:
...         return 1
...     else:
...         return base * power(base, exponential-1)
...

# using named parameter we know it's cube of 2, not square of 3
>>> power(base = 2, exponential = 3)
8

In addition, Named parameters help in the default value case

>>> def greet(message="Hello", name="world"):
...     print(f"{message}, {name}")
...

# use the default value of `message` parameter
>>> greet(name="NYC")
Hello, NYC

Variable numbers of parameters allow the function to handle arbitrary numbers of parameters. There is no method overloading in Python like the one in Java, and variable numbers of parameters can be used to mimic the feature. In addition, decorator pattern can be implemented with variable numbers of parameters.

def decrate(fn):
    def decorated_fn(*parameters, **key_val_pairs):
        print("Doing decoration tasks...")
        fn(*parameters, **key_val_pairs)
        print("End\n")
    return decorated_fn

def greet(message="Hello", name="world"):
    print(f"{message}, {name}")

decorated_greet = decrate(greet)
decorated_greet("Hi")
decorated_greet(name="NYC")

Functions as the First-class Citizens

Functions can be assigned to variables, just as other values in Python.

>>> def foo():
...     print("foo funciton!")
...
>>> fn = foo
>>> fn()
foo funciton!

Anonymous functions are implemented as lambda expressions.

>>> fn = lambda: print("bar function!")
>>> fn()
bar function!
>>> fn
<function <lambda> at 0x000002E3BDD3EB90>

High order functions are supported ’natively’, since functions are first-class citizens.

# a function can accept functions and return function
def combine(outer_fn, inner_fn):
    def combined_fn(*parameters, **key_values):
        return outer_fn(inner_fn(*parameters, **key_values))
    return combined_fn

def square(x):
    return x * x

dbl = lambda x: x * 2

double_of_squred = combine(dbl, square)
r = double_of_squred(5)
print(r)

Scope: `global` and `nonlocal`

Local variables vs. global variables vs. nonlocal variables

local variables: variables defined in the function
global variables: variables defined outside the function
nonlocal variables: variables defined in the ’enclosing’ scope

Compare local variables and global variables.

a = 10

def foo():
    # global a
    a = 20
    print(f"a in foo: {a}")

foo()
print(f"global a: {a}")

# we get below result
#   a in foo: 20
#   global a: 10

# when `global a` is uncommented, we get below result
#   a in foo: 20
#   global a: 20

nonlocal refers to the one defined in the enclsoing function.

a = 10

def foo():
    a = 20
    def bar():
        nonlocal a
        a = 30
        print(f"a in bar: {a}")
    bar()
    print(f"a in foo: {a}")

foo()
print(f"global a: {a}")

# we get below result
#   a in bar: 30
#   a in foo: 30
#   global a: 10

Generator Functions

Besides generator expressions, there are generator functions to help on better memory usage.

# generator function
def gen_1_m():
    i = 1
    while i < 1_000_000:
        yield i
        i = i + 1

s = 0
for elt in gen_1_m():
    s = s + elt

print(s)

yield from can be used to delegate the generator to another generator.

g1 = range(1,500_000)
g2 = range(500_000,1_000_000)

def gen_1_m():
    yield from g1
    yield from g2

s = 0
for elt in gen_1_m():
    s = s + elt
print(s)

Modules and Scoping Rules

What is a module?

a file containing Python code, which defines Python functions or objects
the name of the file defines the name of the module

Why use modules?

to better organize source code
modules help avoid name-clash issues. Suppose two people both define greet function.
- module_a.greet
- module_b.greet

To use a module, import it first.

# import the built-in `math` module
>>> import math

# check the members of the module
>>> dir(math)
['__doc__', '__loader__', '__name__', '__package__', '__spec__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'comb', 'copysign', 'cos', 'cosh', 'degrees', 'dist', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 'isclose', 'isfinite', 'isinf', 'isnan', 'isqrt', 'lcm', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'nan', 'nextafter', 'perm', 'pi', 'pow', 'prod', 'radians', 'remainder', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'tau', 'trunc', 'ulp']

# reference to `pi`defined in `math`
>>> math.pi
3.141592653589793

Another import form is from <module> import <member/*>

>>> from math import pi
>>> pi
3.141592653589793

# we can even import all members using `*`
>>> from math import *
>>> gcd
<built-in function gcd>

The Module Search Path

To make module files available to Python to import, put it under any path entries defined in sys.path

>>> import sys
>>> sys.path
['', 'C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python310\\python310.zip', 'C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python310\\DLLs', 'C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python310\\lib', 'C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python310', 'C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages']

Note

The first module file found in the entries is used.
If no desired module can be found, an ImportError exception is raised.

How to define the path entries in sys.path list?

sys.path list is initialized based on PYTHONPATH environment variable if it exists.
Define .pth file to indicate the path entries, and put the .pth file under the directory defined by sys.prefix

Scoping Rules and Namespaces

A namespace maintains the mapping from identifiers to objects. A statement like x = 1 adds x to a namespace and associates x with the value 1.

In Python there are three namespaces

local: holding local functions and variables
global: holding module functions and module variables
built-in: holding built-in functions

When Python needs to ’locate’ the identifier, it follows below sequence

Check local namespace.
If the identifier doesn’t exist in local namespace, check global namespace.
If the identifier doesn’t exist in global namespace, check built-in namespace.
If the identifier doesn’t exist in any of above, NameError occurs.

When a function call is made, a local namespace is created.

def foo():
    x = 1
    print(f"In foo locals: {locals()}")
    print(f"In foo globals: {globals()}")

y = 2
foo()
# on global level, locals() is equivalent to globals()
print(locals() == globals())
print(dir(__builtins__))

# executing the code snippet above, we get
In foo locals: {'x': 1}
In foo globals: {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x0000013F1797C700>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': 'C:\\sandbox\\PythonLab\\Scripts\\lab.py', '__cached__': None, 'foo': <function foo at 0x0000013F178B3E20>, 'y': 2}
True
['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning', 'ChildProcessError', 'ConnectionAbortedError', 'ConnectionError', 'ConnectionRefusedError', 'ConnectionResetError', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EncodingWarning', 'EnvironmentError', 'Exception', 'False', 'FileExistsError', 'FileNotFoundError', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning', 'IndentationError', 'IndexError', 'InterruptedError', 'IsADirectoryError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'ModuleNotFoundError', 'NameError', 'None', 'NotADirectoryError', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'PermissionError', 'ProcessLookupError', 'RecursionError', 'ReferenceError', 'ResourceWarning', 'RuntimeError', 'RuntimeWarning', 'StopAsyncIteration', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'TimeoutError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'WindowsError', 'ZeroDivisionError', '__build_class__', '__debug__', '__doc__', '__import__', '__loader__', '__name__', '__package__', '__spec__', 'abs', 'aiter', 'all', 'anext', 'any', 'ascii', 'bin', 'bool', 'breakpoint', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']

Interaction between Python Program and System

Combine Script and Module

A Python program can be treated as a script or a module depending on the execution context. The structure below does the trick.

if __name__ == '__main__':
    main()
else:
    # module-specific initialization code if needed

When the Python file is executed as Python script, its __name__ is set to __main__.

Commandline Arguments

The arguments passed from commandline can be retrieved via sys.argv.

import sys

def main():
    print(sys.argv)

main()

sys.argv is a list

The first element is the name of the script file.
The following elements are the arguments passed from commandline.

PS C:\sandbox\PythonLab\TempLab> python .\my_sciprt.py Hello World "Test Script"
['.\\my_sciprt.py', 'Hello', 'World', 'Test Script']

# omit `.\` to invoke the script file
PS C:\sandbox\PythonLab\TempLab> python my_sciprt.py Hello World "Test Script"
['my_sciprt.py', 'Hello', 'World', 'Test Script']

Use argparse module if more advanced features are needed to handle commandline arguments.

Filesystem Interaction

File Paths

Path related modules

os.path: before Python 3.5, and it’s imperative style.
pathlib: since Python 3.5, and it’s OO style.

os.path provides a useful abstraction layer to ease operations on filesystems. For example, file path separator may be differnt from OS to OS.

\ in Windows OS
/ in *nix OS

Using os.path.sep, we don’t have to worry about the difference. As a result, program with that abstraction layer

use os.path.curdir
NOT use .

Unfortunately, there is no unified concept of root path. Think about the path types we have in Windows OS

C:\ means the C drive
\\myftp\share\ means a UNC root path As a result, we do NOT have something like os.path.root in Python.

To form a path

# form a path with os.path
os.path.join("c:/", "Sandbox", "Temp")

# form a path with pathlib
# note `joinpath` of Path object is an instance method
pathlib.Path().joinpath("c:/", "Sandbox", "Temp")
pathlib.Path() / "c:/"/ "Sandbox"/ "Temp"
pathlib.Path("c:/") / "Sandbox"/ "Temp"

Filesystem Operations

Filesystem operations are performed via os module. Don’t get confused with sys module. Think ‘sys’ as ‘Python System’.

# change directory
os.chdir("My Target Dir")

# print current working directory
os.getcwd()

# list items in the directory
# Note in windows, we may encunter PermissionError if the dir is read-only
os.listdir(os.path.curdir)

# get file/dir info
os.path.exist(path_as_arg))
os.path.isfile(path_as_arg))
os.path.isdir(path_as_arg))
os.path.getsize(path_as_arg))
os.path.getatime(path_as_arg))

# renme file/dir
os.rename("original", "target")

# remove a file
# `remove` function cannot remove a directory
os.remove("file_to_be_removed")
# `rmdir` can remove an empty directory
os.rmdir("empty_dir_to_be_removed")

# create a directory
os.mkdir("dir_name")
os.makedirs("auto_create_intermediate_dirs")

If OO style is preferred, use pathlib module. With pathlib we create different objects to represent different paths, so we don’t do operation like pathlib_obj.chdir("Target Dir")

# create the obj representing the current dir
curr_dir = pathlib.Path()

# create the obj representing the specified path
root_dir = pathlib.Path("/")

# list the items in the directory
for fs_item in curr_dir.iterdir():
    print(fs_item) # fs_item is Path object as well

# print current working directory
# below two expressions return the same value
# note current working directory is determined by where we started Python program and if we switched to another dir later
curr_dir.cwd()
root_dir.cwd()

# get file/dir info
path_obj.exists()
path_obj.is_file()
path_obj.is_dir()
path_obj.stat()

# rename a file or directory
path_obj.rename("new_name")

# remove a file
path_obj.unlink()
# remove an empty directory
path_obj.rmdir()

# create a directory
path_obj.mkdir() # requires intermediate directories exist
path_obj.mkdir(parents=True) # intermediate directories will be created automatically

Utilities for Filesystem Operation

os.scandir provides an easy approach to get metadata of filesystem entries under a directory.

# use a context manager to ensure the file descriptor is released
# regardless of whether the iterator is fully iterated
with os.scandir(os.curdir) as my_dir:
    for fs_entry in my_dir:
        print(f"{fs_entry.name}: {fs_entry.stat()}")

glob.glob provides the globbing functionality.

import glob
os.chdir("c:/sandbox/pythonlab/scripts")
py_files = glob.glob("*.py")

for py_file in py_files:
    print(f"Python File: {py_file}")

shutil.rmtree can remove a non-empty directory, and shutil.copyree can recursively make copies of all the files and subdirectories in a given directory.

import shutil

shutil.rmtree(nonempty_dir_to_be_removed)

shutil.copytree(src, dst)

os.walk(directory, topdown=True, onerror=None, followlinks=False) traverses directory structure recursively. The function returns three things

root or path of the directory
a list of its subdirectories (os.walk will be called on each subdir respectively)
a list of its files

for root, subdirs, files in os.walk("Test"):
    for file in files:
        print(f"file name: {file}")
    # remove backup directory from the recursion
    subdirs[:] = [e for e in subdirs if e != "backup"]
    print(f"Subdir list now is {subdirs}")

Note

If topdown is True or not present, the files in each directory are processed before moving to subdirectories. That means we have a chance to remove some subdirectories, such as .git/, .config/ from the recursion.

File I/O

Open and Close Files

The classic open-process-close file operation is as shown below

file_obj = open("c:/temp/hello.txt")
print(file_obj.readline())
file_obj.close()
print(f"File closed? {file_obj.closed}")

Using context managers, we don’t need to explictly close the file.

with open("c:/temp/hello.txt") as file_obj:
    print(file_obj.readline())

Specify the mode to open file with

r: read mode, the default mode
w: write mode, data in file will be truncated before writing operation
a: append mode, new data will be appended to the end of the file
x: new file mode, it throws FileExistsError if the file exists already
+: read and write mode
t: text mode, the default mode
b: binary mode, it supports random access

With the modes above, we have

rt: read as text
w+b: random accessing the file in binary mode with truncating the file first
r+b: random accessing the file in binary mode without truncating the file first

In addition, pay attention to the options below when open the file

encoding: sepcify the encoding to open the file with
newline: different operating systems may use different characters as the new line character

Suppose we have a txt file containing below unicode chars with utf-8 encoding

✓💓🍁

We can specify the encoding as utf-8 when open the file

with open("c:/temp/unicode.txt", encoding="utf-8") as file_obj:
    print(file_obj.read(1))
    print(file_obj.read(1))
    print(file_obj.read(1))

Read and Write with `TextIOWrapper`

In most cases, read, readline and readlines without argument are good enough to handle file reading. However, there will be some exceptional cases like

the file is too large
the line contains too many contents
there are too many lines

Two approaches to tackle the issue

provide additional arguments to affect the amount of data being read every time
use iterator to lazily load file contents

# argument to affect the amount of data being read every time
size_to_read = 50
with open("c:/temp/the_zen_of_python.txt", mode="rt") as file_obj:
    while sized_content := file_obj.read(size_to_read):
        print(sized_content, end='')

# treat file object as generator
# `open` returns a file object which is an iterator
# `isinstance(fo, collections.abc.Iterator)` returns True
with open("c:/temp/the_zen_of_python.txt", mode="rt") as file_obj:
    for line in file_obj:
        print(line, end="")

Note

size parameter of readline indicates the max size of chars to read before encoutering the newline character, so we may read less than the size on some lines.
hint parameter of readlines indicates the size of chars to be exceeded by reading lines, so we may read an ’extra’ line, just for exceeding the hint size.

We perform ‘write’ operation mainly with functions

write
writelines

The code snippet below implements a dummy version of ‘copy’

# dummy copy
import os

size_of_chunk = 128
source_file = os.path.join("C:/", "temp", "the_zen_of_python.txt")
target_file = os.path.join("C:/", "temp", "zen.txt")
# binary mode so both binary files and text files can be handled
with open(source_file, "rb") as sf_obj:
    with open(target_file, "wb") as tf_obj:
        while content_chunk := sf_obj.read(size_of_chunk):
            print(">", end="")
            tf_obj.write(content_chunk)

print("Done")

Read and Write with `pathlib`

pathlib provides OO style read/write operations. It encapsulates actions like ‘open’ and ‘close’, so we don’t need to do them by ourselves. Below are the related functions

pathlib.Path.write_bytes
pathlib.Path.write_text
pathlib.Path.read_bytes
pathlib.Path.read_text

# dummy copy via pathlib's OO style
import pathlib

source_file = pathlib.Path() / "C:/" / "temp" / "the_zen_of_python.txt"
target_file = pathlib.Path() / "C:/" / "temp" / "zen.txt"

target_file.write_bytes(source_file.read_bytes())
print("Done using pathlib")

read_bytes and read_text don’t provide a paramter to specify the chunk size to read each time, and those functions read the entire file into memory. If memory-efficient is important, use the open function of the Path object to get the ‘file object’ and then work as the classic open style

import pathlib

chunk_size = 128
source_file = pathlib.Path() / "C:/" / "temp" / "the_zen_of_python.txt"
target_file = pathlib.Path() / "C:/" / "temp" / "zen.txt"

with source_file.open(mode="rb") as sf_obj:
    with target_file.open(mode="wb") as tf_obj:
        while chunk := sf_obj.read(chunk_size):
            tf_obj.write(chunk)
            
print("Done!")

File as Standard Out

A file can be set as stdout, so that print function will write the content to the file instead of to the terminal.

import sys

with open("c:/temp/output.txt", mode="wt") as of_obj:
    sys.stdout = of_obj
    print("Hello")
    print("World")
    # reset stdout back
    sys.stdout = sys.__stdout__
    print("Hi")

Alternative to setting sys.stdout to a file, in each print we can set the file parameter to the specified file.

Exceptions

Intro

The exception mechanism in Python is built around OO paradigm. An exception in Python is an object. The act of generating an exception is called raising or throwing an exception. Exceptions can be raised by

explcitly using raise statement in our own code
any other functions

The raise statement does below things

hold the normal execution path of Python program
raise an exception
search for an exception handler that can deal with the exception
- if such a handler is found, execute it
- if no such handler, the program aborts with an error message

In Python, error handlers are put together after the ‘happy path’ code snippet. The classic structure is

try:
    # do something
except <ExceptionType>:
    # handler code
finally:
    # do something regardless of whether exception happens

Multiple handlers can be ‘chained’ to handler different exceptions. The rule of thumb is to put general excpetion handler in the end.

l = [1,2,3]

try:
    print(l[100])
except Exception:
    print("Captured general excpetion!")
except IndexError:
    print("Captured Index Error")

In the above code snippet, by design Exception handler is put above IndexError handler, which leads to the fact that IndexError is completely ‘shadowed’.

Most exceptions inherit from Exception class, and they have ’ Error’ as their suffix instead of Exception, such as IndexError, FileNotFoundError, PermissionError, ValueError, etc. There are exception types, such as KeyboardInterrupt, inheriting from BaseException. That way, when users pressing Ctrl-c, the KeyboardInterrupt will NOT be captured by our handlers, and it can be passed to OS for terminating the program.

Raise Exceptions

The straight way to raise exception is to use raise statement like raise <ExceptionType>(<msg>). Below is an example raising NameError

raise NameError("For testing NameError")

Another way to raise exceptions is to use Python’s built-in functions/features. For example

l = [1,2,3]
print(l[100])
print("End")

Execute above Python script, and we will get below error message

C:\sandbox\PythonLab\scripts> python lab.py
Traceback (most recent call last):
  File "C:\sandbox\PythonLab\scripts\lab.py", line 2, in <module>
    print(l[100])
IndexError: list index out of range

In the above case, l[100] raises IndexError, but there is no corresponding error handler, so the IndexError goes all the way up to Python interpreter, and there the program terminates with an error message being printed out.

If print("End") is a must to us, put it in finally as shown below

try:
    l = [1,2,3]
    print(l[100])
finally:
    print("End")

Execute the code above, and we get the output below

C:\sandbox\PythonLab\scripts> python lab.py
End
Traceback (most recent call last):
  File "C:\sandbox\PythonLab\scripts\lab.py", line 3, in <module>
    print(l[100])
IndexError: list index out of range

Explanation

There is no error handler, so the error will be propagated upward along the stack of functions/callers, in this case the Python interpreter.
Before the execution flow pauses and goes to the caller, the finally section gets a chance to execute, and that is the reason why END gets printed out even before the error message.

Catch and Handle Exception

Error handlers are

NOT for causing a program to halt with error messages
perhaps for displaying error messages to users as reminders
perhaps for fixing the problem in the first place

With the as keyword, we can access the exception to get detailed information. For example

try:
    raise ValueError("VE-001","For testing purpose!")
except ValueError as ve:
    print(f"Code: {ve.args[0]} | Msg: {ve.args[1]}")
except Exception as e:
    print(e)
finally:
    print("Fin")

Execute the code above, and we will get the output below

In [3]: %run lab.py
Code: VE-001 | Msg: For testing purpose!
Fin

Explanation

ValueError accepts *args and **kwargs. Usually, we may only pass ‘message’, not as shown above case passing both ‘code’ and ‘message’.
There is the handler to deal with the exception. After that, the finally section gets a chance to execute.
In the handler, we can even further raise the exception. One use case is to log the error message in the handler and then raise the exception to let upward handler deal with the exception.

Define Custom Exception

When defining our own exceptions, use Exception as the base class, instead of BaseException. For example

# Exception as base class
class CustomError(Exception):
    pass

try:
    raise CustomError("This is custom err!")
except CustomError as ce:
    print(ce)
    raise ce
except Exception as e:
    print(e)

Execute above snippet in iPython, and we get

In [10]: %run lab.py
This is custom err!
-----------------------------------------------------------------------
CustomError                           Traceback (most recent call last)
File C:\Sandbox\PythonLab\Scripts\lab.py:8
      6 except CustomError as ce:
      7     print(ce)
----> 8     raise ce
      9 except Exception as e:
     10     print(e)

File C:\Sandbox\PythonLab\Scripts\lab.py:5
      2     pass
      4 try:
----> 5     raise CustomError("This is custom err!")
      6 except CustomError as ce:
      7     print(ce)

CustomError: This is custom err!

Custom Exception in Practice

Typical useage of custom exception is

For small program, there will be a couple of unique exceptions, and it’s common to only create a general base exception inheriting from Exception.
For large program, we may define a general base exception, and then define each unique exception inheriting from that general base exception.
- For example, for a Robot application. We can define class RobotError(Exception) as the base exception. Then define exceptions like class TransmissionError(RobotError), BatteryError(RobotError).

Debug Program with `assert`

assert is a special form of raise.

assert <expression>, <argument>

In practice, it can be used to debug programs.

# business logic code to generate x's value
x = [1,2,3] # hard code to mimic the value assigned to x

assert len(x) > 5, f"x should contain at least 5 elements, but x is {x}"

assert in the above code snippet will cause AssertionError as shown below

C:\sandbox\PythonLab\scripts> python lab.py
Traceback (most recent call last):
  File "C:\sandbox\PythonLab\scripts\lab.py", line 3, in <module>
    assert len(x) > 5, f"x should contain at least 5 elements, but x is {x}"
AssertionError: x should contain at least 5 elements, but x is [1, 2, 3]

To ignore/turn off the assert debug feature, start Python interpreter with the -O option as shown below

python -O lab.py

That means we can safely use assert statements during development, and even leave them in the code for future use with no runtime cost.

The Exception Inheritance Hierarchy

The except clause matters in exception handling.

# code snippet with defect
try:
    body
except LookupError as error:
    exception code
except IndexError as error:
    exception code

Explanation

IndexError is a subtype of LookupError, which means the IndexError handler is subsumed by the LookupError handler, so IndexError never gets a chance to execute.
To fix the issue, simply move IndexError handler above LookupError handler.

Context Managers

There are situations that we follow a predictable pattern with a set ‘beginning’ and ’end’, for example when reading contents from a file

beginning: open the file
body: operations according to the business logic
end: close the file

Python3 offers context managers to ease above operation. Context managers wrap a block and manage requirements on entry and departure from the block. File objects are context managers, so we can do

with open(filename) as infile:
  data = infile.read()
  # further operation with data

# the code above is logically equivalent to
try:
  infile = open(filename)
  data = infile.read()
  # further operation with data
finally:
  infile.close()

Explanation

Using with, there is no need to manaully invoke file.close, becuase the file-closing operation is handled by the context manager.

Context managers are great for things like

locking and unlocking resources
committing data transactions

Classes and OOP

Defining Classes

All the data types built into Python are classes and we can define our own classes

class MyClass:
    pass

my_instance = MyClass()
print(my_instance)

Explanation

With pass we define an class with ’empty’ body.
By convention, class identifiers are in CamelCase.
To create an instance of the class type, call the class name as a function without new.

Using Objcts as Structs/Records

In Python, the data fields(attributes) of an object/instance doen’t need to be declared ahed of time on class level, and they can be ‘attached’ on the fly.

class MyClass:
    pass

my_instance = MyClass()
# attach `field1` on the fly
my_instance.field1 = "Hello Python"
print(my_instance.field1)

Explanation

field1 is attached to my_instance as an attribute/data filed/instance variable.
Use dot notation to reference to the instance variable.

Instance Variables and Initialization

To create instance variables like what constructor does, we can use __init__ method as shown below

class Robot:
    def __init__(self, name):
        self.name = name

def main():
    arale = Robot("Arale")
    print(arale.name)

Explanation

self in __init__ method represents the ‘current’ instance, so self.name refers to the instance variable.
All uses of instance variables in Python require explicit mention of the containing instance. Without instance, it means the variable in the local namespace.
- In the above example, self.name = name contains two name. The first name has the instance container self, so it refers to the instance variable, and the second name has no instance container so it is the method parameter name.

Instance Methods

Similar to instance variables, instance methods also ’link’ to instances. This is reflected by the method invocation forms

bound invocation
unbound invocation (less commonly used)

class Robot:
    def __init__(self, name):
        self.name = name
    def greet(self, name):
        print(f"Hello {name}, I'm {self.name}")

def main():
    arale = Robot("Arale")
    # bound method invocation
    arale.greet("Dr. Slump")
    # unbound method invocation
    Robot.greet(arale, "Dr. Slump")

Explanation

An instance method has the reference to the instance as the first parameter, and conventionally it’s named self.
To invoke the instance method, we can use either bound form or unbound form.
- In unbound form, the instance is explicitly passed to the method as the first parameter.

Think of unbound form as one function defined in the ’namespace’ of a class, and to refer to that function, we need to use the ‘qualified name’.

class Robot:
    def __init__(self, name):
        self.name = name
    def greet(self, name):
        print(f"Hello {name}, I'm {self.name}")

def main():
    arale = Robot("Arale")
    fn = Robot.greet
    fn(arale, "Dr. Slump")

Similar to ‘attaching’ instance variables on the fly, instance methods can be attached or overridden on the fly.

class Robot:
    def __init__(self, name):
        self.name = name
    def greet(self, name):
        print(f"Hello {name}, I'm {self.name}")

def main():
    # define a function to override <inst>.greet
    def greet(name):
        print(f"Hi, {name}")
    arale = Robot("Arale")
    arale.greet = greet
    arale.greet("Dr. Slump")
    # remove the dynamically overridden `greet` from the instance to revert back to the one defined in the class
    del(arale.greet)
    # define a function and attach to an instance
    def desc():
        print("This is Robot desc")
    arale.desc = desc
    arale.desc()

Explanation

We dynamically define two functions, greet and desc to override/attach to the instance.
- Note those functions do NOT have self as the first parameter.
del can be used to delete the attached functions.

Class Variables

A class variable is a variable associated with a class, not an instance, and it’s accessible by all instances of the class.

A class variable is created by an assignment in the class body, not via self in the __init__ method.

class Robot:
    creator = "Dr. Slump"

def main():
    r1 = Robot()
    r2 = Robot()
    print(f"R One creator: {r1.creator}")
    print(f"R Two creator: {r2.creator}")
    print(f"Robot  creator: {Robot.creator}")
    # change `creator` via instance `r1` and that is reflected by `r2` and `Robot`, b/c `creator` is a class variable
    r1.__class__.creator = "???"
    print(f"R Two creator: {r2.creator}")
    print(f"Robot  creator: {Robot.creator}")
    # change `creator` via class directly
    Robot.creator = "Stark"
    print(f"R One creator: {r1.creator}")
    print(f"R Two creator: {r2.creator}")

Explanation

creator is defined as a class variable, and it can be accessed by all instances of the class.
To update the value of the class variable, we can
- either use <instance>.__class__.<class_variable>
- or use <class>.<class_variable>

Class Variables as ‘Fallback’

When accessing an instance variable, if the instance variable cannot be found, Python will try to find the class variable of the same name. If that cannot be found either, Python will signal an error.

This is actually what happened to the above Robot code snippet. r1.creator means to access the instance variable creator of instance r1. Howerver, there is no such instance variable, but luckily there is a class variable with the same name and it is accessed instead.

Class Variable Trap

Suppose we’d like to update the value of a class variable.

class Robot:
    creator = "Dr. Slump"

def main():
    r1 = Robot()
    # `r1.creator` below attaches an instance variable, instead of updating the class variable
    r1.creator = "???"
    print(Robot.creator)
    r1.__class__.creator = "???"
    print(Robot.creator)

Note

r1.creator = "???" attaches an instance variable, instead of updating the class variable.
r1.__class__.creator refers to the class variable.

Class Methods and Static Methods

There are two types of class level methods in Python

class methods with @classmethod decorator
static methods with @staticmethod decorator

class Robot:
    creator = "Dr. Slump"

    @classmethod
    def desc_in_detail(cls):
        print(f"This is Robot created by {cls.creator}.")

    @staticmethod
    def desc():
        print("This is Robot.")
        # print(f"Access class variable with class name hardcoded: {Robot.creator}")


def main():
    Robot.desc()
    Robot.desc_in_detail()

Explanation

Class methods and static methods are very similar.
- The difference is the first parameter of a class method is the reference to the class, conventionally named cls. As a result, it’s easy to access the class variables via that reference in class method.
- In the static method case, to access the class variable, hardcoding the class name is needed, such as the commented line, which is usually considered as bac code smell.
When invoking class methods and static methods, use the form <class>.<methods>

It’s possible to ‘attach’ class methods and static methods on the fly.

class Robot:
    creator = "Dr. Slump"

def main():
    r1 = Robot()
    Robot.say_hello = staticmethod(lambda: print("Hello from Robot."))
    Robot.say_hi = classmethod(lambda cls: print(f"Hi, I'm created by {cls.creator}."))
    r1.__class__.say_hello()
    r1.__class__.say_hi()

    # attach static method via `<instance>.__class__`
    r1.__class__.say_goodbye = staticmethod(lambda: print("Goodbye!"))
    Robot.say_goodbye()
    # attach class method via `<instance>.__class__`
    r1.__class__.say_bye = classmethod(lambda cls: print(f"Robot[by {cls.creator}] says bye"))
    Robot.say_bye()

Explanation

To add static methods and class methods, use the built-in staticmethod function and classmethod function correspondingly.

We may ask why staticmethod and classmethod functions are needed. Without them, we are actually attaching instance methods.

class Robot:
    creator = "Dr. Slump"
    def __init__(self, name):
        self.name = name

def main():
    r1 = Robot("Arale")
    # add instance method as if it is defined inside the class definition
    r1.__class__.greet = lambda self, name: print(f"Hello {name}, I'm {self.name}.")
    r1.greet("Jarvis")
    Robot.greet(r1, "Jarvis")

Explanation

greet is added as an instance method, instead of a class method or static method.

Inheritance

Like other object oriented programming languages, Python supports inheritance with the form SubClass(BaseClass).

class Robot:
    def __init__(self, name):
        self.name = name

class CleaningRobot(Robot):
    def __init__(self, name, type):
        super().__init__(name)
        self.type = type
        
def main():
    eva = CleaningRobot("Eva", "CR-TA")
    print(f"name: {eva.name} | type: {eva.type}")

Explanation

CleaningRobot is defined as a subtype of Robot. super() is used in the __init__ method of the subtype to invoke the __init__ defined in the superclass to fulfill initialization.
- This action is not performed by default, so manually invoking the __init__ defined in the super type is needed.
super is defined as a method, not a keyword like in Java. The reason is Python supports multiple inheritance, and by passing different arguments to super(), we can invoke the __init__ methods defined in different supertypes correspondingly to finish initialization.

Inheritance makes the instance variables and class variables defined in the superclass accessible in the subclass.

from datetime import datetime

class Robot:
    desc = "Robot"
    def __init__(self, name):
        self.name = name
        self.creation_time = datetime.now()

class CleaningRobot(Robot):
    def __init__(self, name, type):
        super().__init__(name)
        self.type = type
        
def main():
    eva = CleaningRobot("Eva", "CR-TA")
    # `creation_time` is defined in super class
    print(eva.creation_time)

    # `desc` is the class variable of super class
    print(eva.desc)

    # 'intercept' `desc` by 'attaching' `desc` on `CleaningRobot` class
    CleaningRobot.desc = "Cleaning Robot"
    print(eva.desc)

    # further 'intercept' `desc` by adding instance variable `desc`
    eva.desc = "Cleaning Robot Eva[CR-TA]"
    print(eva.desc)

Explanation

creation_time is defined in the __init__ method of super class Robot. Think about when that __init__ is executed, the self parameter represents the newly created instance.
desc is defined in the super class Robot. It is like a fallback if desc cannot be found in the instance of the sub class CleaningRobot.
When eva.desc is evaluated, the order of identifying desc is
1. look for ‘ad-hoc’ desc attribute in the instance, and fails to find one;
2. look for the instance variable desc defined in CleaningRobot definition and fails to find one;
3. look for the class variable desc, and finds one inherited from the super class Robot.
Once we provide ‘ad-hoc’ desc class variable to CleaningRobot or desc attribute to eva instance, Python does not need to reach out to the super class anymore, just as reflected in the last a few lines of main.

Similarly, inheritance makes the instance methods defined in super class available in sub class.

class Robot:
    def __init__(self, name):
        self.name = name
        
    def greet(self, name):
        print(f"Hello {name}, I'm {self.name}.")

class CleaningRobot(Robot):
    def __init__(self, name, model):
        super().__init__(name)
        self.model = model

def main():
    # access `greet` defined in super class
    eva = CleaningRobot("Eva", "CR-TA")
    eva.greet("WallE")

    # 'intercept' the `greet` instance method on class definition level
    eva.__class__.greet = lambda self: print("Greeting from CleaningRobot")
    eva.greet()

    # 'intercept' the `greet` instance method via ad-hoc function attached to instance
    eva.greet = lambda name: print(f"Hi {name}.")
    eva.greet("Jarvis")

    # remove the ad-hoc function to 'revert' back
    del(eva.greet)
    eva.greet()

Explanation

The greet instance method is defined in the super class Robot and it is available to the instance of sub class CleaningRobot
When invoking instance method eva.greet, the identification process follows below order
1. look for ad-hoc greet on the instance level
2. look for greet defined in the class definition
3. look for greet defined in the super class defition

Private Items

A private variable or private method is one that cannot been accessed outside the methods of the class in which it is defined. Private items make it easier to read code, because they intentionally indicate that they are used internally in a class only.

To define private items in Python, name them as __<desired_name>. The rule is the name begins, but does NOT end, with a double underscore is private.

class Robot:
    def __init__(self, id, name):
        self.__id = id
        self.name = name

class CleaningRobot(Robot):
    def __init__(self, id, name, type):
        super().__init__(id, name)
        self.type = type
    def desc(self):
        print(f"CleaningRobot[{self.name}|{self.type}]")
        # below statement causes error
        print(f"CleaningRobot[{self.__id}|{self.name}|{self.type}]")

def main():
    r = CleaningRobot("R001", "Eva", "CR-TA")
    r.desc()

if __name__ == "__main__":
    main()

Explanation

__id is defined as a private instance variable of Robot. It cannot be visited in the method of the sub type CleaningRobot, so the second print statement in desc method will cause error.

In addition, pay attention to the error message

'CleaningRobot' object has no attribute '_CleaningRobot__id'

What the heck is _CleaningRobot__id? It’s related to Python’s philosophy on handling private items.

Privacy is implemented by mangling the name of the private item, so it looks like hiding the private items from accidental access.
Technically, we can still access the priviate item via the mangled name.

In [112]: r = CleaningRobot("R001", "Eva", "CR-TA")

In [113]: dir(r)
Out[113]:
['_Robot__id',
 '__class__',
 '__delattr__',
 '__dict__',
 ...]
 
In [114]: r._Robot__id
Out[114]: 'R001'

Properties

In Python, even without getter and setter defined for instance variables in a class, we can still access those instance variables. However, Python indeed provides @property and @<property_name>.setter decorators, in case ‘plain access’ is not sufficient.

class Thermometer:
    def __init__(self):
        self._temp_fahr = 0

    @property
    def temperature(self):
        return (self._temp_fahr - 32) * 5 / 9

    @temperature.setter
    def temperature(self, temperature_in_celsius):
        self._temp_fahr = temperature_in_celsius * 9 / 5 + 32
    
def main():
    t = Thermometer()
    t.temperature = 37
    print(t.temperature)
    # _temp_fahr is intentionally used iternally, but no name mangling happens
    print(t._temp_fahr)

if __name__ == "__main__":
    main()

Explanation

@property provides the opportunity to update the ‘property’ to a suitable form before returning it to the caller.
The leading single underscore in _temp_fahr indicates the instance variable is intentially for internal use, just as private, but no name mangling happens.

One big advantage of using properies in Python is we can start from simple/plain-old instance variables and then seamlessly migrate to properties, because both use the same dot form, and from caller’s perspective, it may even not realized the change under the hood. That’s a good example of data abstraction.

Scoping and Namespace

In an instance method of a class, we have access to the following namespaces, just as regular function

local
global
built-in

In addition, via instance reference, conventionally self, we have access to instance and class namespaces.

mod_desc = "Robot Module"

class Robot:
    def __init__(self, name):
        self.name = name
        
    def greet(self, name):
        print(locals())
        print(globals())
        print(f"From {mod_desc}:Hello {name}, I'm {self.name}.")

def main():
    arale = Robot("Arale")
    arale.greet("Dr. Slump")

if __name__ == "__main__":
    main()

Explanation

locals() reflects the local variables. In greet method, there are two local variables, self and name.
globals() in greet reflects the global items, such as __name__, __builtins__, mod_desc, main, and etc.
self provides the access to instance and class related items, such as self.name

Note the order of identifying the items in the namespaces

local namespace
global namespace
built-in namespace

For instance/class related items

instance namespace
class namespace
superclass namespace

Multiple Inheritance

In Python, a class can inherit from any number of parent classes. In the multiple inheritance case, it’s fine if none of the super classes contains the methods of the same name, but inevitably, we will have the notorious diamond inheritance. Below is an example

    A
   / \
  B   C
   \ /
    D

With the above structure, we have the following code

class A:
    def m(self):
        print("m in A")

class B(A):
    def m(self):
        print("m in B")
        super().m()
        
class C(A):
    def m(self):
        print("m in C")
        super().m()

class D(B, C):
    def m(self):
        print("m in D")
        super().m()

def main():
    d = D()
    print(d.__class__.mro())
    d.m()

if __name__ == "__main__":
    main()

# execute the code above, and we get
In [1]: %run lab.py
[<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class 'object'>]
m in D
m in B
m in C
m in A

Explanation

Python uses Method Resolution Order(MRO) with the C3 lineraization algorithm to determine which method gets called.

When multiple inheritance gets a little complex, check the MRO is one effective way to understand which method will get invoked. Below is a complex inheritance example

Suppose we have above multiple inheritance hierarchy. A method named m is defined in C, F, and G. Once m is callled on an instance of A, which m method will be invoked?

class E:
    pass

class F:
    def m(self):
        print("m in F")
        # super().m()
        
class G:
    def m(self):
        print("m in G")
        
class B(E, F):
    pass

class C:
    def m(self):
        print("m in C")
        
class D(G):
    pass

class A(B, C, D):
    pass

def main():
    a = A()
    a.m()
    print(a.__class__.mro())

if __name__ == "__main__":
    main()

# execute the code above, and we get
In [13]: %run lab.py
m in F
[<class '__main__.A'>, <class '__main__.B'>, <class '__main__.E'>, <class '__main__.F'>, <class '__main__.C'>, <class '__main__.D'>, <class '__main__.G'>, <class 'object'>]

Explanation

According to MRO, the m defined in F is invoked.
Generally speaking, MRO follows a depth first, left to right order.
Try uncommenting the super().m() line and the m defined in C will be invoked additionally. Following the MRO, the one after F is C, so super() in m of F means C.

Regular Expression

The re module provides regular expression support in Python.

Suppose we have a text file with the following contents

123
hello world
lucky 523
F789E65
The End

Using re we can easily capture the lines with figures.

import re

def main():
    regex = re.compile("\d")
    in_file = open("sample.txt")
    for line in in_file:
        if regex.search(line):
            print(line, end="")
    in_file.close()

if __name__ == "__main__":
    main()

Explanation

regex is a compiled regular expression for better performance, using re.compile function.
re.search is used in the above case. If re.match was used, only 123 would be printed out, because re.match applies the regular expression to the beginning of the string.

Extract Matching Part

Suppose we need to do one step further, extracting the matching part. For example, to extract consecutive number characters from the file

123 --> 123
hello world
lucky 523 --> 523
F789E65 --> 789 and 65
The End

Use the code snippet below

import re

def main():
    regex = re.compile("\d+")
    in_file = open("sample.txt")
    for line in in_file:
        matching_nums = regex.findall(line)
        output_str = f"{line.strip()} --> {' and '.join(matching_nums)}" if matching_nums else line.strip()
        print(output_str)
        
    in_file.close()

if __name__ == "__main__":
    main()

Explanation

re.findall returns a list containing all the matching entries.
If more control is desired, re.finditer is the function to use.

Raw Strings for Regular Expression

In Python, regular expressions recognize special characters such as \n for a newline, \t for a tab, and \\ for a literal backslash. In addition, Python itself processes escape sequences in string literals. Together, these two layers of escaping can easily become a source of confusion.

import re

def main():
    s = "\\ten" # \ten
    print(s)
    # regex = re.compile("\\ten")
    regex = re.compile("\\\\ten")
    print(regex.search(s))

if __name__ == "__main__":
    main()

Explanation

In re.compile("\\ten"), the string literal "\\ten" is first processed by Python as a norma string. During this stage, escape processed, so the value ultimately passed to re.compile function is \ten.
\t is recognized as a tab when compiling the regular expression. That’s the reason why the commented out line does NOT work.

To make regular expressions easier to write and read, raw strings can be used. A raw string looks like a normal string with a leading r character. Raw strings can be used

single quotation marks
double quotation marks
triple quotation marks to span lines

Raw strings tell Python to NOT process escape sequence in this strings.

>>> r"Hello" == "Hello"
True
>>> r"\the" == "\\the"
True
>>> r"\the" == "\the"
False
>>> print(r"\the")
\the
>>> print("\the")
   he

Going back to the above regular expression case with raw string in use

regexp = re.compile(r"\\ten")

Explanation

r"\\ten" makes Python not process escape sequence, so \\ten is passed to re.compile function.
In re.compile, \\ means to form a backslash, so \\ten is ’translated’ to \ten.

Substitute Text with Regular Expressions

Regular expressions can be used for string substitution.

import re

def main():
    s = "the the quick brown fox jumps over the the the lazy fox"
    regex = re.compile(r"(\bthe\s+)+")
    processed_s = regex.sub("the ", s)
    print(processed_s)

if __name__ == "__main__":
    main()

Explanation

regex.sub("the", s) means to search pattern regex in s, and replace the matching entries to "the".
() meta characters in regular expression mean a group.

re.sub also accepts a function for more complex substition. Suppose we want to convert all three-letter words to uppercase.

import re

def main():
    s = "the quick brown fox jumps over the lazy fox"
    regex = re.compile(r"\b\w{3}\b")
    processed_s = regex.sub(lambda m: str.upper(m.group()), s)
    print(processed_s)

if __name__ == "__main__":
    main()

Explanation

regex.sub accepts a function, and a lambda expression is passed to convert the matching entry to uppercase.
re.Match object represents the matching entry, and it contains much useful info
- The whole match can be retrieved from re.Match object via group(0) method or simply group()
- The separate groups can be retrieved from re.Match object via group(<num>). For example, to retrieve the first group, use group(1).

Types

The type built-in function returns an object representing the type of the object

>>> type(523)
<class 'int'>

>>> type("Python")
<class 'str'>

>>> type(1) == type(2)
True
>>> type(1) == int
True

>>> type({}) == dict
True

The __class__ on instance and __base__ on type also return a type object.

# 1.__class__ causes error because python expects fractional part after `1.`
>>> (1).__class__
<class 'int'>

>>> (1).__class__.__base__
<class 'object'>

>>> type(1) == (1).__class__
True

isinstance(<inst>, <type>) and issubclass(<sub_class>, <super_class>) also provide useful type info.

>>> class Person:
...     pass
...

>>> class Student(Person):
...     pass
...

>>> alex = Student()
>>> isinstance(alex, Person)
True
>>> issubclass(Student, Person)
True

# notice the __main__ module is part of the full qualified name
>>> type(alex)
<class '__main__.Student'>
>>> alex.__class__.__base__
<class '__main__.Person'>

>>> type(alex).__name__
'Student'

Duck Typing

If it walks like a duck and it quacks like a duck, then it must be a duck.

This is Python’s philosophy for determining whether an object is of the desired type — focusing on its interface rather than its declared type, as languages like Java and C# do. As a result, the ntimes function below works for any type that supports multiplication operator:

>>> def ntimes(obj, times):
...     return obj * times
...

>>> ntimes(2, 3)
6
>>> ntimes("2", 3)
'222'
>>> ntimes([7, 8, 9], 3)
[7, 8, 9, 7, 8, 9, 7, 8, 9]

Dunder Methods for Type Behaviors

In Python some methods are named with leading and trailing double underscores, for example __init__, __repr__ and __str__. These methods have special meaning to Python and are not meant to be called directly. For instance, __str__ is called by Python when it needs a human-readable string representation of an object.

class Color:
    def __init__(self, red, green, blue):
        self.red = red
        self.green = green
        self.blue = blue

    def __str__(self):
        return f"Color(r={self.red}, g={self.green}, b={self.blue})"

>>> c = Color(255, 128, 23)
>>> print(c)
Color(r=255, g=128, b=23)

Packages

Python code can be organized in a module. However when a Python project scales, holding the entire project in a single module file is not a good design. Instead, code can be split into multiple module files and those module files form a package.

A package is a directory containing modules and possibly sub-directories. The name of the package is determined by the name of the directory. The qualified name of an item in a module is <package_name>.<sub_dir_name>.<module_name>.<item>. For example, suppose there is below structure

package my_package
sub-directory my_sdir
module file my_module.py
variable x defined in my_module.py

The qualified name of x is my_package.my_sdir.my_module.x.

>>> import my_package.my_sdir.my_module
>>> my_package.my_sdir.my_module.x
523

Simply referring to x as x or my_module.x causes NameError.

>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

>>> my_module.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'my_module' is not defined

`init.py` in Packages

A package is a generalization of a module. When a package is imported for the first time, its __init__.py file is executed. Given the following setup:

# my_package/__init__.py
print("Loading my_package...")

# my_package/my_sdir/__init__.py
print("Loading my_sdir...")

Importing a deeply nested module triggers each __init__.py in order, from the outermost package inward:

>>> import my_package.my_sdir.my_module
Loading my_package...
Loadin my_sdir...

On subsequent imports, Python skips execution since the packages are already cached in sys.modules:

# no output; already loaded
>>> import my_package.my_sdir.my_module
>>>

Note:

Imports are resolved level by level, from outermost to innermost.
Each __init__.py runs only once, on the first import.

Building an import Chain with `init.py`

You can use __init__.py to automatically import subpackages and modules, so that importing a top-level package pulls in everything beneath it:

# `my_package/__init__.py` file contains below contents
print("Entering my_package, importing my_sdir...")
from . import my_sdir
print("Loaded my_sdir.")


# `my_package/my_sdir/__init__.py` file contains below contents
print("Entering my_sdir, importing my_module...")
from . import my_module
print("Loaded my_module.")

Now importing just my_package triggers the entire chain:

>>> import my_package
Entering my_package, importing my_sdir...
Entering my_sdir, importing my_module...
Loaded my_module.
Loaded my_sdir.

Because the chain imported my_sdir and my_module as attributes of their parent packages, the fully qualified name resolves correctly:

>>> my_package.my_sdir.my_module.x
523

However, unqualified names are not injected into the current namespace, so these still raise a NameError:

# my_module.x and x still cause NameError
>>> my_module.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'my_module' is not defined
>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

`from <package> import *`

from <package/module> import * imports all public items from a package or module. Public items are defined by an __all__ list in __init__.py or the module file itself.

# my_package/my_sdir/my_module.py
# note items are listed as strings
__all__ = ['x'] 

x = 523
y = 311

With __all__ set to ['x'], a wildcard import brings in only x, and y is excluded:

>>> from my_package.my_sdir.my_module import *

>>> x
523

>>> y
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'y' is not defined

Use Case of Wildcard Imports

from <package/module> import * is generally considered a code smell, since it can pollute the namespace with names you didn’t explicitly request. That said, one legitimate use case is flattening a package’s public API in __init__.py. For example:

# my_package/my_sdir/my_module.py
__all__ = ['x']

x = 523
y = 311


# my_package/__init__.py
from .my_sdir.my_module import *

Now x is surfaced directly under the my_package namespace, even though it’s defined deep inside a submodule. Users can access it without knowing the internal structure:

>>> import my_package

>>> my_package.x
523

It’s worth noting that __all__ should also be defined in my_package/__init__.py itself. Without it, from my_package import * falls back to exporting everything that doesn’t start with an underscore — which is unpredictable. Adding it explicitly claims x as part of my_package’s public API at the facade level:

# my_package/__init__.py
__all__ = ['x']

from .my_sdir.my_module import *

Now the two __all__ lists work in tandem:

__all__ in my_module.py controls what the wildcard import pulls in.
__all__ in my_package/__init__.py controls what my_package itself advertises as public.

Libraries with Virtul Environment

Python follows a “batteries included” philosophy, and many useful libraries are shipped with the Python interpreter in the installer. For example:

re module: regular expression support
math and match modules: for real and complex number math
copy module: for shallow and deep copy operations
os module: for miscellaneous OS operations
pdb module: Python debugger

Install Libraries

However, there will be cases where a desired library is not installed by default. We can use pip to install it. For example, to use the requests package (Python HTTP for Humans):

python -m pip install requests

# or use `pip` command directly

pip install requests

pip searches for the module in the Python Package index(PyPI). On the web page, you can search for a desired package by name. If a specific version is needed, specifiy it in the command:

pip install requests==2.23.0

To upgrade an installed package, use the --upgrade option:

pip install --upgrade requests

Virtual Environment with `venv`

To avoid the classic version conflict issue, it is recommended to use a virtual environment. Think of a Python virtual envionrment as a self-contained directory representing an entire Python environment, including the Python interpreter and installed packages.

The venv module is included in the Python 3 installer, so it is available by default. The example below is based on Windows PowerShell:

# create a virtual environment named 'lab1-env'
python -m venv lab1-env

This command creates a lab1-env directory representing the virtual environment. To use it, you must activate it first. The activation script is located in <virtual_environment>/Scripts. Depending on your OS, the script file may differ. On Windows, there are two options:

A Powershell activation script Activate.ps1
A Command Prompt batch file activate.bat

Run the appropriate file for your shell. Once activated, the environment name appears at the beginning of your prompt:

PS C:\temp\lab1-env\Scripts> .\Activate.ps1
(lab1-env) PS C:\temp\lab1-env\Scripts>

Now pip install command will install packages into the activated virtual environment:

(lab1-env) PS C:\temp\lab1-env\Scripts> pip install requests==2.23.0
Collecting requests==2.23.0
  Downloading requests-2.23.0-py2.py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.4/58.4 kB 77.1 kB/s eta 0:00:00
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.4/133.4 kB 67.9 kB/s eta 0:00:00
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.8/58.8 kB 66.2 kB/s eta 0:00:00
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.0/128.0 kB 127.8 kB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Using cached certifi-2026.2.25-py3-none-any.whl (153 kB)
Installing collected packages: chardet, urllib3, idna, certifi, requests
Successfully installed certifi-2026.2.25 chardet-3.0.4 idna-2.10 requests-2.23.0 urllib3-1.25.11

The downloaded package files are stored in <virtual_environment>/Lib/site-packages directory.

101 on Tech Notes by ClaudeXu

Docker Essentials

Introduction

Work with Containers

Run Containers

List Containers

Inspect a Container

Remove a Container

Images

Image Layers

Reuse Layers

Build a Custom Image

ENTRYPOINT and CMD

Docker Networks

Network Operations

DNS Naming

Persistent Storage with Volumes

Data Volumes

Collision Pitfall

Bind Mounts

Data Volumes vs. Bind Mounts

Docker Compose

Example - Drupal CMS

Compose with Build

Vifm: The Vim Way to Manage Files

Introduction

Motions

Window Operations

File Operations

File Management

File Selection

Delete and Restore

Filters

Dot Files Filter

Local Filter

Permanent Filter

Sorting

:sort for Simple Sorting Order

set sort for Complex Sorting Order

Configuration

Bookmarks

File Handler

File Previewer

Metadata Columns

Interaction with Shell

Macros

htop Essentials

Basic Usage

Kill a Process

Find the Desired Process

Shortcuts

UI Setup

fd - the Cmdl Version of Everything

Intro

Alias

Usage

Search by File Full Path

Search by File Size

Search by File Type

AI 101

What is AI?

What is Machine Learning?

AI, Machine Learning & Deep Learning

AI vs. Data Science

Common AI Terms Explained

MCP vs. Skill: What’s the Difference?

Machine Learning 101

What is Machine Learning?

Matching your problem

Typical Machine Learning Flow

Evaluation

Modeling

Commonly Used Tools

Scala Quick Guide

Fundamental

Primitive Types

Expression

Evaluation

Recursion

Tail Recursion

`:sort` for Simple Sorting Order

`set sort` for Complex Sorting Order

`object` for Singleton

`for` Loop

`range`, Generator and Memory Usage

`==`/`!=` vs. `is`/`is not`

`and` and `or` Used in Non-Boolean Context

Alternative to Ternary Operator `? :`

Scope: `global` and `nonlocal`

Read and Write with `TextIOWrapper`

Read and Write with `pathlib`

Debug Program with `assert`