Python 原理笔记

Table of Contents

环境:Python 3.7.5

1. 编译 Python

$ ./configure
$ make

2. 使用 LLDB 调试

例,跟踪内置函数 locals 的实现:

$ lldb ./python
(lldb) target create "./python"
Current executable set to './python' (x86_64).
(lldb) breakpoint set --file bltinmodule.c --line 1605
Breakpoint 1: where = python`builtin_locals + 4 [inlined] builtin_locals_impl at bltinmodule.c.h:472, address = 0x00000000004d9664
(lldb) run
Process 543212 launched: '/my/SourceCode/Python/Python-3.7.5/python' (x86_64)
Python 3.7.5 (default, Nov 24 2019, 09:25:01)
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> locals()
Process 543212 stopped
* thread #1, name = 'python', stop reason = breakpoint 1.1
    frame #0: 0x00000000004d9664 python`builtin_locals at bltinmodule.c:1610:9
   1607	{
   1608	    PyObject *d;
   1609
-> 1610	    d = PyEval_GetLocals();
   1611	    Py_XINCREF(d);
   1612	    return d;
   1613	}
(lldb)

3. 字节码

Python3 的执行原理:

解析 Python 语法 -> 生成字节码 -> Python 虚拟机执行字节码

Python 作为动态语言,解析器并不关心不同的类型,一切都是由 Python 虚拟机来操作。

我很大一部分分析工作,都是在分析字节码的实现,Python/ceval.c 就是字节码解析的实现。

要取字节码,最简单的就是用 dis 模块,如下,取 test.py 的字节码:

python3 -m dis test.py
  3           0 LOAD_CONST               0 (<code object hello at 0x7f88f3bb9810, file "/tmp/test.py", line 3>)
              2 LOAD_CONST               1 ('hello')
              4 MAKE_FUNCTION            0
              6 STORE_NAME               0 (hello)
              8 LOAD_CONST               2 (None)
             10 RETURN_VALUE

也可以在交互式中使用 dis 模块:

>>> import dis
>>> dis.dis("print(123)")
  1           0 LOAD_NAME                0 (print)
              2 LOAD_CONST               0 (123)
              4 CALL_FUNCTION            1
              6 RETURN_VALUE

其实字节码就是一堆数字而已,每个数字对应的指令可以到 Include/opcode.h 中查看,Lib/opcode.py 中也有定义。

4. 对象

Python3 所有对象都是基于 PyObject,定义在 Include/object.h 中:

typedef struct _object {
    _PyObject_HEAD_EXTRA
    Py_ssize_t ob_refcnt;       /* 引用计数,用于垃圾收集 */
    struct _typeobject *ob_type; /* 对象类型 */
} PyObject;

也就是对象中都包含了 PyObject,如列表(Include/listobject.h)对象的定义:

typedef struct {
    PyObject_VAR_HEAD
    PyObject **ob_item;
    Py_ssize_t allocated;
} PyListObject;

注意 PyObject_VAR_HEAD 宏,它的定义及展开的代码如下(Include/object.h):

#define PyObject_VAR_HEAD      PyVarObject ob_base;

typedef struct {
    PyObject ob_base;
    Py_ssize_t ob_size; /* Number of items in variable part */
} PyVarObject;

4.1. 函数

我们取下面代码的字节码:

def hello():
    print("hello world")

字节码如下:

3           0 LOAD_CONST               0 (<code object hello at 0x7f19578f9030, file "/tmp/test.py", line 3>)
            3 LOAD_CONST               1 ('hello')
            6 MAKE_FUNCTION            0
            9 STORE_NAME               0 (hello)
           12 LOAD_CONST               2 (None)
           15 RETURN_VALUE

在 Python/ceval.c 中可看到函数是怎么建立的:

PyObject *qualname = POP(); /* 取函数名 */
PyObject *code = POP(); /* 获得 code 对象,所以说,代码块也是一种 Python 对象 */
PyFunctionObject *func = (PyFunctionObject *)
  PyFunction_NewWithQualName(codeobj, f->f_globals, qualname);

首先对当前栈帧做了 POP 字节码操作,取到了函数名,然后再做一次 POP 操作取到 code 类型的对象,可以对照上面字节码观察下栈帧内容。

注意上面代码的第二行 code 变量,即便代码本身,它也是一个 Python 对象,通过 Python 可以观察下:

>>> def hello():
    print("hello world")
... ...
>>> print(hello.__code__)
<code object hello at 0x7f6c830bb810, file "<stdin>", line 1>

code 类型定义在 Include/code.h 中:

typedef struct {
    PyObject_HEAD
    int co_argcount;            /* #arguments, except *args */
    int co_kwonlyargcount;      /* #keyword only arguments */
    int co_nlocals;             /* #local variables */
    int co_stacksize;           /* #entries needed for evaluation stack */
    int co_flags;               /* CO_..., see below */
    int co_firstlineno;         /* first source line number */
    PyObject *co_code;          /* instruction opcodes */
    PyObject *co_consts;        /* list (constants used) */
    PyObject *co_names;         /* list of strings (names used) */
    PyObject *co_varnames;      /* tuple of strings (local variable names) */
    PyObject *co_freevars;      /* tuple of strings (free variable names) */
    PyObject *co_cellvars;      /* tuple of strings (cell variable names) */
    /* The rest aren't used in either hash or comparisons, except for co_name,
       used in both. This is done to preserve the name and line number
       for tracebacks and debuggers; otherwise, constant de-duplication
       would collapse identical functions/lambdas defined on different lines.
    */
    Py_ssize_t *co_cell2arg;    /* Maps cell vars which are arguments. */
    PyObject *co_filename;      /* unicode (where it was loaded from) */
    PyObject *co_name;          /* unicode (name, for reference) */
    PyObject *co_lnotab;        /* string (encoding addr<->lineno mapping) See
                                   Objects/lnotab_notes.txt for details. */
    void *co_zombieframe;       /* for optimization only (see frameobject.c) */
    PyObject *co_weakreflist;   /* to support weakrefs to code objects */
    /* Scratch space for extra data relating to the code object.
       Type is a void* to keep the format private in codeobject.c to force
       people to go through the proper APIs. */
    void *co_extra;
} PyCodeObject;

可以看到 code 类型也有很多属性,不妨参考 PyCodeObject 的定义,用 Python 交互式看一看:

>>> hello.__code__.co_name
'hello'
>>> hello.__code__.co_code # 函数字节码也是属性之一
b't\x00\x00d\x01\x00\x83\x01\x00\x01d\x00\x00S'

4.2. 对象引用

4.2.1. v=[1,2];v[1]=v 的打印结果

执行:

>>> v = [1, 2]
>>> v[1] = v
>>> v
[1, [...]]

为什么结果会出现 […]?

这个和 Lisp 里处理 list 一样的道理——出现无限循环引用了,见 Objects/listobject.c 中 list_repr 函数的实现,其中一个片段:

/* list 为空的情况下 */
if (Py_SIZE(v) == 0) {
  return PyUnicode_FromString("[]");
 }

/* 检查是否有引用自身 */
i = Py_ReprEnter((PyObject*)v);
if (i != 0) {
  return i > 0 ? PyUnicode_FromString("[...]") : NULL;
 }

打印 list 时,会先调用 Py_ReprEnter 函数检查是否存在无限循环,如果对象出现无限循环,就打印 […]。

如果把这段代码去掉,就无限循环打印了。

4.3. 迭代对象

示例代码:

class Hello(object):
    i = 0

    def __iter__(self):
        return self

    def __next__(self):
        self.i += 1

        if self.i > 10:
            raise StopIteration()

        return self.i


hello = Hello()

for i in hello:
    print(i)

如果不需要继续迭代,抛出 StopIteration 异常就可以结束循环。

5. 栈帧

每个函数在被调用时,Python 都会创建栈帧。Python 源码中,PyEval_EvalCodeEx 负责创建栈帧,PyEval_EvalCodeEx 实质上是调用 _PyEval_EvalCodeWithName 创建了新的栈帧。相关代码如下:

/* file: Python/ceval.c */

PyObject *
_PyEval_EvalCodeWithName(PyObject *_co, PyObject *globals, PyObject *locals,
           PyObject *const *args, Py_ssize_t argcount,
           PyObject *const *kwnames, PyObject *const *kwargs,
           Py_ssize_t kwcount, int kwstep,
           PyObject *const *defs, Py_ssize_t defcount,
           PyObject *kwdefs, PyObject *closure,
           PyObject *name, PyObject *qualname)
{
    PyCodeObject* co = (PyCodeObject*)_co;
    PyFrameObject *f;
    PyObject *retval = NULL;
    PyObject **fastlocals, **freevars;
    PyThreadState *tstate;
    PyObject *x, *u;
    const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount;
    Py_ssize_t i, n;
    PyObject *kwdict;

    if (globals == NULL) {
        PyErr_SetString(PyExc_SystemError,
                        "PyEval_EvalCodeEx: NULL globals");
        return NULL;
    }

    /* Create the frame */
    tstate = PyThreadState_GET();
    assert(tstate != NULL);
    /* 创建新的栈帧 */
    f = _PyFrame_New_NoTrack(tstate, co, globals, locals);
    if (f == NULL) {
        return NULL;
    }
    fastlocals = f->f_localsplus;
    freevars = f->f_localsplus + co->co_nlocals;

    /* Create a dictionary for keyword parameters (**kwags) */
    if (co->co_flags & CO_VARKEYWORDS) {
        kwdict = PyDict_New();
        if (kwdict == NULL)
            goto fail;
        i = total_args;
        if (co->co_flags & CO_VARARGS) {
            i++;
        }
        SETLOCAL(i, kwdict);
    }
    else {
        kwdict = NULL;
    }

    /* Copy positional arguments into local variables */
    if (argcount > co->co_argcount) {
        n = co->co_argcount;
    }
    else {
        n = argcount;
    }
    for (i = 0; i < n; i++) {
        x = args[i];
        Py_INCREF(x);
        SETLOCAL(i, x);
    }

    /* Pack other positional arguments into the *args argument */
    if (co->co_flags & CO_VARARGS) {
        u = PyTuple_New(argcount - n);
        if (u == NULL) {
            goto fail;
        }
        SETLOCAL(total_args, u);
        for (i = n; i < argcount; i++) {
            x = args[i];
            Py_INCREF(x);
            PyTuple_SET_ITEM(u, i-n, x);
        }
    }

    /* Handle keyword arguments passed as two strided arrays */
    kwcount *= kwstep;
    for (i = 0; i < kwcount; i += kwstep) {
        PyObject **co_varnames;
        PyObject *keyword = kwnames[i];
        PyObject *value = kwargs[i];
        Py_ssize_t j;

        if (keyword == NULL || !PyUnicode_Check(keyword)) {
            PyErr_Format(PyExc_TypeError,
                         "%U() keywords must be strings",
                         co->co_name);
            goto fail;
        }

        /* Speed hack: do raw pointer compares. As names are
           normally interned this should almost always hit. */
        co_varnames = ((PyTupleObject *)(co->co_varnames))->ob_item;
        for (j = 0; j < total_args; j++) {
            PyObject *name = co_varnames[j];
            if (name == keyword) {
                goto kw_found;
            }
        }

        /* Slow fallback, just in case */
        for (j = 0; j < total_args; j++) {
            PyObject *name = co_varnames[j];
            int cmp = PyObject_RichCompareBool( keyword, name, Py_EQ);
            if (cmp > 0) {
                goto kw_found;
            }
            else if (cmp < 0) {
                goto fail;
            }
        }

        assert(j >= total_args);
        if (kwdict == NULL) {
            PyErr_Format(PyExc_TypeError,
                         "%U() got an unexpected keyword argument '%S'",
                         co->co_name, keyword);
            goto fail;
        }

        if (PyDict_SetItem(kwdict, keyword, value) == -1) {
            goto fail;
        }
        continue;

      kw_found:
        if (GETLOCAL(j) != NULL) {
            PyErr_Format(PyExc_TypeError,
                         "%U() got multiple values for argument '%S'",
                         co->co_name, keyword);
            goto fail;
        }
        Py_INCREF(value);
        SETLOCAL(j, value);
    }

    /* Check the number of positional arguments */
    if (argcount > co->co_argcount && !(co->co_flags & CO_VARARGS)) {
        too_many_positional(co, argcount, defcount, fastlocals);
        goto fail;
    }

    /* Add missing positional arguments (copy default values from defs) */
    if (argcount < co->co_argcount) {
        Py_ssize_t m = co->co_argcount - defcount;
        Py_ssize_t missing = 0;
        for (i = argcount; i < m; i++) {
            if (GETLOCAL(i) == NULL) {
                missing++;
            }
        }
        if (missing) {
            missing_arguments(co, missing, defcount, fastlocals);
            goto fail;
        }
        if (n > m)
            i = n - m;
        else
            i = 0;
        for (; i < defcount; i++) {
            if (GETLOCAL(m+i) == NULL) {
                PyObject *def = defs[i];
                Py_INCREF(def);
                SETLOCAL(m+i, def);
            }
        }
    }

    /* Add missing keyword arguments (copy default values from kwdefs) */
    if (co->co_kwonlyargcount > 0) {
        Py_ssize_t missing = 0;
        for (i = co->co_argcount; i < total_args; i++) {
            PyObject *name;
            if (GETLOCAL(i) != NULL)
                continue;
            name = PyTuple_GET_ITEM(co->co_varnames, i);
            if (kwdefs != NULL) {
                PyObject *def = PyDict_GetItem(kwdefs, name);
                if (def) {
                    Py_INCREF(def);
                    SETLOCAL(i, def);
                    continue;
                }
            }
            missing++;
        }
        if (missing) {
            missing_arguments(co, missing, -1, fastlocals);
            goto fail;
        }
    }

    /* Allocate and initialize storage for cell vars, and copy free
       vars into frame. */
    for (i = 0; i < PyTuple_GET_SIZE(co->co_cellvars); ++i) {
        PyObject *c;
        Py_ssize_t arg;
        /* Possibly account for the cell variable being an argument. */
        if (co->co_cell2arg != NULL &&
            (arg = co->co_cell2arg[i]) != CO_CELL_NOT_AN_ARG) {
            c = PyCell_New(GETLOCAL(arg));
            /* Clear the local copy. */
            SETLOCAL(arg, NULL);
        }
        else {
            c = PyCell_New(NULL);
        }
        if (c == NULL)
            goto fail;
        SETLOCAL(co->co_nlocals + i, c);
    }

    /* Copy closure variables to free variables */
    for (i = 0; i < PyTuple_GET_SIZE(co->co_freevars); ++i) {
        PyObject *o = PyTuple_GET_ITEM(closure, i);
        Py_INCREF(o);
        freevars[PyTuple_GET_SIZE(co->co_cellvars) + i] = o;
    }

    /* Handle generator/coroutine/asynchronous generator */
    if (co->co_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR)) {
        PyObject *gen;
        PyObject *coro_wrapper = tstate->coroutine_wrapper;
        int is_coro = co->co_flags & CO_COROUTINE;

        if (is_coro && tstate->in_coroutine_wrapper) {
            assert(coro_wrapper != NULL);
            PyErr_Format(PyExc_RuntimeError,
                         "coroutine wrapper %.200R attempted "
                         "to recursively wrap %.200R",
                         coro_wrapper,
                         co);
            goto fail;
        }

        /* Don't need to keep the reference to f_back, it will be set
         * when the generator is resumed. */
        Py_CLEAR(f->f_back);

        /* Create a new generator that owns the ready to run frame
         * and return that as the value. */
        if (is_coro) {
            gen = PyCoro_New(f, name, qualname);
        } else if (co->co_flags & CO_ASYNC_GENERATOR) {
            gen = PyAsyncGen_New(f, name, qualname);
        } else {
            gen = PyGen_NewWithQualName(f, name, qualname);
        }
        if (gen == NULL) {
            return NULL;
        }

        _PyObject_GC_TRACK(f);

        if (is_coro && coro_wrapper != NULL) {
            PyObject *wrapped;
            tstate->in_coroutine_wrapper = 1;
            wrapped = PyObject_CallFunction(coro_wrapper, "N", gen);
            tstate->in_coroutine_wrapper = 0;
            return wrapped;
        }

        return gen;
    }
    /* 这里执行指令 */
    retval = PyEval_EvalFrameEx(f,0);

fail: /* Jump here from prelude on failure */

    /* decref'ing the frame can cause __del__ methods to get invoked,
       which can call back into Python.  While we're done with the
       current Python frame (f), the associated C stack is still in use,
       so recursion_depth must be boosted for the duration.
    */
    assert(tstate != NULL);
    if (Py_REFCNT(f) > 1) {
        Py_DECREF(f);
        _PyObject_GC_TRACK(f);
    }
    else {
        ++tstate->recursion_depth;
        Py_DECREF(f);
        --tstate->recursion_depth;
    }
    return retval;
}

实际上每个对象作为函数调用时,都是由 function_call(Objects/funcobject.c)完成,实现如下:

static PyObject *
function_call(PyObject *func, PyObject *args, PyObject *kwargs)
{
    PyObject **stack;
    Py_ssize_t nargs;

    stack = &PyTuple_GET_ITEM(args, 0);
    nargs = PyTuple_GET_SIZE(args);
    return _PyFunction_FastCallDict(func, stack, nargs, kwargs);
}

而最终执行指令的是由 PyEval_EvalFrameEx 负责,这是一个庞大的函数,里面实现了所有的指令。

5.1. globals 和 locals 实现

Python 虚拟机通过 frame 保存了调用栈的栈帧,每个栈帧保存了当前的上下文。globals 和 locals 函数分别返回所有全局栈帧和局部栈帧的对象。globals 函数实际上返回的是第一个栈帧内的所有对象:

def test():
    a = 0
    print(locals())

test()
print("=" * 10)
print(globals())

执行输出如下:

{'a': 0}
==========
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x7fcba121f3c8>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': 'test.py', '__cached__': None, 'test': <function test at 0x7fcba924ee18>}

globals 函数实现在 Python/bltinmodule.c 中:

static PyObject *
builtin_globals_impl(PyObject *module)
{
    PyObject *d;

    d = PyEval_GetGlobals();
    Py_XINCREF(d);
    return d;
}

最关键是调用了 PyEval_GetGlobals 函数,该函数在 Python/ceval.c 中实现:

PyObject *
PyEval_GetGlobals(void)
{
    PyFrameObject *current_frame = PyEval_GetFrame(); /* 这里获取了当前栈的栈帧对象 */
    if (current_frame == NULL)
        return NULL;

    assert(current_frame->f_globals != NULL); /* 栈针里保存了全局对象的指针 */
    return current_frame->f_globals;
}

PyFrameObject 定义在 Include/frameobject.h 中,PyFrameObject 保存了栈帧里所有的对象以及局部对象,locals 函数的实现如下:

PyObject *
PyEval_GetLocals(void)
{
    PyFrameObject *current_frame = PyEval_GetFrame();
    if (current_frame == NULL) {
        PyErr_SetString(PyExc_SystemError, "frame does not exist");
        return NULL;
    }

    if (PyFrame_FastToLocalsWithError(current_frame) < 0)
        return NULL;

    assert(current_frame->f_locals != NULL);
    return current_frame->f_locals;
}

6. 从字节码角度分析

测试代码:

hello = 'nihao'

保存为 test.py,然后调用 dis 库查看执行的字节码:

$ python3 -m dis test.py
  3           0 LOAD_CONST               0 ('nihao')
              2 STORE_NAME               0 (hello)
              4 LOAD_CONST               1 (None)
              6 RETURN_VALUE

字节码操作实现位于 Python/ceval.c 中,我们找到 STORE_NAME 的实现:

TARGET(STORE_NAME) {
  PyObject *name = GETITEM(names, oparg);
  PyObject *v = POP();
  PyObject *ns = f->f_locals;
  int err;
  if (ns == NULL) {
    PyErr_Format(PyExc_SystemError,
                 "no locals found when storing %R", name);
    Py_DECREF(v);
    goto error;
  }
  if (PyDict_CheckExact(ns))
    err = PyDict_SetItem(ns, name, v);
  else
    err = PyObject_SetItem(ns, name, v);  /* 将对象存储到当前栈帧中 */
  Py_DECREF(v);
  if (err != 0)
    goto error;
  DISPATCH();
}

注意这句:

err = PyObject_SetItem(ns, name, v);

由于我们脚本的变量定义在顶层栈帧中的,所以当前的 f->f_locals 和 f->f_globals 是指向相同的地方,可以把脚本改一下来验证:

hello = 'nihao'
print(globals() == locals())

执行:

$ python3 test.py
True

7. inspect 模块

inspect 模块可以很方便地获得指定对象的实现源码。

import requests
print(inspect.getsource(requests.get))

输出:

def get(url, params=None, **kwargs):
    r"""Sends a GET request.

    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response
    """

    kwargs.setdefault('allow_redirects', True)
    return request('get', url, params=params, **kwargs)

但是仅限于对象是 Python 实现的,对于 C 实现的,目前有个名叫 cinspect 的库,可以查看函数的 C 实现,不过当前只支持 Python 2.x