Python 原理笔记
Table of Contents
环境:Python 3.7.5
1. 编译 Python
$ ./configure $ make
2. 使用 LLDB 调试
例,跟踪内置函数 locals 的实现:
$ lldb ./python (lldb) target create "./python" Current executable set to './python' (x86_64). (lldb) breakpoint set --file bltinmodule.c --line 1605 Breakpoint 1: where = python`builtin_locals + 4 [inlined] builtin_locals_impl at bltinmodule.c.h:472, address = 0x00000000004d9664 (lldb) run Process 543212 launched: '/my/SourceCode/Python/Python-3.7.5/python' (x86_64) Python 3.7.5 (default, Nov 24 2019, 09:25:01) [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> locals() Process 543212 stopped * thread #1, name = 'python', stop reason = breakpoint 1.1 frame #0: 0x00000000004d9664 python`builtin_locals at bltinmodule.c:1610:9 1607 { 1608 PyObject *d; 1609 -> 1610 d = PyEval_GetLocals(); 1611 Py_XINCREF(d); 1612 return d; 1613 } (lldb)
3. 字节码
Python3 的执行原理:
解析 Python 语法 -> 生成字节码 -> Python 虚拟机执行字节码
Python 作为动态语言,解析器并不关心不同的类型,一切都是由 Python 虚拟机来操作。
我很大一部分分析工作,都是在分析字节码的实现,Python/ceval.c 就是字节码解析的实现。
要取字节码,最简单的就是用 dis 模块,如下,取 test.py 的字节码:
python3 -m dis test.py 3 0 LOAD_CONST 0 (<code object hello at 0x7f88f3bb9810, file "/tmp/test.py", line 3>) 2 LOAD_CONST 1 ('hello') 4 MAKE_FUNCTION 0 6 STORE_NAME 0 (hello) 8 LOAD_CONST 2 (None) 10 RETURN_VALUE
也可以在交互式中使用 dis 模块:
>>> import dis >>> dis.dis("print(123)") 1 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (123) 4 CALL_FUNCTION 1 6 RETURN_VALUE
其实字节码就是一堆数字而已,每个数字对应的指令可以到 Include/opcode.h 中查看,Lib/opcode.py 中也有定义。
4. 对象
Python3 所有对象都是基于 PyObject,定义在 Include/object.h 中:
typedef struct _object { _PyObject_HEAD_EXTRA Py_ssize_t ob_refcnt; /* 引用计数,用于垃圾收集 */ struct _typeobject *ob_type; /* 对象类型 */ } PyObject;
也就是对象中都包含了 PyObject,如列表(Include/listobject.h)对象的定义:
typedef struct { PyObject_VAR_HEAD PyObject **ob_item; Py_ssize_t allocated; } PyListObject;
注意 PyObject_VAR_HEAD 宏,它的定义及展开的代码如下(Include/object.h):
#define PyObject_VAR_HEAD PyVarObject ob_base; typedef struct { PyObject ob_base; Py_ssize_t ob_size; /* Number of items in variable part */ } PyVarObject;
4.1. 函数
我们取下面代码的字节码:
def hello(): print("hello world")
字节码如下:
3 0 LOAD_CONST 0 (<code object hello at 0x7f19578f9030, file "/tmp/test.py", line 3>) 3 LOAD_CONST 1 ('hello') 6 MAKE_FUNCTION 0 9 STORE_NAME 0 (hello) 12 LOAD_CONST 2 (None) 15 RETURN_VALUE
在 Python/ceval.c 中可看到函数是怎么建立的:
PyObject *qualname = POP(); /* 取函数名 */ PyObject *code = POP(); /* 获得 code 对象,所以说,代码块也是一种 Python 对象 */ PyFunctionObject *func = (PyFunctionObject *) PyFunction_NewWithQualName(codeobj, f->f_globals, qualname);
首先对当前栈帧做了 POP 字节码操作,取到了函数名,然后再做一次 POP 操作取到 code 类型的对象,可以对照上面字节码观察下栈帧内容。
注意上面代码的第二行 code 变量,即便代码本身,它也是一个 Python 对象,通过 Python 可以观察下:
>>> def hello(): print("hello world") ... ... >>> print(hello.__code__) <code object hello at 0x7f6c830bb810, file "<stdin>", line 1>
code 类型定义在 Include/code.h 中:
typedef struct { PyObject_HEAD int co_argcount; /* #arguments, except *args */ int co_kwonlyargcount; /* #keyword only arguments */ int co_nlocals; /* #local variables */ int co_stacksize; /* #entries needed for evaluation stack */ int co_flags; /* CO_..., see below */ int co_firstlineno; /* first source line number */ PyObject *co_code; /* instruction opcodes */ PyObject *co_consts; /* list (constants used) */ PyObject *co_names; /* list of strings (names used) */ PyObject *co_varnames; /* tuple of strings (local variable names) */ PyObject *co_freevars; /* tuple of strings (free variable names) */ PyObject *co_cellvars; /* tuple of strings (cell variable names) */ /* The rest aren't used in either hash or comparisons, except for co_name, used in both. This is done to preserve the name and line number for tracebacks and debuggers; otherwise, constant de-duplication would collapse identical functions/lambdas defined on different lines. */ Py_ssize_t *co_cell2arg; /* Maps cell vars which are arguments. */ PyObject *co_filename; /* unicode (where it was loaded from) */ PyObject *co_name; /* unicode (name, for reference) */ PyObject *co_lnotab; /* string (encoding addr<->lineno mapping) See Objects/lnotab_notes.txt for details. */ void *co_zombieframe; /* for optimization only (see frameobject.c) */ PyObject *co_weakreflist; /* to support weakrefs to code objects */ /* Scratch space for extra data relating to the code object. Type is a void* to keep the format private in codeobject.c to force people to go through the proper APIs. */ void *co_extra; } PyCodeObject;
可以看到 code 类型也有很多属性,不妨参考 PyCodeObject 的定义,用 Python 交互式看一看:
>>> hello.__code__.co_name 'hello' >>> hello.__code__.co_code # 函数字节码也是属性之一 b't\x00\x00d\x01\x00\x83\x01\x00\x01d\x00\x00S'
4.2. 对象引用
4.2.1. v=[1,2];v[1]=v 的打印结果
执行:
>>> v = [1, 2] >>> v[1] = v >>> v [1, [...]]
为什么结果会出现 […]?
这个和 Lisp 里处理 list 一样的道理——出现无限循环引用了,见 Objects/listobject.c 中 list_repr 函数的实现,其中一个片段:
/* list 为空的情况下 */ if (Py_SIZE(v) == 0) { return PyUnicode_FromString("[]"); } /* 检查是否有引用自身 */ i = Py_ReprEnter((PyObject*)v); if (i != 0) { return i > 0 ? PyUnicode_FromString("[...]") : NULL; }
打印 list 时,会先调用 Py_ReprEnter 函数检查是否存在无限循环,如果对象出现无限循环,就打印 […]。
如果把这段代码去掉,就无限循环打印了。
4.3. 迭代对象
示例代码:
class Hello(object): i = 0 def __iter__(self): return self def __next__(self): self.i += 1 if self.i > 10: raise StopIteration() return self.i hello = Hello() for i in hello: print(i)
如果不需要继续迭代,抛出 StopIteration 异常就可以结束循环。
5. 栈帧
每个函数在被调用时,Python 都会创建栈帧。Python 源码中,PyEval_EvalCodeEx 负责创建栈帧,PyEval_EvalCodeEx 实质上是调用 _PyEval_EvalCodeWithName 创建了新的栈帧。相关代码如下:
/* file: Python/ceval.c */ PyObject * _PyEval_EvalCodeWithName(PyObject *_co, PyObject *globals, PyObject *locals, PyObject *const *args, Py_ssize_t argcount, PyObject *const *kwnames, PyObject *const *kwargs, Py_ssize_t kwcount, int kwstep, PyObject *const *defs, Py_ssize_t defcount, PyObject *kwdefs, PyObject *closure, PyObject *name, PyObject *qualname) { PyCodeObject* co = (PyCodeObject*)_co; PyFrameObject *f; PyObject *retval = NULL; PyObject **fastlocals, **freevars; PyThreadState *tstate; PyObject *x, *u; const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount; Py_ssize_t i, n; PyObject *kwdict; if (globals == NULL) { PyErr_SetString(PyExc_SystemError, "PyEval_EvalCodeEx: NULL globals"); return NULL; } /* Create the frame */ tstate = PyThreadState_GET(); assert(tstate != NULL); /* 创建新的栈帧 */ f = _PyFrame_New_NoTrack(tstate, co, globals, locals); if (f == NULL) { return NULL; } fastlocals = f->f_localsplus; freevars = f->f_localsplus + co->co_nlocals; /* Create a dictionary for keyword parameters (**kwags) */ if (co->co_flags & CO_VARKEYWORDS) { kwdict = PyDict_New(); if (kwdict == NULL) goto fail; i = total_args; if (co->co_flags & CO_VARARGS) { i++; } SETLOCAL(i, kwdict); } else { kwdict = NULL; } /* Copy positional arguments into local variables */ if (argcount > co->co_argcount) { n = co->co_argcount; } else { n = argcount; } for (i = 0; i < n; i++) { x = args[i]; Py_INCREF(x); SETLOCAL(i, x); } /* Pack other positional arguments into the *args argument */ if (co->co_flags & CO_VARARGS) { u = PyTuple_New(argcount - n); if (u == NULL) { goto fail; } SETLOCAL(total_args, u); for (i = n; i < argcount; i++) { x = args[i]; Py_INCREF(x); PyTuple_SET_ITEM(u, i-n, x); } } /* Handle keyword arguments passed as two strided arrays */ kwcount *= kwstep; for (i = 0; i < kwcount; i += kwstep) { PyObject **co_varnames; PyObject *keyword = kwnames[i]; PyObject *value = kwargs[i]; Py_ssize_t j; if (keyword == NULL || !PyUnicode_Check(keyword)) { PyErr_Format(PyExc_TypeError, "%U() keywords must be strings", co->co_name); goto fail; } /* Speed hack: do raw pointer compares. As names are normally interned this should almost always hit. */ co_varnames = ((PyTupleObject *)(co->co_varnames))->ob_item; for (j = 0; j < total_args; j++) { PyObject *name = co_varnames[j]; if (name == keyword) { goto kw_found; } } /* Slow fallback, just in case */ for (j = 0; j < total_args; j++) { PyObject *name = co_varnames[j]; int cmp = PyObject_RichCompareBool( keyword, name, Py_EQ); if (cmp > 0) { goto kw_found; } else if (cmp < 0) { goto fail; } } assert(j >= total_args); if (kwdict == NULL) { PyErr_Format(PyExc_TypeError, "%U() got an unexpected keyword argument '%S'", co->co_name, keyword); goto fail; } if (PyDict_SetItem(kwdict, keyword, value) == -1) { goto fail; } continue; kw_found: if (GETLOCAL(j) != NULL) { PyErr_Format(PyExc_TypeError, "%U() got multiple values for argument '%S'", co->co_name, keyword); goto fail; } Py_INCREF(value); SETLOCAL(j, value); } /* Check the number of positional arguments */ if (argcount > co->co_argcount && !(co->co_flags & CO_VARARGS)) { too_many_positional(co, argcount, defcount, fastlocals); goto fail; } /* Add missing positional arguments (copy default values from defs) */ if (argcount < co->co_argcount) { Py_ssize_t m = co->co_argcount - defcount; Py_ssize_t missing = 0; for (i = argcount; i < m; i++) { if (GETLOCAL(i) == NULL) { missing++; } } if (missing) { missing_arguments(co, missing, defcount, fastlocals); goto fail; } if (n > m) i = n - m; else i = 0; for (; i < defcount; i++) { if (GETLOCAL(m+i) == NULL) { PyObject *def = defs[i]; Py_INCREF(def); SETLOCAL(m+i, def); } } } /* Add missing keyword arguments (copy default values from kwdefs) */ if (co->co_kwonlyargcount > 0) { Py_ssize_t missing = 0; for (i = co->co_argcount; i < total_args; i++) { PyObject *name; if (GETLOCAL(i) != NULL) continue; name = PyTuple_GET_ITEM(co->co_varnames, i); if (kwdefs != NULL) { PyObject *def = PyDict_GetItem(kwdefs, name); if (def) { Py_INCREF(def); SETLOCAL(i, def); continue; } } missing++; } if (missing) { missing_arguments(co, missing, -1, fastlocals); goto fail; } } /* Allocate and initialize storage for cell vars, and copy free vars into frame. */ for (i = 0; i < PyTuple_GET_SIZE(co->co_cellvars); ++i) { PyObject *c; Py_ssize_t arg; /* Possibly account for the cell variable being an argument. */ if (co->co_cell2arg != NULL && (arg = co->co_cell2arg[i]) != CO_CELL_NOT_AN_ARG) { c = PyCell_New(GETLOCAL(arg)); /* Clear the local copy. */ SETLOCAL(arg, NULL); } else { c = PyCell_New(NULL); } if (c == NULL) goto fail; SETLOCAL(co->co_nlocals + i, c); } /* Copy closure variables to free variables */ for (i = 0; i < PyTuple_GET_SIZE(co->co_freevars); ++i) { PyObject *o = PyTuple_GET_ITEM(closure, i); Py_INCREF(o); freevars[PyTuple_GET_SIZE(co->co_cellvars) + i] = o; } /* Handle generator/coroutine/asynchronous generator */ if (co->co_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR)) { PyObject *gen; PyObject *coro_wrapper = tstate->coroutine_wrapper; int is_coro = co->co_flags & CO_COROUTINE; if (is_coro && tstate->in_coroutine_wrapper) { assert(coro_wrapper != NULL); PyErr_Format(PyExc_RuntimeError, "coroutine wrapper %.200R attempted " "to recursively wrap %.200R", coro_wrapper, co); goto fail; } /* Don't need to keep the reference to f_back, it will be set * when the generator is resumed. */ Py_CLEAR(f->f_back); /* Create a new generator that owns the ready to run frame * and return that as the value. */ if (is_coro) { gen = PyCoro_New(f, name, qualname); } else if (co->co_flags & CO_ASYNC_GENERATOR) { gen = PyAsyncGen_New(f, name, qualname); } else { gen = PyGen_NewWithQualName(f, name, qualname); } if (gen == NULL) { return NULL; } _PyObject_GC_TRACK(f); if (is_coro && coro_wrapper != NULL) { PyObject *wrapped; tstate->in_coroutine_wrapper = 1; wrapped = PyObject_CallFunction(coro_wrapper, "N", gen); tstate->in_coroutine_wrapper = 0; return wrapped; } return gen; } /* 这里执行指令 */ retval = PyEval_EvalFrameEx(f,0); fail: /* Jump here from prelude on failure */ /* decref'ing the frame can cause __del__ methods to get invoked, which can call back into Python. While we're done with the current Python frame (f), the associated C stack is still in use, so recursion_depth must be boosted for the duration. */ assert(tstate != NULL); if (Py_REFCNT(f) > 1) { Py_DECREF(f); _PyObject_GC_TRACK(f); } else { ++tstate->recursion_depth; Py_DECREF(f); --tstate->recursion_depth; } return retval; }
实际上每个对象作为函数调用时,都是由 function_call(Objects/funcobject.c)完成,实现如下:
static PyObject * function_call(PyObject *func, PyObject *args, PyObject *kwargs) { PyObject **stack; Py_ssize_t nargs; stack = &PyTuple_GET_ITEM(args, 0); nargs = PyTuple_GET_SIZE(args); return _PyFunction_FastCallDict(func, stack, nargs, kwargs); }
而最终执行指令的是由 PyEval_EvalFrameEx 负责,这是一个庞大的函数,里面实现了所有的指令。
5.1. globals 和 locals 实现
Python 虚拟机通过 frame 保存了调用栈的栈帧,每个栈帧保存了当前的上下文。globals 和 locals 函数分别返回所有全局栈帧和局部栈帧的对象。globals 函数实际上返回的是第一个栈帧内的所有对象:
def test(): a = 0 print(locals()) test() print("=" * 10) print(globals())
执行输出如下:
{'a': 0} ========== {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x7fcba121f3c8>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': 'test.py', '__cached__': None, 'test': <function test at 0x7fcba924ee18>}
globals 函数实现在 Python/bltinmodule.c 中:
static PyObject * builtin_globals_impl(PyObject *module) { PyObject *d; d = PyEval_GetGlobals(); Py_XINCREF(d); return d; }
最关键是调用了 PyEval_GetGlobals 函数,该函数在 Python/ceval.c 中实现:
PyObject * PyEval_GetGlobals(void) { PyFrameObject *current_frame = PyEval_GetFrame(); /* 这里获取了当前栈的栈帧对象 */ if (current_frame == NULL) return NULL; assert(current_frame->f_globals != NULL); /* 栈针里保存了全局对象的指针 */ return current_frame->f_globals; }
PyFrameObject 定义在 Include/frameobject.h 中,PyFrameObject 保存了栈帧里所有的对象以及局部对象,locals 函数的实现如下:
PyObject * PyEval_GetLocals(void) { PyFrameObject *current_frame = PyEval_GetFrame(); if (current_frame == NULL) { PyErr_SetString(PyExc_SystemError, "frame does not exist"); return NULL; } if (PyFrame_FastToLocalsWithError(current_frame) < 0) return NULL; assert(current_frame->f_locals != NULL); return current_frame->f_locals; }
6. 从字节码角度分析
测试代码:
hello = 'nihao'
保存为 test.py,然后调用 dis 库查看执行的字节码:
$ python3 -m dis test.py 3 0 LOAD_CONST 0 ('nihao') 2 STORE_NAME 0 (hello) 4 LOAD_CONST 1 (None) 6 RETURN_VALUE
字节码操作实现位于 Python/ceval.c 中,我们找到 STORE_NAME 的实现:
TARGET(STORE_NAME) { PyObject *name = GETITEM(names, oparg); PyObject *v = POP(); PyObject *ns = f->f_locals; int err; if (ns == NULL) { PyErr_Format(PyExc_SystemError, "no locals found when storing %R", name); Py_DECREF(v); goto error; } if (PyDict_CheckExact(ns)) err = PyDict_SetItem(ns, name, v); else err = PyObject_SetItem(ns, name, v); /* 将对象存储到当前栈帧中 */ Py_DECREF(v); if (err != 0) goto error; DISPATCH(); }
注意这句:
err = PyObject_SetItem(ns, name, v);
由于我们脚本的变量定义在顶层栈帧中的,所以当前的 f->f_locals 和 f->f_globals 是指向相同的地方,可以把脚本改一下来验证:
hello = 'nihao' print(globals() == locals())
执行:
$ python3 test.py True
7. inspect 模块
inspect 模块可以很方便地获得指定对象的实现源码。
import requests print(inspect.getsource(requests.get))
输出:
def get(url, params=None, **kwargs): r"""Sends a GET request. :param url: URL for the new :class:`Request` object. :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`. :param \*\*kwargs: Optional arguments that ``request`` takes. :return: :class:`Response <Response>` object :rtype: requests.Response """ kwargs.setdefault('allow_redirects', True) return request('get', url, params=params, **kwargs)
但是仅限于对象是 Python 实现的,对于 C 实现的,目前有个名叫 cinspect 的库,可以查看函数的 C 实现,不过当前只支持 Python 2.x