当多个进程试图同时写入文件然后从文件中读取文件时，如何防止出现竞争状况

一尘不染

当多个进程试图同时写入文件然后从文件中读取文件时，如何防止出现竞争状况

python

我有以下代码（为清楚起见已简化）：

import os
import errno
import imp


lib_dir = os.path.expanduser('~/.brian/cython_extensions')
module_name = '_cython_magic_5'
module_path = os.path.join(lib_dir, module_name + '.so')
code = 'some code'

have_module = os.path.isfile(module_path)
if not have_module:
    pyx_file = os.path.join(lib_dir, module_name + '.pyx')

    # THIS IS WHERE EACH PROCESS TRIES TO WRITE TO THE FILE.  THE CODE HERE 
    # PREVENTS A RACE CONDITION.
    try:
        fd = os.open(pyx_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
    except OSError as e:
        if e.errno == errno.EEXIST:
            pass
        else:
            raise
    else:
        os.fdopen(fd, 'w').write(code)

# THIS IS WHERE EACH PROCESS TRIES TO READ FROM THE FILE.  CURRENTLY THERE IS A
# RACE CONDITION.
module = imp.load_dynamic(module_name, module_path)

一次运行多个进程时，此代码仅导致一个进程打开并写入pyx_file（假定pyx_file尚不存在）。问题在于，当该进程正在写入时pyx_file，其他进程会尝试加载pyx_file-在后者的进程中会引发错误，因为在读取时pyx_file，它是不完整的。（特别ImportError是，引发s，因为进程正在尝试导入文件的内容。）

避免这些错误的最佳方法是什么？一种想法是让进程继续尝试pyx_file在while循环中进行导入，直到成功导入为止。（此解决方案似乎不是最优的。）

阅读 200

2021-01-20

共1个答案

一尘不染

这样做的方法是每次打开时都使用排他锁。写入器在写入数据时会持有锁，而读取器会阻塞直到写入器通过fdclose调用释放锁。如果文件已被部分写入并且写入过程异常退出，那么这当然会失败，因此，如果无法加载模块，则会显示删除文件的适当错误：

import os
import fcntl as F

def load_module():
    pyx_file = os.path.join(lib_dir, module_name + '.pyx')

    try:
        # Try and create/open the file only if it doesn't exist.
        fd = os.open(pyx_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY):

        # Lock the file exclusively to notify other processes we're writing still.
        F.flock(fd, F.LOCK_EX)
        with os.fdopen(fd, 'w') as f:
            f.write(code)

    except OSError as e:
        # If the error wasn't EEXIST we should raise it.
        if e.errno != errno.EEXIST:
            raise

    # The file existed, so let's open it for reading and then try and
    # lock it. This will block on the LOCK_EX above if it's held by
    # the writing process.
    with file(pyx_file, "r") as f:
        F.flock(f, F.LOCK_EX)

    return imp.load_dynamic(module_name, module_path)

module = load_module()

2021-01-20