小能豆

Named Pipes in Linux with Python read/write miss some lines of text

py

I am learning how to use named pipes to pass data between processes. To test it out, I wrote two Python scripts, which I called pipewriter.py and pipereader.py, and a named pipe called my_pipe (using mkfifo my_pipe), all in the same directory. The Python scripts are included below.

pipewriter.py

#!/usr/bin/env python3
for i in range(1, 5):
  with open('my_pipe', 'a') as outfile:
    outfile.write(f'Hello {i}\n')
    if i==4:
      outfile.write('\n')

print('Done')

pipereader.py

#!/usr/bin/env python3
while True:
  with open('my_pipe', 'r') as infile:
    input = infile.read()
    print(input,end="")

The problem is that pipereader.py sometimes misses some of the lines output by pipewriter.py. When I run ./pipereader.py in one terminal and ./pipewriter.py in another, then the lines written are

Hello 1
Hello 2
Hello 3
Hello 4

But if I run ./pipewriter.py; ./pipewriter.py; ./pipewriter.py (with pipereader.py running in a different terminal), then the output of pipereader.py sometimes looks like this:

Hello 1
Hello 2
Hello 3
Hello 1    <- Missing "Hello 4"
Hello 2
Hello 3
Hello 4

Hello 1
Hello 2
Hello 3
Hello 4

How can I figure out why the lines are disappearing and ensure it doesn’t happen?

By rewriting ./pipewriter.py to open my_pipe only once for each execution, I can seemingly prevent missing lines. Still, I’m not sure if this is just a matter of reducing the frequency of missing lines, so I’ve just “gotten lucky” the several times I’ve tested it.

pipewriter.py (Modified)

#!/usr/bin/env python3 
with open('my_pipe', 'a') as outfile:
  for i in range(1, 5):
    outfile.write(f'Hello {i}\n')
  outfile.write('\n')

print('Done')

Edit: Output When using an Input Argument per this comment

Modified pipewriter.

#!/usr/bin/env python3
import sys

for i in range(1, 5):
  with open('my_pipe', 'a') as outfile:
    outfile.write(f'Hello {sys.argv[1]} {i}\n')
    if i==4:
      outfile.write('\n')

print('Done')

Output 1 of ./pipewriter.py A; ./pipewriter.py B; ./pipewriter.py C

Hello A 1
Hello A 2
Hello A 3
Hello A 4

Hello B 1
Hello B 2
Hello B 3
Hello B 4

Hello C 1
Hello C 2
Hello C 3

Output 2 of ./pipewriter.py A; ./pipewriter.py B; ./pipewriter.py C

Hello A 1
Hello A 2
Hello A 3
Hello A 4

Hello B 1
Hello B 2
Hello B 3
Hello B 4

Hello C 1
Hello C 2

阅读 68

收藏
2023-12-07

共1个答案

小能豆

The issue you’re encountering may be related to buffering. When you open the named pipe in append mode ('a'), it can result in unexpected behavior due to buffering. Specifically, the contents might not be immediately written to the pipe, leading to missing lines when reading.

To avoid this issue, you can try using line buffering explicitly when opening the file for writing. Modify your pipewriter.py script as follows:

#!/usr/bin/env python3
import sys

for i in range(1, 5):
    with open('my_pipe', 'a', buffering=1) as outfile:  # Set buffering to 1 for line buffering
        outfile.write(f'Hello {sys.argv[1]} {i}\n')
        if i == 4:
            outfile.write('\n')

print('Done')

This modification ensures that the output is line-buffered, meaning the data is flushed to the pipe after each newline character. This can help prevent missing lines when reading from the named pipe.

Additionally, you can experiment with the -u option when running your scripts. The -u option stands for unbuffered and can be used to force the stdin, stdout, and stderr streams to be unbuffered. Try running your scripts with this option:

./pipewriter.py | ./pipereader.py

or

python -u pipewriter.py | python -u pipereader.py

This can also help ensure that data is immediately flushed without buffering, reducing the chances of missing lines.

If the issue persists, you might want to consider using a higher-level inter-process communication method, such as sockets or queues, as named pipes can be tricky to handle correctly in some scenarios.

2023-12-07