根据评论者的要求,将答案发布到我对类似问题的回答上,其中使用相同的技术来改变文件的最后一行,而不仅仅是获取文件。
对于很大的文件,这mmap
是最好的方法。为了改善现有mmap
答案,此版本可在Windows和Linux之间移植,并且运行速度应更快(尽管如果不对文件大小在GB范围内的32位Python进行一些修改,它将无法正常运行,请参见其他答案以获取处理此问题的提示) ,并进行修改以在Python 2上运行)。
import io # Gets consistent version of open for both Py2.7 and Py3.x
import itertools
import mmap
def skip_back_lines(mm, numlines, startidx):
'''Factored out to simplify handling of n and offset'''
for _ in itertools.repeat(None, numlines):
startidx = mm.rfind(b'\n', 0, startidx)
if startidx < 0:
break
return startidx
def tail(f, n, offset=0):
# Reopen file in binary mode
with io.open(f.name, 'rb') as binf, mmap.mmap(binf.fileno(), 0, access=mmap.ACCESS_READ) as mm:
# len(mm) - 1 handles files ending w/newline by getting the prior line
startofline = skip_back_lines(mm, offset, len(mm) - 1)
if startofline < 0:
return [] # Offset lines consumed whole file, nothing to return
# If using a generator function (yield-ing, see below),
# this should be a plain return, no empty list
endoflines = startofline + 1 # Slice end to omit offset lines
# Find start of lines to capture (add 1 to move from newline to beginning of following line)
startofline = skip_back_lines(mm, n, startofline) + 1
# Passing True to splitlines makes it return the list of lines without
# removing the trailing newline (if any), so list mimics f.readlines()
return mm[startofline:endoflines].splitlines(True)
# If Windows style \r\n newlines need to be normalized to \n, and input
# is ASCII compatible, can normalize newlines with:
# return mm[startofline:endoflines].replace(os.linesep.encode('ascii'), b'\n').splitlines(True)
假设尾行的数目足够小,您可以安全地一次将它们全部读取到内存中;您还可以将其作为生成器函数,并通过将最后一行替换为以下内容来手动读取一行:
mm.seek(startofline)
# Call mm.readline n times, or until EOF, whichever comes first
# Python 3.2 and earlier:
for line in itertools.islice(iter(mm.readline, b''), n):
yield line
# 3.3+:
yield from itertools.islice(iter(mm.readline, b''), n)
最后,以二进制模式(必须使用mmap
)进行读取,因此得到str
行(Py2)和bytes
行(Py3);如果您想要unicode
(Py2)或str
(Py3),则可以调整迭代方法为您解码和/或修复换行符:
lines = itertools.islice(iter(mm.readline, b''), n)
if f.encoding: # Decode if the passed file was opened with a specific encoding
lines = (line.decode(f.encoding) for line in lines)
if 'b' not in f.mode: # Fix line breaks if passed file opened in text mode
lines = (line.replace(os.linesep, '\n') for line in lines)
# Python 3.2 and earlier:
for line in lines:
yield line
# 3.3+:
yield from lines
注意:我在无法访问Python进行测试的机器上输入了所有内容。如果我有错字,请告诉我;这与我认为应该起作用的其他答案足够相似,但是这些调整(例如,处理)可能会导致细微的错误。如果有任何错误,请在评论中让我知道。offset
seek(0,2)
然后为tell()
),并使用该值相对于开始位置进行查找。