如何使用python一次从文件读取两行


80

我正在编写一个解析文本文件的python脚本。该文本文件的格式使得文件中的每个元素都使用两行,为了方便起见,我想在解析之前先读取这两行。可以用Python完成吗?

我想要一些类似的东西:

f = open(filename, "r")
for line in f:
    line1 = line
    line2 = f.readline()

f.close

但这打破了这样的说法:

ValueError:混合迭代和读取方法将丢失数据

有关:


8
将f.readline()更改为f.next()即可。
Paul


@Paul这个f.next()仍然有效吗?我收到此错误AttributeError:'_io.TextIOWrapper'对象没有属性'next'–
SKR

1
您必须在Python 3上使用@SKR next(f)
鲍里斯(Boris)

Answers:


50

这里有类似的问题。您不能混合使用迭代和读取行,因此您需要使用其中之一。

while True:
    line1 = f.readline()
    line2 = f.readline()
    if not line2: break  # EOF
    ...

47
import itertools
with open('a') as f:
    for line1,line2 in itertools.zip_longest(*[f]*2):
        print(line1,line2)

itertools.zip_longest() 返回一个迭代器,因此即使文件长数十亿行也可以正常工作。

如果行数为奇数,则line2None上一次迭代中将其设置为。

在Python2上,您需要使用izip_longest


在评论中,已经询问此解决方案是否先读取整个文件,然后再次遍历该文件。我相信事实并非如此。该with open('a') as f行打开文件句柄,但不读取文件。f是一个迭代器,因此在请求之前不会读取其内容。zip_longest将迭代器作为参数,并返回一个迭代器。

zip_longest确实给同一个迭代器f两次了。但是最终发生的是next(f)在第一个参数上调用,然后在第二个参数上调用。由于next()在同一基础迭代器上被调用,因此会产生连续的行。这与读取整个文件非常不同。确实,使用迭代器的目的恰恰是避免读取整个文件。

因此,我相信该解决方案可以按需工作-for循环只读取一次该文件。

为了证实这一点,我运行了zip_longest解决方案,而不是使用的解决方案f.readlines()。我input()在结尾处放一个脚本,然后ps axuw在每个脚本上运行:

% ps axuw | grep zip_longest_method.py

unutbu 11119 2.2 0.2 4520 2712 pts/0 S+ 21:14 0:00 python /home/unutbu/pybin/zip_longest_method.py bigfile

% ps axuw | grep readlines_method.py

unutbu 11317 6.5 8.8 93908 91680 pts/0 S+ 21:16 0:00 python /home/unutbu/pybin/readlines_method.py bigfile

readlines明确整个文件一次读取。由于zip_longest_method使用的内存少得多,因此我可以得出结论,它不会一次读入整个文件。


6
我喜欢(*[f]*2)它,因为它表明您只需更改数字即可获得所需的任何大小的块(因此,我不会编辑答案来更改它),但是在这种情况下(f, f),键入起来可能会更容易。
史蒂夫·洛斯

如果您使用lines而不是,line1, line2则只需更改一个数字(2)一次即可读取n行。
jfs

26

使用next(),例如

with open("file") as f:
    for line in f:
        print(line)
        nextline = next(f)
        print("next line", nextline)
        ....

1
正如RedGlyph在他的答案版本中指出的那样,奇数行将导致StopIteration被提出。
drevicko 2015年

2
next()现在支持默认参数以避免该异常:nextline = next(f,None)
gerardw

11

我将以与ghostdog74类似的方式进行,仅需进行外部尝试并进行一些修改:

try:
    with open(filename) as f:
        for line1 in f:
            line2 = f.next()
            # process line1 and line2 here
except StopIteration:
    print "(End)" # do whatever you need to do with line1 alone

这使代码简单而健壮。with如果发生其他情况,使用close可以关闭文件,或者在耗尽文件并退出循环后立即关闭资源。

请注意,启用withwith_statement功能需要2.6或2.5 。


8

这个怎么样,任何人看到它的问题

with open('file_name') as f:
    for line1, line2 in zip(f, f):
        print(line1, line2)

1
如果文件的行数为奇数,则将丢弃最后一行。令人高兴的是,您可以将其扩展为一次读取3行,for l1, l2, l3 in zip(f, f, f):以此类推;同样,如果行数不能被3整除,则最后1或2行将被丢弃。–
Boris

4

适用于偶数和奇数长度的文件。它只是忽略了不匹配的最后一行。

f=file("file")

lines = f.readlines()
for even, odd in zip(lines[0::2], lines[1::2]):
    print "even : ", even
    print "odd : ", odd
    print "end cycle"
f.close()

如果文件很大,这不是正确的方法。您正在使用readlines()将所有文件加载到内存中。我曾经写过一个类来读取文件,保存每个行的起始位置。这使您可以在不将所有文件都存储在内存中的情况下获得特定的行,并且还可以前进和后退。

我把它贴在这里。许可证是公共领域,意味着您可以使用它来做您想做的事情。请注意,这堂课是六年前写的,自那以后我再也没有摸过或检查过。我认为它甚至都不符合文件标准。买者自负。另外,请注意,这对于您的问题来说过于矫kill过正。我并不是说您一定要这样做,但是我有这段代码,如果您需要更复杂的访问权限,我很乐意分享。

import string
import re

class FileReader:
    """ 
    Similar to file class, but allows to access smoothly the lines 
    as when using readlines(), with no memory payload, going back and forth,
    finding regexps and so on.
    """
    def __init__(self,filename): # fold>>
        self.__file=file(filename,"r")
        self.__currentPos=-1
        # get file length
        self.__file.seek(0,0)
        counter=0
        line=self.__file.readline()
        while line != '':
            counter = counter + 1
            line=self.__file.readline()
        self.__length = counter
        # collect an index of filedescriptor positions against
        # the line number, to enhance search
        self.__file.seek(0,0)
        self.__lineToFseek = []

        while True:
            cur=self.__file.tell()
            line=self.__file.readline()
            # if it's not null the cur is valid for
            # identifying a line, so store
            self.__lineToFseek.append(cur)
            if line == '':
                break
    # <<fold
    def __len__(self): # fold>>
        """
        member function for the operator len()
        returns the file length
        FIXME: better get it once when opening file
        """
        return self.__length
        # <<fold
    def __getitem__(self,key): # fold>>
        """ 
        gives the "key" line. The syntax is

        import FileReader
        f=FileReader.FileReader("a_file")
        line=f[2]

        to get the second line from the file. The internal
        pointer is set to the key line
        """

        mylen = self.__len__()
        if key < 0:
            self.__currentPos = -1
            return ''
        elif key > mylen:
            self.__currentPos = mylen
            return ''

        self.__file.seek(self.__lineToFseek[key],0)
        counter=0
        line = self.__file.readline()
        self.__currentPos = key
        return line
        # <<fold
    def next(self): # fold>>
        if self.isAtEOF():
            raise StopIteration
        return self.readline()
    # <<fold
    def __iter__(self): # fold>>
        return self
    # <<fold
    def readline(self): # fold>>
        """
        read a line forward from the current cursor position.
        returns the line or an empty string when at EOF
        """
        return self.__getitem__(self.__currentPos+1)
        # <<fold
    def readbackline(self): # fold>>
        """
        read a line backward from the current cursor position.
        returns the line or an empty string when at Beginning of
        file.
        """
        return self.__getitem__(self.__currentPos-1)
        # <<fold
    def currentLine(self): # fold>>
        """
        gives the line at the current cursor position
        """
        return self.__getitem__(self.__currentPos)
        # <<fold
    def currentPos(self): # fold>>
        """ 
        return the current position (line) in the file
        or -1 if the cursor is at the beginning of the file
        or len(self) if it's at the end of file
        """
        return self.__currentPos
        # <<fold
    def toBOF(self): # fold>>
        """
        go to beginning of file
        """
        self.__getitem__(-1)
        # <<fold
    def toEOF(self): # fold>>
        """
        go to end of file
        """
        self.__getitem__(self.__len__())
        # <<fold
    def toPos(self,key): # fold>>
        """
        go to the specified line
        """
        self.__getitem__(key)
        # <<fold
    def isAtEOF(self): # fold>>
        return self.__currentPos == self.__len__()
        # <<fold
    def isAtBOF(self): # fold>>
        return self.__currentPos == -1
        # <<fold
    def isAtPos(self,key): # fold>>
        return self.__currentPos == key
        # <<fold

    def findString(self, thestring, count=1, backward=0): # fold>>
        """
        find the count occurrence of the string str in the file
        and return the line catched. The internal cursor is placed
        at the same line.
        backward is the searching flow.
        For example, to search for the first occurrence of "hello
        starting from the beginning of the file do:

        import FileReader
        f=FileReader.FileReader("a_file")
        f.toBOF()
        f.findString("hello",1,0)

        To search the second occurrence string from the end of the
        file in backward movement do:

        f.toEOF()
        f.findString("hello",2,1)

        to search the first occurrence from a given (or current) position
        say line 150, going forward in the file 

        f.toPos(150)
        f.findString("hello",1,0)

        return the string where the occurrence is found, or an empty string
        if nothing is found. The internal counter is placed at the corresponding
        line number, if the string was found. In other case, it's set at BOF
        if the search was backward, and at EOF if the search was forward.

        NB: the current line is never evaluated. This is a feature, since
        we can so traverse occurrences with a

        line=f.findString("hello")
        while line == '':
            line.findString("hello")

        instead of playing with a readline every time to skip the current
        line.
        """
        internalcounter=1
        if count < 1:
            count = 1
        while 1:
            if backward == 0:
                line=self.readline()
            else:
                line=self.readbackline()

            if line == '':
                return ''
            if string.find(line,thestring) != -1 :
                if count == internalcounter:
                    return line
                else:
                    internalcounter = internalcounter + 1
                    # <<fold
    def findRegexp(self, theregexp, count=1, backward=0): # fold>>
        """
        find the count occurrence of the regexp in the file
        and return the line catched. The internal cursor is placed
        at the same line.
        backward is the searching flow.
        You need to pass a regexp string as theregexp.
        returns a tuple. The fist element is the matched line. The subsequent elements
        contains the matched groups, if any.
        If no match returns None
        """
        rx=re.compile(theregexp)
        internalcounter=1
        if count < 1:
            count = 1
        while 1:
            if backward == 0:
                line=self.readline()
            else:
                line=self.readbackline()

            if line == '':
                return None
            m=rx.search(line)
            if m != None :
                if count == internalcounter:
                    return (line,)+m.groups()
                else:
                    internalcounter = internalcounter + 1
    # <<fold
    def skipLines(self,key): # fold>>
        """
        skip a given number of lines. Key can be negative to skip
        backward. Return the last line read.
        Please note that skipLines(1) is equivalent to readline()
        skipLines(-1) is equivalent to readbackline() and skipLines(0)
        is equivalent to currentLine()
        """
        return self.__getitem__(self.__currentPos+key)
    # <<fold
    def occurrences(self,thestring,backward=0): # fold>>
        """
        count how many occurrences of str are found from the current
        position (current line excluded... see skipLines()) to the
        begin (or end) of file.
        returns a list of positions where each occurrence is found,
        in the same order found reading the file.
        Leaves unaltered the cursor position.
        """
        curpos=self.currentPos()
        list = []
        line = self.findString(thestring,1,backward)
        while line != '':
            list.append(self.currentPos())
            line = self.findString(thestring,1,backward)
        self.toPos(curpos)
        return list
        # <<fold
    def close(self): # fold>>
        self.__file.close()
    # <<fold

您可能要改用itertools.izip(),特别是对于大文件!
RedGlyph

即使使用izip,对列表进行这样的切片也会将所有内容拉入内存。
史蒂夫·洛斯

实际上,该readlines()调用也会将所有内容都拉到内存中。
史蒂夫·洛斯

我不喜欢你上课 在初始化文件时,您要遍历整个文件两次。对于行短的大文件,节省的内存不多。
乔治·Schölly

@Steve:是的,非常可悲。但是zip会通过创建元组的整个列表(除非是Python 3)在内存中增加一层,izip会一次生成一个元组。我认为这就是您的意思,但是无论如何我还是想澄清一下我以前的评论:-)
RedGlyph

3
file_name ='您的文件名'
file_open =打开(file_name,'r')

def处理程序(line_one,line_two):
    打印(line_one,line_two)

在file_open时:
    尝试:
        一个= file_open.next()
        两个= file_open.next() 
        处理程序(一,二)
    除了(StopIteration):
        file_open.close()
        打破

1
while file_open:误导性的,因为while True:在这种情况下它等同于
jfs

这是有意的,尽管我同意做“ while True”表示您需要休息才能跳出循环,这可以说是更清洁的说法。我之所以选择不这样做,是因为我相信(再次有争议)它的读取方式会更好,毫无疑问,文件需要保持打开状态多长时间,与此同时该怎么做。不过,大多数时候我也会为自己做“ while True”。
Martin P. Hellwig,2009年

2
def readnumlines(file, num=2):
    f = iter(file)
    while True:
        lines = [None] * num
        for i in range(num):
            try:
                lines[i] = f.next()
            except StopIteration: # EOF or not enough lines available
                return
        yield lines

# use like this
f = open("thefile.txt", "r")
for line1, line2 in readnumlines(f):
    # do something with line1 and line2

# or
for line1, line2, line3, ..., lineN in readnumlines(f, N):
    # do something with N lines

1

我的想法是创建一个生成器,一次从文件中读取两行,并将其作为2元组返回,这意味着您可以遍历结果。

from cStringIO import StringIO

def read_2_lines(src):   
    while True:
        line1 = src.readline()
        if not line1: break
        line2 = src.readline()
        if not line2: break
        yield (line1, line2)


data = StringIO("line1\nline2\nline3\nline4\n")
for read in read_2_lines(data):
    print read

如果行数奇数,将无法完美工作,但这应该为您提供良好的轮廓。


1

我上个月曾处理过类似的问题。我尝试了f.readline()和f.readlines()的while循环。我的数据文件不是很大,所以我最终选择了f.readlines(),它使我可以更好地控制索引,否则我必须使用f.seek()来回移动文件指针。

我的情况比OP要复杂。由于我的数据文件在每次要解析多少行时更加灵活,因此在解析数据之前,我必须检查一些条件。

我发现的关于f.seek()的另一个问题是,当我使用codecs.open('','r','utf-8')时,它无法很好地处理utf-8,(不确定罪魁祸首,最终我放弃了这种方法。)


1

简单的小读者。当您遍历对象时,它将成对拉成两对,然后将它们作为元组返回。您可以手动关闭它,或者当它超出范围时它将自行关闭。

class doublereader:
    def __init__(self,filename):
        self.f = open(filename, 'r')
    def __iter__(self):
        return self
    def next(self):
        return self.f.next(), self.f.next()
    def close(self):
        if not self.f.closed:
            self.f.close()
    def __del__(self):
        self.close()

#example usage one
r = doublereader(r"C:\file.txt")
for a, h in r:
    print "x:%s\ny:%s" % (a,h)
r.close()

#example usage two
for x,y in doublereader(r"C:\file.txt"):
    print "x:%s\ny:%s" % (x,y)
#closes itself as soon as the loop goes out of scope

1
f = open(filename, "r")
for line in f:
    line1 = line
    f.next()

f.close

现在,您可以每两行读取一次文件。如果愿意,您还可以在之前检查f状态f.next()


0

如果文件的大小合理,则使用列表理解将整个文件读入2元组列表的另一种方法是:

filaname = '/path/to/file/name'

with open(filename, 'r') as f:
    list_of_2tuples = [ (line,f.readline()) for line in f ]

for (line1,line2) in list_of_2tuples: # Work with them in pairs.
    print('%s :: %s', (line1,line2))

-2

此Python代码将输出前两行:

import linecache  
filename = "ooxx.txt"  
print(linecache.getline(filename,2))
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.