我正在编写一个解析文本文件的python脚本。该文本文件的格式使得文件中的每个元素都使用两行,为了方便起见,我想在解析之前先读取这两行。可以用Python完成吗?
我想要一些类似的东西:
f = open(filename, "r")
for line in f:
line1 = line
line2 = f.readline()
f.close
但这打破了这样的说法:
ValueError:混合迭代和读取方法将丢失数据
我正在编写一个解析文本文件的python脚本。该文本文件的格式使得文件中的每个元素都使用两行,为了方便起见,我想在解析之前先读取这两行。可以用Python完成吗?
我想要一些类似的东西:
f = open(filename, "r")
for line in f:
line1 = line
line2 = f.readline()
f.close
但这打破了这样的说法:
ValueError:混合迭代和读取方法将丢失数据
next(f)。
Answers:
import itertools
with open('a') as f:
for line1,line2 in itertools.zip_longest(*[f]*2):
print(line1,line2)
itertools.zip_longest() 返回一个迭代器,因此即使文件长数十亿行也可以正常工作。
如果行数为奇数,则line2在None上一次迭代中将其设置为。
在Python2上,您需要使用izip_longest。
在评论中,已经询问此解决方案是否先读取整个文件,然后再次遍历该文件。我相信事实并非如此。该with open('a') as f行打开文件句柄,但不读取文件。f是一个迭代器,因此在请求之前不会读取其内容。zip_longest将迭代器作为参数,并返回一个迭代器。
zip_longest确实给同一个迭代器f两次了。但是最终发生的是next(f)在第一个参数上调用,然后在第二个参数上调用。由于next()在同一基础迭代器上被调用,因此会产生连续的行。这与读取整个文件非常不同。确实,使用迭代器的目的恰恰是避免读取整个文件。
因此,我相信该解决方案可以按需工作-for循环只读取一次该文件。
为了证实这一点,我运行了zip_longest解决方案,而不是使用的解决方案f.readlines()。我input()在结尾处放一个脚本,然后ps axuw在每个脚本上运行:
% ps axuw | grep zip_longest_method.py
unutbu 11119 2.2 0.2 4520 2712 pts/0 S+ 21:14 0:00 python /home/unutbu/pybin/zip_longest_method.py bigfile
% ps axuw | grep readlines_method.py
unutbu 11317 6.5 8.8 93908 91680 pts/0 S+ 21:16 0:00 python /home/unutbu/pybin/readlines_method.py bigfile
在readlines明确整个文件一次读取。由于zip_longest_method使用的内存少得多,因此我可以得出结论,它不会一次读入整个文件。
(*[f]*2)它,因为它表明您只需更改数字即可获得所需的任何大小的块(因此,我不会编辑答案来更改它),但是在这种情况下(f, f),键入起来可能会更容易。
lines而不是,line1, line2则只需更改一个数字(2)一次即可读取n行。
使用next(),例如
with open("file") as f:
for line in f:
print(line)
nextline = next(f)
print("next line", nextline)
....
StopIteration被提出。
nextline = next(f,None)
我将以与ghostdog74类似的方式进行,仅需进行外部尝试并进行一些修改:
try:
with open(filename) as f:
for line1 in f:
line2 = f.next()
# process line1 and line2 here
except StopIteration:
print "(End)" # do whatever you need to do with line1 alone
这使代码简单而健壮。with如果发生其他情况,使用close可以关闭文件,或者在耗尽文件并退出循环后立即关闭资源。
请注意,启用with该with_statement功能需要2.6或2.5 。
适用于偶数和奇数长度的文件。它只是忽略了不匹配的最后一行。
f=file("file")
lines = f.readlines()
for even, odd in zip(lines[0::2], lines[1::2]):
print "even : ", even
print "odd : ", odd
print "end cycle"
f.close()
如果文件很大,这不是正确的方法。您正在使用readlines()将所有文件加载到内存中。我曾经写过一个类来读取文件,保存每个行的起始位置。这使您可以在不将所有文件都存储在内存中的情况下获得特定的行,并且还可以前进和后退。
我把它贴在这里。许可证是公共领域,意味着您可以使用它来做您想做的事情。请注意,这堂课是六年前写的,自那以后我再也没有摸过或检查过。我认为它甚至都不符合文件标准。买者自负。另外,请注意,这对于您的问题来说过于矫kill过正。我并不是说您一定要这样做,但是我有这段代码,如果您需要更复杂的访问权限,我很乐意分享。
import string
import re
class FileReader:
"""
Similar to file class, but allows to access smoothly the lines
as when using readlines(), with no memory payload, going back and forth,
finding regexps and so on.
"""
def __init__(self,filename): # fold>>
self.__file=file(filename,"r")
self.__currentPos=-1
# get file length
self.__file.seek(0,0)
counter=0
line=self.__file.readline()
while line != '':
counter = counter + 1
line=self.__file.readline()
self.__length = counter
# collect an index of filedescriptor positions against
# the line number, to enhance search
self.__file.seek(0,0)
self.__lineToFseek = []
while True:
cur=self.__file.tell()
line=self.__file.readline()
# if it's not null the cur is valid for
# identifying a line, so store
self.__lineToFseek.append(cur)
if line == '':
break
# <<fold
def __len__(self): # fold>>
"""
member function for the operator len()
returns the file length
FIXME: better get it once when opening file
"""
return self.__length
# <<fold
def __getitem__(self,key): # fold>>
"""
gives the "key" line. The syntax is
import FileReader
f=FileReader.FileReader("a_file")
line=f[2]
to get the second line from the file. The internal
pointer is set to the key line
"""
mylen = self.__len__()
if key < 0:
self.__currentPos = -1
return ''
elif key > mylen:
self.__currentPos = mylen
return ''
self.__file.seek(self.__lineToFseek[key],0)
counter=0
line = self.__file.readline()
self.__currentPos = key
return line
# <<fold
def next(self): # fold>>
if self.isAtEOF():
raise StopIteration
return self.readline()
# <<fold
def __iter__(self): # fold>>
return self
# <<fold
def readline(self): # fold>>
"""
read a line forward from the current cursor position.
returns the line or an empty string when at EOF
"""
return self.__getitem__(self.__currentPos+1)
# <<fold
def readbackline(self): # fold>>
"""
read a line backward from the current cursor position.
returns the line or an empty string when at Beginning of
file.
"""
return self.__getitem__(self.__currentPos-1)
# <<fold
def currentLine(self): # fold>>
"""
gives the line at the current cursor position
"""
return self.__getitem__(self.__currentPos)
# <<fold
def currentPos(self): # fold>>
"""
return the current position (line) in the file
or -1 if the cursor is at the beginning of the file
or len(self) if it's at the end of file
"""
return self.__currentPos
# <<fold
def toBOF(self): # fold>>
"""
go to beginning of file
"""
self.__getitem__(-1)
# <<fold
def toEOF(self): # fold>>
"""
go to end of file
"""
self.__getitem__(self.__len__())
# <<fold
def toPos(self,key): # fold>>
"""
go to the specified line
"""
self.__getitem__(key)
# <<fold
def isAtEOF(self): # fold>>
return self.__currentPos == self.__len__()
# <<fold
def isAtBOF(self): # fold>>
return self.__currentPos == -1
# <<fold
def isAtPos(self,key): # fold>>
return self.__currentPos == key
# <<fold
def findString(self, thestring, count=1, backward=0): # fold>>
"""
find the count occurrence of the string str in the file
and return the line catched. The internal cursor is placed
at the same line.
backward is the searching flow.
For example, to search for the first occurrence of "hello
starting from the beginning of the file do:
import FileReader
f=FileReader.FileReader("a_file")
f.toBOF()
f.findString("hello",1,0)
To search the second occurrence string from the end of the
file in backward movement do:
f.toEOF()
f.findString("hello",2,1)
to search the first occurrence from a given (or current) position
say line 150, going forward in the file
f.toPos(150)
f.findString("hello",1,0)
return the string where the occurrence is found, or an empty string
if nothing is found. The internal counter is placed at the corresponding
line number, if the string was found. In other case, it's set at BOF
if the search was backward, and at EOF if the search was forward.
NB: the current line is never evaluated. This is a feature, since
we can so traverse occurrences with a
line=f.findString("hello")
while line == '':
line.findString("hello")
instead of playing with a readline every time to skip the current
line.
"""
internalcounter=1
if count < 1:
count = 1
while 1:
if backward == 0:
line=self.readline()
else:
line=self.readbackline()
if line == '':
return ''
if string.find(line,thestring) != -1 :
if count == internalcounter:
return line
else:
internalcounter = internalcounter + 1
# <<fold
def findRegexp(self, theregexp, count=1, backward=0): # fold>>
"""
find the count occurrence of the regexp in the file
and return the line catched. The internal cursor is placed
at the same line.
backward is the searching flow.
You need to pass a regexp string as theregexp.
returns a tuple. The fist element is the matched line. The subsequent elements
contains the matched groups, if any.
If no match returns None
"""
rx=re.compile(theregexp)
internalcounter=1
if count < 1:
count = 1
while 1:
if backward == 0:
line=self.readline()
else:
line=self.readbackline()
if line == '':
return None
m=rx.search(line)
if m != None :
if count == internalcounter:
return (line,)+m.groups()
else:
internalcounter = internalcounter + 1
# <<fold
def skipLines(self,key): # fold>>
"""
skip a given number of lines. Key can be negative to skip
backward. Return the last line read.
Please note that skipLines(1) is equivalent to readline()
skipLines(-1) is equivalent to readbackline() and skipLines(0)
is equivalent to currentLine()
"""
return self.__getitem__(self.__currentPos+key)
# <<fold
def occurrences(self,thestring,backward=0): # fold>>
"""
count how many occurrences of str are found from the current
position (current line excluded... see skipLines()) to the
begin (or end) of file.
returns a list of positions where each occurrence is found,
in the same order found reading the file.
Leaves unaltered the cursor position.
"""
curpos=self.currentPos()
list = []
line = self.findString(thestring,1,backward)
while line != '':
list.append(self.currentPos())
line = self.findString(thestring,1,backward)
self.toPos(curpos)
return list
# <<fold
def close(self): # fold>>
self.__file.close()
# <<fold
readlines()调用也会将所有内容都拉到内存中。
file_name ='您的文件名'
file_open =打开(file_name,'r')
def处理程序(line_one,line_two):
打印(line_one,line_two)
在file_open时:
尝试:
一个= file_open.next()
两个= file_open.next()
处理程序(一,二)
除了(StopIteration):
file_open.close()
打破
while file_open:误导性的,因为while True:在这种情况下它等同于
def readnumlines(file, num=2):
f = iter(file)
while True:
lines = [None] * num
for i in range(num):
try:
lines[i] = f.next()
except StopIteration: # EOF or not enough lines available
return
yield lines
# use like this
f = open("thefile.txt", "r")
for line1, line2 in readnumlines(f):
# do something with line1 and line2
# or
for line1, line2, line3, ..., lineN in readnumlines(f, N):
# do something with N lines
我的想法是创建一个生成器,一次从文件中读取两行,并将其作为2元组返回,这意味着您可以遍历结果。
from cStringIO import StringIO
def read_2_lines(src):
while True:
line1 = src.readline()
if not line1: break
line2 = src.readline()
if not line2: break
yield (line1, line2)
data = StringIO("line1\nline2\nline3\nline4\n")
for read in read_2_lines(data):
print read
如果行数奇数,将无法完美工作,但这应该为您提供良好的轮廓。
简单的小读者。当您遍历对象时,它将成对拉成两对,然后将它们作为元组返回。您可以手动关闭它,或者当它超出范围时它将自行关闭。
class doublereader:
def __init__(self,filename):
self.f = open(filename, 'r')
def __iter__(self):
return self
def next(self):
return self.f.next(), self.f.next()
def close(self):
if not self.f.closed:
self.f.close()
def __del__(self):
self.close()
#example usage one
r = doublereader(r"C:\file.txt")
for a, h in r:
print "x:%s\ny:%s" % (a,h)
r.close()
#example usage two
for x,y in doublereader(r"C:\file.txt"):
print "x:%s\ny:%s" % (x,y)
#closes itself as soon as the loop goes out of scope