伙计们,我这里有200个独立的csv文件,分别从SH(1)命名为SH(200)。我想将它们合并到单个csv文件中。我该怎么做?
伙计们,我这里有200个独立的csv文件,分别从SH(1)命名为SH(200)。我想将它们合并到单个csv文件中。我该怎么做?
Answers:
正如ghostdog74所说的,但是这次有标题:
fout=open("out.csv","a")
# first file:
for line in open("sh1.csv"):
fout.write(line)
# now the rest:
for num in range(2,201):
f = open("sh"+str(num)+".csv")
f.next() # skip the header
for line in f:
fout.write(line)
f.close() # not really needed
fout.close()
f.__next__()
如果f.next()
在python3.x中,则可以改用。
with open
语法,避免手动查看.close()
文件。
f.next()
和之间有什么区别f.__next__()
?当我使用前者时,我得到了'_io.TextIOWrapper' object has no attribute 'next'
fout.write(line)
我做之前:if line[-1] != '\n': line += '\n'
你为什么不能sed 1d sh*.csv > merged.csv
呢?
有时甚至不需要使用python!
使用公认的StackOverflow答案创建要附加的csv文件列表,然后运行以下代码:
import pandas as pd
combined_csv = pd.concat( [ pd.read_csv(f) for f in filenames ] )
如果要将其导出到单个csv文件,请使用以下命令:
combined_csv.to_csv( "combined_csv.csv", index=False )
我只是通过篮子中的另一个代码示例
from glob import glob
with open('singleDataFile.csv', 'a') as singleFile:
for csvFile in glob('*.csv'):
for line in open(csvFile, 'r'):
singleFile.write(line)
这取决于您“合并”的意思-它们是否具有相同的列?他们有标头吗?例如,如果它们都具有相同的列,但没有标题,则简单的串联就足够了(打开要写入的目标文件,循环打开每个要读取的源代码,将openil.copyfileobj从开放读取源代码中使用打开写目标,关闭源,继续循环-使用该with
语句代表您进行关闭)。如果它们具有相同的列和标题,则readline
在打开第一个源文件以进行读取之前将其复制到目标位置之前,您需要在每个源文件上的第一个文件之外都跳过一个标题行。
如果CSV文件的列均不相同,则需要定义“合并”它们的方式(例如SQL JOIN?或“水平”,如果它们都具有相同的行数?等等) )-在这种情况下,我们很难猜测您的意思。
如果合并的CSV将在Python中使用,则只需使用glob
来获取要fileinput.input()
通过files
参数传递给的文件列表,然后使用csv
模块一次性读取所有文件。
将目录中的所有文件合并并合并非常容易
import glob
import csv
# Open result file
with open('output.txt','wb') as fout:
wout = csv.writer(fout,delimiter=',')
interesting_files = glob.glob("*.csv")
h = True
for filename in interesting_files:
print 'Processing',filename
# Open and process file
with open(filename,'rb') as fin:
if h:
h = False
else:
fin.next()#skip header
for line in csv.reader(fin,delimiter=','):
wout.writerow(line)
或者,你可以做
cat sh*.csv > merged.csv
您可以导入csv,然后遍历所有CSV文件并将它们读入列表。然后将列表写回到磁盘。
import csv
rows = []
for f in (file1, file2, ...):
reader = csv.reader(open("f", "rb"))
for row in reader:
rows.append(row)
writer = csv.writer(open("some.csv", "wb"))
writer.writerows("\n".join(rows))
上面的代码不是很健壮,因为它没有错误处理,也没有关闭任何打开的文件。无论单个文件中是否包含一行或多行CSV数据,这都应该起作用。我也没有运行此代码,但是它应该使您知道该怎么做。
在使@Adders以及后来由@varun改进的解决方案上,我实现了一些小的改进,也使整个合并的CSV仅带有主标头:
from glob import glob
filename = 'main.csv'
with open(filename, 'a') as singleFile:
first_csv = True
for csv in glob('*.csv'):
if csv == filename:
pass
else:
header = True
for line in open(csv, 'r'):
if first_csv and header:
singleFile.write(line)
first_csv = False
header = False
elif header:
header = False
else:
singleFile.write(line)
singleFile.close()
最好的祝福!!!
您可以简单地使用内置csv
库。即使您的某些CSV文件的列名或标题略有不同,也可以使用此解决方案,这与其他投票最多的答案不同。
import csv
import glob
filenames = [i for i in glob.glob("SH*.csv")]
header_keys = []
merged_rows = []
for filename in filenames:
with open(filename) as f:
reader = csv.DictReader(f)
merged_rows.extend(list(reader))
header_keys.extend([key for key in reader.fieldnames if key not in header_keys])
with open("combined.csv", "w") as f:
w = csv.DictWriter(f, fieldnames=header_keys)
w.writeheader()
w.writerows(merged_rows)
合并的文件将包含header_keys
在文件中可以找到的所有可能的列()。文件中任何不存在的列都将呈现为空白/空白(但保留文件的其余数据)。
注意:
csv
库,但是除了使用DictReader
&之外DictWriter
,您还必须使用基本的reader
&writer
。merged_rows
列表)。我修改了@wisty所说的可与python 3.x一起使用的功能,对于那些有编码问题的人,我也使用os模块以避免硬编码
import os
def merge_all():
dir = os.chdir('C:\python\data\\')
fout = open("merged_files.csv", "ab")
# first file:
for line in open("file_1.csv",'rb'):
fout.write(line)
# now the rest:
list = os.listdir(dir)
number_files = len(list)
for num in range(2, number_files):
f = open("file_" + str(num) + ".csv", 'rb')
f.__next__() # skip the header
for line in f:
fout.write(line)
f.close() # not really needed
fout.close()
这是一个脚本:
SH1.csv
,以SH200.csv
import glob
import re
# Looking for filenames like 'SH1.csv' ... 'SH200.csv'
pattern = re.compile("^SH([1-9]|[1-9][0-9]|1[0-9][0-9]|200).csv$")
file_parts = [name for name in glob.glob('*.csv') if pattern.match(name)]
with open("file_merged.csv","wb") as file_merged:
for (i, name) in enumerate(file_parts):
with open(name, "rb") as file_part:
if i != 0:
next(file_part) # skip headers if not first file
file_merged.write(file_part.read())
更新wisty的python3答案
fout=open("out.csv","a")
# first file:
for line in open("sh1.csv"):
fout.write(line)
# now the rest:
for num in range(2,201):
f = open("sh"+str(num)+".csv")
next(f) # skip the header
for line in f:
fout.write(line)
f.close() # not really needed
fout.close()
假设您有2个这样的csv
文件:
csv1.csv:
id,name
1,Armin
2,Sven
csv2.csv:
id,place,year
1,Reykjavik,2017
2,Amsterdam,2018
3,Berlin,2019
并且您希望结果像这样的csv3.csv:
id,name,place,year
1,Armin,Reykjavik,2017
2,Sven,Amsterdam,2018
3,,Berlin,2019
然后,您可以使用以下代码段执行此操作:
import csv
import pandas as pd
# the file names
f1 = "csv1.csv"
f2 = "csv2.csv"
out_f = "csv3.csv"
# read the files
df1 = pd.read_csv(f1)
df2 = pd.read_csv(f2)
# get the keys
keys1 = list(df1)
keys2 = list(df2)
# merge both files
for idx, row in df2.iterrows():
data = df1[df1['id'] == row['id']]
# if row with such id does not exist, add the whole row
if data.empty:
next_idx = len(df1)
for key in keys2:
df1.at[next_idx, key] = df2.at[idx, key]
# if row with such id exists, add only the missing keys with their values
else:
i = int(data.index[0])
for key in keys2:
if key not in keys1:
df1.at[i, key] = df2.at[idx, key]
# save the merged files
df1.to_csv(out_f, index=False, encoding='utf-8', quotechar="", quoting=csv.QUOTE_NONE)
借助循环,您可以针对多个文件(200个csv文件)获得相同的结果。
如果文件没有按顺序编号,请采用以下简便的方法:Windows计算机上的Python 3.6:
import pandas as pd
from glob import glob
interesting_files = glob("C:/temp/*.csv") # it grabs all the csv files from the directory you mention here
df_list = []
for filename in sorted(interesting_files):
df_list.append(pd.read_csv(filename))
full_df = pd.concat(df_list)
# save the final file in same/different directory:
full_df.to_csv("C:/temp/merged_pandas.csv", index=False)
易于使用的功能:
def csv_merge(destination_path, *source_paths):
'''
Merges all csv files on source_paths to destination_path.
:param destination_path: Path of a single csv file, doesn't need to exist
:param source_paths: Paths of csv files to be merged into, needs to exist
:return: None
'''
with open(destination_path,"a") as dest_file:
with open(source_paths[0]) as src_file:
for src_line in src_file.read():
dest_file.write(src_line)
source_paths.pop(0)
for i in range(len(source_paths)):
with open(source_paths[i]) as src_file:
src_file.next()
for src_line in src_file:
dest_file.write(src_line)
import pandas as pd
import os
df = pd.read_csv("e:\\data science\\kaggle assign\\monthly sales\\Pandas-Data-Science-Tasks-master\\SalesAnalysis\\Sales_Data\\Sales_April_2019.csv")
files = [file for file in os.listdir("e:\\data science\\kaggle assign\\monthly sales\\Pandas-Data-Science-Tasks-master\\SalesAnalysis\\Sales_Data")
for file in files:
print(file)
all_data = pd.DataFrame()
for file in files:
df=pd.read_csv("e:\\data science\\kaggle assign\\monthly sales\\Pandas-Data-Science-Tasks-master\\SalesAnalysis\\Sales_Data\\"+file)
all_data = pd.concat([all_data,df])
all_data.head()