如何停止Spark控制台上显示的INFO消息?


179

我想停止Spark Shell上出现的各种消息。

我试图编辑log4j.properties文件以停止这些消息。

这是内容 log4j.properties

# Define the root logger with appender file
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

但是消息仍在控制台上显示。

这是一些示例消息

15/01/05 15:11:45 INFO SparkEnv: Registering BlockManagerMaster
15/01/05 15:11:45 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150105151145-b1ba
15/01/05 15:11:45 INFO MemoryStore: MemoryStore started with capacity 0.0 B.
15/01/05 15:11:45 INFO ConnectionManager: Bound socket to port 44728 with id = ConnectionManagerId(192.168.100.85,44728)
15/01/05 15:11:45 INFO BlockManagerMaster: Trying to register BlockManager
15/01/05 15:11:45 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager 192.168.100.85:44728 with 0.0 B RAM
15/01/05 15:11:45 INFO BlockManagerMaster: Registered BlockManager
15/01/05 15:11:45 INFO HttpServer: Starting HTTP Server
15/01/05 15:11:45 INFO HttpBroadcast: Broadcast server star

如何停止这些?

Answers:


167

编辑conf/log4j.properties文件并更改以下行:

log4j.rootCategory=INFO, console

log4j.rootCategory=ERROR, console

另一种方法是:

启动spark-shell并输入以下内容:

import org.apache.log4j.Logger
import org.apache.log4j.Level

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

之后,您将看不到任何日志。

对于级别其他选项包括:alldebugerrorfatalinfoofftracetrace_intwarn

有关每种方法的详细信息,请参见文档。


15
我认为关闭过于严格。警告或错误可能更适合此处。
snowindy

2
将其添加到您的项目Main class中。
AkhlD

2
好答案。有什么办法可以通过编程方式从PySpark做同样的事情?
塔加

2
此功能的程序化部分无效。而是从@cantdutchthis看到这个答案stackoverflow.com/a/37836847/1056563
javadba

1
@AkhlD是谁?
阿维亚德·克莱因
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.