如何将OutputStream转换为InputStream?


336

我正处于开发阶段,在那里我有两个模块,从一个模块我得到的输出为a OutputStream,第二个模块仅接受InputStream。您知道如何转换OutputStreamInputStream可以连接这两部分的方法吗?(反之亦然,我是说真的)

谢谢



3
@ c0mrade,操作员仅在另一个方向上需要IOUtils.copy之类的东西。当某人写入OutputStream时,其他人可以在InputStream中使用它。基本上,这就是PipedOutputStream / PipedInputStream所做的。不幸的是,不能从其他流中构建管道流。
MeBigFatGuy 2011年

那么PipedOutputStream / PipedInputStream是解决方案吗?
Waypoint,

基本上,为了使PipedStreams能够在您的情况下工作,您的OutputStream需要像这样构造,new YourOutputStream(thePipedOutputStream)并且new YourInputStream(thePipedInputStream)这可能不是流的工作方式。所以我认为这不是解决方案。
MeBigFatGuy 2011年

Answers:


109

An OutputStream是您向其中写入数据的地方。如果某个模块公开了OutputStream,则期望在另一端读取一些内容。

InputStream另一方面,暴露出的信息表示您需要侦听此流,并且会有一些数据可以读取。

因此,它是可能的连接InputStreamOutputStream

InputStream----read---> intermediateBytes[n] ----write----> OutputStream

正如有人提到的那样,这就是IOUtilscopy()方法可以让您完成的工作。走另一条路没有道理...希望这有道理

更新:

当然,我对这一点的思考越多,我越能看到这实际上是一项要求。我知道一些提及Piped输入/输出流的评论,但是还有另一种可能性。

如果公开的输出流是ByteArrayOutputStream,那么您始终可以通过调用toByteArray()方法来获取全部内容。然后,您可以使用ByteArrayInputStream子类创建输入流包装器。这两个都是伪流,它们基本上都只是包装一个字节数组。因此,以这种方式使用流在技术上是可行的,但对我来说仍然很奇怪...


4
copy()根据API将其执行到OS,我需要它向后执行
Waypoint,

1
请参阅顶部的编辑,我有必要进行一些转换
Waypoint,

86
用例非常简单:假设您有一个序列化库(例如,序列化为JSON)和一个使用InputStream的传输层(例如,Tomcat)。因此,您需要通过HTTP连接从JSON传递来自JSON的OutputStream,该HTTP连接要从InputStream中读取。
JBCP

6
当进行单元测试时,这很有用,并且您对避免接触文件系统非常着迷。
2014年

26
@JBCP的评论是当场。另一个用例是在HTTP请求期间使用PDFBox生成PDF。PDFBox使用OutputStream保存PDF对象,并且REST API接受InputStream来回复客户端。因此,OutputStream-> InputStream是一个非常实际的用例。
John Manko

200

似乎有许多链接和其他类似内容,但没有使用管道的实际代码。使用java.io.PipedInputStream和的优点java.io.PipedOutputStream是没有额外的内存消耗。ByteArrayOutputStream.toByteArray()返回原始缓冲区的副本,这意味着内存中的任何内容现在都有两个副本。然后写一个数据InputStream意味着您现在有了数据的三个副本。

编码:

// take the copy of the stream and re-write it to an InputStream
PipedInputStream in = new PipedInputStream();
final PipedOutputStream out = new PipedOutputStream(in);
new Thread(new Runnable() {
    public void run () {
        try {
            // write the original OutputStream to the PipedOutputStream
            // note that in order for the below method to work, you need
            // to ensure that the data has finished writing to the
            // ByteArrayOutputStream
            originalByteArrayOutputStream.writeTo(out);
        }
        catch (IOException e) {
            // logging and exception handling should go here
        }
        finally {
            // close the PipedOutputStream here because we're done writing data
            // once this thread has completed its run
            if (out != null) {
                // close the PipedOutputStream cleanly
                out.close();
            }
        }   
    }
}).start();

该代码假定originalByteArrayOutputStreama是a,ByteArrayOutputStream因为它通常是唯一可用的输出流,除非您要写入文件。我希望这有帮助!这样做的好处在于,由于它在一个单独的线程中,因此它也可以并行工作,因此,消耗您输入流的任何内容也将从您的旧输出流中流出。这是有益的,因为缓冲区可以保持较小,并且您将拥有更少的延迟和更少的内存使用量。


21
我对此表示赞成,但最好将其传递outin的构造函数,否则您可能会in由于竞争条件(我经历过)而收到封闭管道异常。使用Java 8 Lambda表达式:PipedInputStream in = new PipedInputStream(out); ((Runnable)() -> {originalOutputStream.writeTo(out);}).run(); return in;
约翰Manko

1
@JohnManko嗯...我从来没有遇到过这个问题。您是否因为其他线程或主线程正在调用out.close()而遇到了这种情况?的确,此代码假定您的PipedOutputStream的寿命比originalOutputStream应为true 的寿命长,但没有假定您如何控制流。这留给开发人员。这段代码中没有任何内容会导致封闭或损坏的管道异常。
mikeho

3
不,我的情况源于我将PDF存储在Mongo GridFS中,然后使用Jax-RS流传输到客户端的情况。MongoDB提供了OutputStream,但是Jax-RS需要InputStream。我的path方法似乎会在OutputStream完全建立之前返回带有InputStream的容器(似乎缓冲区可能尚未缓存)。无论如何,Jax-RS会在InputStream上引发管道关闭异常。奇怪,但这就是一半时间发生的情况。更改为上面的代码可以防止这种情况。
约翰·曼科

1
@JohnManko我正在研究更多内容,并且从PipedInputStreamJavadocs中看到:如果正在向连接的管道输出流提供数据字节的线程不再存在,则据说管道已损坏。因此,我怀疑如果您使用的是上面的示例,则该线程在Jax-RS使用输入流之前就已经完成了。同时,我查看了MongoDB Javadocs。GridFSDBFile有一个输入流,那么为什么不将其传递给Jax-RS
mikeho

3
@DennisCheung是的,当然。没有什么是免费的,但是肯定会小于15MB。优化将包括使用线程池来减少具有恒定线程/对象创建的GC流失率。
mikeho '16

40

由于输入和输出流只是起点和终点,因此解决方案是将数据临时存储在字节数组中。因此,您必须创建中间体ByteArrayOutputStream,从中创建中间体,并将其byte[]用作新的输入ByteArrayInputStream

public void doTwoThingsWithStream(InputStream inStream, OutputStream outStream){ 
  //create temporary bayte array output stream
  ByteArrayOutputStream baos = new ByteArrayOutputStream();
  doFirstThing(inStream, baos);
  //create input stream from baos
  InputStream isFromFirstData = new ByteArrayInputStream(baos.toByteArray()); 
  doSecondThing(isFromFirstData, outStream);
}

希望能帮助到你。


baos.toByteArray()使用System.arraycopy创建一个副本。由于@mikeho您指出developer.classpath.org/doc/java/io/...
米蒂亚Gustin的

20

您将需要一个中间类,将在它们之间进行缓冲。每次InputStream.read(byte[]...)调用时,缓冲类都会使用从中传入的下一个块填充传入的字节数组。OutputStream.write(byte[]...)。由于块的大小可能不相同,因此适配器类将需要存储一定数量,直到它足以填充读取缓冲区和/或能够存储任何缓冲区溢出为止。

本文详细介绍了解决此问题的几种不同方法:

http://blog.ostermiller.org/convert-java-outputstream-inputstream


1
谢谢@mckamey,基于循环缓冲区的方法正是我所需要的!
王晖

18
ByteArrayOutputStream buffer = (ByteArrayOutputStream) aOutputStream;
byte[] bytes = buffer.toByteArray();
InputStream inputStream = new ByteArrayInputStream(bytes);

2
您不应该使用此toByteArray()方法,因为方法体是这样的return Arrays.copyOf(buf, count);,它将返回一个新数组。
Root G


9

我在将a转换ByteArrayOutputStream为a时遇到了相同的问题,ByteArrayInputStream并通过使用派生类解决了该问题,该派生类ByteArrayOutputStream可以返回ByteArrayInputStream使用的内部缓冲区初始化的a ByteArrayOutputStream。这样,不使用额外的内存,并且“转换”非常快:

package info.whitebyte.utils;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;

/**
 * This class extends the ByteArrayOutputStream by 
 * providing a method that returns a new ByteArrayInputStream
 * which uses the internal byte array buffer. This buffer
 * is not copied, so no additional memory is used. After
 * creating the ByteArrayInputStream the instance of the
 * ByteArrayInOutStream can not be used anymore.
 * <p>
 * The ByteArrayInputStream can be retrieved using <code>getInputStream()</code>.
 * @author Nick Russler
 */
public class ByteArrayInOutStream extends ByteArrayOutputStream {
    /**
     * Creates a new ByteArrayInOutStream. The buffer capacity is
     * initially 32 bytes, though its size increases if necessary.
     */
    public ByteArrayInOutStream() {
        super();
    }

    /**
     * Creates a new ByteArrayInOutStream, with a buffer capacity of
     * the specified size, in bytes.
     *
     * @param   size   the initial size.
     * @exception  IllegalArgumentException if size is negative.
     */
    public ByteArrayInOutStream(int size) {
        super(size);
    }

    /**
     * Creates a new ByteArrayInputStream that uses the internal byte array buffer 
     * of this ByteArrayInOutStream instance as its buffer array. The initial value 
     * of pos is set to zero and the initial value of count is the number of bytes 
     * that can be read from the byte array. The buffer array is not copied. This 
     * instance of ByteArrayInOutStream can not be used anymore after calling this
     * method.
     * @return the ByteArrayInputStream instance
     */
    public ByteArrayInputStream getInputStream() {
        // create new ByteArrayInputStream that respects the current count
        ByteArrayInputStream in = new ByteArrayInputStream(this.buf, 0, this.count);

        // set the buffer of the ByteArrayOutputStream 
        // to null so it can't be altered anymore
        this.buf = null;

        return in;
    }
}

我把东西放在github上:https : //github.com/nickrussler/ByteArrayInOutStream


如果内容不适合缓冲区怎么办?
瓦迪莫2014年

然后,您不应该首先使用ByteArrayInputStream。
Nick Russler 2014年

此解决方案将在内存中存储所有字节。对于小文件,这是可以的,但是您也可以在ByteArrayOutput Stream上使用getBytes()
Vadimo 2014年

1
如果您的意思是toByteArray,这将导致内部缓冲区被复制,这将占用我的方法两倍的内存。编辑:嗯,我知道,对于小文件,这当然是
可行的。–尼克·罗斯勒

浪费时间。ByteArrayOutputStream具有writeTo方法,可将内容传输到另一个输出流
Tony BenBrahim,2015年

3

io-extras可能有用。例如,如果您要对InputStreamusing 进行gzip压缩,GZIPOutputStream并且希望它同步发生(使用默认缓冲区大小8192):

InputStream is = ...
InputStream gz = IOUtil.pipe(is, o -> new GZIPOutputStream(o));

请注意,该库具有100%的单元测试覆盖率(这当然是值得的!),并且在Maven Central上。Maven依赖项是:

<dependency>
  <groupId>com.github.davidmoten</groupId>
  <artifactId>io-extras</artifactId>
  <version>0.1</version>
</dependency>

确保检查更高版本。


0

从我的角度来看,java.io.PipedInputStream / java.io.PipedOutputStream是考虑的最佳选择。在某些情况下,您可能需要使用ByteArrayInputStream / ByteArrayOutputStream。问题是您需要复制缓冲区以将ByteArrayOutputStream转换为ByteArrayInputStream。ByteArrayOutpuStream / ByteArrayInputStream也限制为2GB。这是我编写的OutpuStream / InputStream实现,用于绕过ByteArrayOutputStream / ByteArrayInputStream的限制(Scala代码,但Java开发人员很容易理解):

import java.io.{IOException, InputStream, OutputStream}

import scala.annotation.tailrec

/** Acts as a replacement for ByteArrayOutputStream
  *
  */
class HugeMemoryOutputStream(capacity: Long) extends OutputStream {
  private val PAGE_SIZE: Int = 1024000
  private val ALLOC_STEP: Int = 1024

  /** Pages array
    *
    */
  private var streamBuffers: Array[Array[Byte]] = Array.empty[Array[Byte]]

  /** Allocated pages count
    *
    */
  private var pageCount: Int = 0

  /** Allocated bytes count
    *
    */
  private var allocatedBytes: Long = 0

  /** Current position in stream
    *
    */
  private var position: Long = 0

  /** Stream length
    *
    */
  private var length: Long = 0

  allocSpaceIfNeeded(capacity)

  /** Gets page count based on given length
    *
    * @param length   Buffer length
    * @return         Page count to hold the specified amount of data
    */
  private def getPageCount(length: Long) = {
    var pageCount = (length / PAGE_SIZE).toInt + 1

    if ((length % PAGE_SIZE) == 0) {
      pageCount -= 1
    }

    pageCount
  }

  /** Extends pages array
    *
    */
  private def extendPages(): Unit = {
    if (streamBuffers.isEmpty) {
      streamBuffers = new Array[Array[Byte]](ALLOC_STEP)
    }
    else {
      val newStreamBuffers = new Array[Array[Byte]](streamBuffers.length + ALLOC_STEP)
      Array.copy(streamBuffers, 0, newStreamBuffers, 0, streamBuffers.length)
      streamBuffers = newStreamBuffers
    }

    pageCount = streamBuffers.length
  }

  /** Ensures buffers are bug enough to hold specified amount of data
    *
    * @param value  Amount of data
    */
  private def allocSpaceIfNeeded(value: Long): Unit = {
    @tailrec
    def allocSpaceIfNeededIter(value: Long): Unit = {
      val currentPageCount = getPageCount(allocatedBytes)
      val neededPageCount = getPageCount(value)

      if (currentPageCount < neededPageCount) {
        if (currentPageCount == pageCount) extendPages()

        streamBuffers(currentPageCount) = new Array[Byte](PAGE_SIZE)
        allocatedBytes = (currentPageCount + 1).toLong * PAGE_SIZE

        allocSpaceIfNeededIter(value)
      }
    }

    if (value < 0) throw new Error("AllocSpaceIfNeeded < 0")
    if (value > 0) {
      allocSpaceIfNeededIter(value)

      length = Math.max(value, length)
      if (position > length) position = length
    }
  }

  /**
    * Writes the specified byte to this output stream. The general
    * contract for <code>write</code> is that one byte is written
    * to the output stream. The byte to be written is the eight
    * low-order bits of the argument <code>b</code>. The 24
    * high-order bits of <code>b</code> are ignored.
    * <p>
    * Subclasses of <code>OutputStream</code> must provide an
    * implementation for this method.
    *
    * @param      b the <code>byte</code>.
    */
  @throws[IOException]
  override def write(b: Int): Unit = {
    val buffer: Array[Byte] = new Array[Byte](1)

    buffer(0) = b.toByte

    write(buffer)
  }

  /**
    * Writes <code>len</code> bytes from the specified byte array
    * starting at offset <code>off</code> to this output stream.
    * The general contract for <code>write(b, off, len)</code> is that
    * some of the bytes in the array <code>b</code> are written to the
    * output stream in order; element <code>b[off]</code> is the first
    * byte written and <code>b[off+len-1]</code> is the last byte written
    * by this operation.
    * <p>
    * The <code>write</code> method of <code>OutputStream</code> calls
    * the write method of one argument on each of the bytes to be
    * written out. Subclasses are encouraged to override this method and
    * provide a more efficient implementation.
    * <p>
    * If <code>b</code> is <code>null</code>, a
    * <code>NullPointerException</code> is thrown.
    * <p>
    * If <code>off</code> is negative, or <code>len</code> is negative, or
    * <code>off+len</code> is greater than the length of the array
    * <code>b</code>, then an <tt>IndexOutOfBoundsException</tt> is thrown.
    *
    * @param      b   the data.
    * @param      off the start offset in the data.
    * @param      len the number of bytes to write.
    */
  @throws[IOException]
  override def write(b: Array[Byte], off: Int, len: Int): Unit = {
    @tailrec
    def writeIter(b: Array[Byte], off: Int, len: Int): Unit = {
      val currentPage: Int = (position / PAGE_SIZE).toInt
      val currentOffset: Int = (position % PAGE_SIZE).toInt

      if (len != 0) {
        val currentLength: Int = Math.min(PAGE_SIZE - currentOffset, len)
        Array.copy(b, off, streamBuffers(currentPage), currentOffset, currentLength)

        position += currentLength

        writeIter(b, off + currentLength, len - currentLength)
      }
    }

    allocSpaceIfNeeded(position + len)
    writeIter(b, off, len)
  }

  /** Gets an InputStream that points to HugeMemoryOutputStream buffer
    *
    * @return InputStream
    */
  def asInputStream(): InputStream = {
    new HugeMemoryInputStream(streamBuffers, length)
  }

  private class HugeMemoryInputStream(streamBuffers: Array[Array[Byte]], val length: Long) extends InputStream {
    /** Current position in stream
      *
      */
    private var position: Long = 0

    /**
      * Reads the next byte of data from the input stream. The value byte is
      * returned as an <code>int</code> in the range <code>0</code> to
      * <code>255</code>. If no byte is available because the end of the stream
      * has been reached, the value <code>-1</code> is returned. This method
      * blocks until input data is available, the end of the stream is detected,
      * or an exception is thrown.
      *
      * <p> A subclass must provide an implementation of this method.
      *
      * @return the next byte of data, or <code>-1</code> if the end of the
      *         stream is reached.
      */
    @throws[IOException]
    def read: Int = {
      val buffer: Array[Byte] = new Array[Byte](1)

      if (read(buffer) == 0) throw new Error("End of stream")
      else buffer(0)
    }

    /**
      * Reads up to <code>len</code> bytes of data from the input stream into
      * an array of bytes.  An attempt is made to read as many as
      * <code>len</code> bytes, but a smaller number may be read.
      * The number of bytes actually read is returned as an integer.
      *
      * <p> This method blocks until input data is available, end of file is
      * detected, or an exception is thrown.
      *
      * <p> If <code>len</code> is zero, then no bytes are read and
      * <code>0</code> is returned; otherwise, there is an attempt to read at
      * least one byte. If no byte is available because the stream is at end of
      * file, the value <code>-1</code> is returned; otherwise, at least one
      * byte is read and stored into <code>b</code>.
      *
      * <p> The first byte read is stored into element <code>b[off]</code>, the
      * next one into <code>b[off+1]</code>, and so on. The number of bytes read
      * is, at most, equal to <code>len</code>. Let <i>k</i> be the number of
      * bytes actually read; these bytes will be stored in elements
      * <code>b[off]</code> through <code>b[off+</code><i>k</i><code>-1]</code>,
      * leaving elements <code>b[off+</code><i>k</i><code>]</code> through
      * <code>b[off+len-1]</code> unaffected.
      *
      * <p> In every case, elements <code>b[0]</code> through
      * <code>b[off]</code> and elements <code>b[off+len]</code> through
      * <code>b[b.length-1]</code> are unaffected.
      *
      * <p> The <code>read(b,</code> <code>off,</code> <code>len)</code> method
      * for class <code>InputStream</code> simply calls the method
      * <code>read()</code> repeatedly. If the first such call results in an
      * <code>IOException</code>, that exception is returned from the call to
      * the <code>read(b,</code> <code>off,</code> <code>len)</code> method.  If
      * any subsequent call to <code>read()</code> results in a
      * <code>IOException</code>, the exception is caught and treated as if it
      * were end of file; the bytes read up to that point are stored into
      * <code>b</code> and the number of bytes read before the exception
      * occurred is returned. The default implementation of this method blocks
      * until the requested amount of input data <code>len</code> has been read,
      * end of file is detected, or an exception is thrown. Subclasses are encouraged
      * to provide a more efficient implementation of this method.
      *
      * @param      b   the buffer into which the data is read.
      * @param      off the start offset in array <code>b</code>
      *                 at which the data is written.
      * @param      len the maximum number of bytes to read.
      * @return the total number of bytes read into the buffer, or
      *         <code>-1</code> if there is no more data because the end of
      *         the stream has been reached.
      * @see java.io.InputStream#read()
      */
    @throws[IOException]
    override def read(b: Array[Byte], off: Int, len: Int): Int = {
      @tailrec
      def readIter(acc: Int, b: Array[Byte], off: Int, len: Int): Int = {
        val currentPage: Int = (position / PAGE_SIZE).toInt
        val currentOffset: Int = (position % PAGE_SIZE).toInt

        val count: Int = Math.min(len, length - position).toInt

        if (count == 0 || position >= length) acc
        else {
          val currentLength = Math.min(PAGE_SIZE - currentOffset, count)
          Array.copy(streamBuffers(currentPage), currentOffset, b, off, currentLength)

          position += currentLength

          readIter(acc + currentLength, b, off + currentLength, len - currentLength)
        }
      }

      readIter(0, b, off, len)
    }

    /**
      * Skips over and discards <code>n</code> bytes of data from this input
      * stream. The <code>skip</code> method may, for a variety of reasons, end
      * up skipping over some smaller number of bytes, possibly <code>0</code>.
      * This may result from any of a number of conditions; reaching end of file
      * before <code>n</code> bytes have been skipped is only one possibility.
      * The actual number of bytes skipped is returned. If <code>n</code> is
      * negative, the <code>skip</code> method for class <code>InputStream</code> always
      * returns 0, and no bytes are skipped. Subclasses may handle the negative
      * value differently.
      *
      * The <code>skip</code> method of this class creates a
      * byte array and then repeatedly reads into it until <code>n</code> bytes
      * have been read or the end of the stream has been reached. Subclasses are
      * encouraged to provide a more efficient implementation of this method.
      * For instance, the implementation may depend on the ability to seek.
      *
      * @param      n the number of bytes to be skipped.
      * @return the actual number of bytes skipped.
      */
    @throws[IOException]
    override def skip(n: Long): Long = {
      if (n < 0) 0
      else {
        position = Math.min(position + n, length)
        length - position
      }
    }
  }
}

易于使用,无缓冲区重复,无2GB内存限制

val out: HugeMemoryOutputStream = new HugeMemoryOutputStream(initialCapacity /*may be 0*/)

out.write(...)
...

val in1: InputStream = out.asInputStream()

in1.read(...)
...

val in2: InputStream = out.asInputStream()

in2.read(...)
...

-1

如果要从InputStream生成OutputStream,则存在一个基本问题。写入OutputStream的方法将阻塞,直到完成。因此,写入方法完成后即可得到结果。这有2个后果:

  1. 如果仅使用一个线程,则需要等到所有内容都写完之后(因此需要将流的数据存储在内存或磁盘中)。
  2. 如果要在完成数据之前访问数据,则需要第二个线程。

变体1可以使用字节数组或字段来实现。变体1可以使用pipies(直接或额外抽象-例如RingBuffer或其他评论的google lib)来实现。

确实,使用标准Java没有其他方法可以解决该问题。每个解决方案都是其中之一的实现。

有一个概念称为“继续”(有关详细信息,请参阅维基百科)。在这种情况下,基本上这意味着:

  • 有一个特殊的输出流,它需要一定数量的数据
  • 如果达到了数量,则流将控制权交给它的对应对象,这是一个特殊的输入流
  • 输入流使数据量可用,直到被读取为止,之后,它将控制权传递回输出流

尽管某些语言内置了此概念,但对于Java,您需要一些“魔术”。例如,来自apache的“ commons-javaflow”实现了Java。缺点是,这需要在构建时进行一些特殊的字节码修改。因此,将所有内容放入带有自定义构建脚本的额外库中是有意义的。


-1

旧帖子,但可能会帮助其他人,请使用以下方式:

OutputStream out = new ByteArrayOutputStream();
...
out.write();
...
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(out.toString().getBytes()));

1
到字符串->大小问题
user1594895

此外,调用toString().getBytes()流*不会返回流的内容。
Maarten Bodewes,

-1

尽管您不能将OutputStream转换为InputStream,但是Java提供了一种使用PipedOutputStream和PipedInputStream的方式,您可以将数据写入PipedOutputStream以便通过关联的PipedInputStream变得可用。
有时候我在处理第三方库时遇到了类似的情况,这些库要求将InputStream实例而不是OutputStream实例传递给它们。
我解决此问题的方法是使用PipedInputStream和PipedOutputStream。
顺便说一下,它们使用起来很棘手,您必须使用多线程来实现所需的功能。我最近在github上发布了一个可以使用的实现。
这是链接。您可以浏览Wiki以了解如何使用它。

By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.