如何从Node.js中的S3 getObject获得响应?


88

在一个Node.js项目中,我试图从S3取回数据。

当我使用时getSignedURL,一切正常:

aws.getSignedUrl('getObject', params, function(err, url){
    console.log(url); 
}); 

我的参数是:

var params = {
              Bucket: "test-aws-imagery", 
              Key: "TILES/Level4/A3_B3_C2/A5_B67_C59_Tiles.par"

如果将URL输出带到控制台并将其粘贴到Web浏览器中,它将下载所需的文件。

但是,如果我尝试使用,则会出现getObject各种奇怪的行为。我相信我只是使用不正确。这是我尝试过的:

aws.getObject(params, function(err, data){
    console.log(data); 
    console.log(err); 
}); 

输出:

{ 
  AcceptRanges: 'bytes',
  LastModified: 'Wed, 06 Apr 2016 20:04:02 GMT',
  ContentLength: '1602862',
  ETag: '9826l1e5725fbd52l88ge3f5v0c123a4"',
  ContentType: 'application/octet-stream',
  Metadata: {},
  Body: <Buffer 01 00 00 00  ... > }

  null

因此看来这工作正常。但是,当我在console.logs之一上设置断点时,我的IDE(NetBeans)会引发错误并拒绝显示数据值。尽管这可能只是IDE,但我还是决定尝试其他使用方式getObject

aws.getObject(params).on('httpData', function(chunk){
    console.log(chunk); 
}).on('httpDone', function(data){
    console.log(data); 
});

这不会输出任何内容。放置一个断点表明代码永远不会到达console.logs中的任何一个。我也尝试过:

aws.getObject(params).on('success', function(data){
    console.log(data); 
});

但是,这也不会输出任何内容,并且放置断点表明console.log永远不会到达。

我究竟做错了什么?


您的aws对象实际上是该aws.S3对象的新实例吗?另外,响应getObject()是否被传递回http响应,还是通过管道传递到文件?
peteb '16

@peteb aws = new AWS.S3()。响应不应通过管道传递到文件。我需要在Javascript功能来使用它
萨拉·蒂贝茨

那么可以安全地假设内容为JSON或XML吗?
peteb '16

@peteb都不是,它们都是自定义文件格式
Sara Tibbetts

显示您在getObject()通话中使用的参数。如果您试图将签名的URL传递给getObject,我认为那将行不通。
Mark B

Answers:


172

当执行getObject()从S3 API,按照文档文件的内容都位于Body属性,您可以从您的样品输出看到。您应该具有类似于以下内容的代码

const aws = require('aws-sdk');
const s3 = new aws.S3(); // Pass in opts to S3 if necessary

var getParams = {
    Bucket: 'abc', // your bucket name,
    Key: 'abc.txt' // path to the object you're looking for
}

s3.getObject(getParams, function(err, data) {
    // Handle any error and exit
    if (err)
        return err;

  // No error happened
  // Convert Body from a Buffer to a String

  let objectData = data.Body.toString('utf-8'); // Use the encoding necessary
});

您可能不需要从data.Body对象创建新的缓冲区,但是如果需要,可以使用上面的示例来实现。


因此返回的数据似乎确实是一个Buffer对象,我并不熟悉。从理论上讲,我可以new Buffer(data.Body).toString('utf-8');用来获取内容吗?
Sara Tibbetts's

4
如果内容已经是一个缓冲区,则无需从中创建一个新的缓冲区。只是做data.Body.toString('utf-8');。缓冲区是节点中二进制数据的表示,如果您需要更多信息,请
参见

4
这适用于文本,但是是否存在用于处理文本文件以及.png,.jpg等的通用解决方案?
卡特

4
@carter这是一个常规解决方案。只要改变.toString('utf8')访问时data.Body.toString('binary'),如果你想为图像的二进制字符串。如果像这样的问题不需要将Bufferindata.Body转换为String,则可以直接返回data.BodyBuffer直接使用in 。
peteb '18

1
如果AWS文档使这一点更加清楚,那么“将主体从缓冲区转换为字符串” ...将非常有用。我对AWS感到非常厌倦。
osullic

29

基于@peteb的答案,但使用PromisesAsync/Await

const AWS = require('aws-sdk');

const s3 = new AWS.S3();

async function getObject (bucket, objectKey) {
  try {
    const params = {
      Bucket: bucket,
      Key: objectKey 
    }

    const data = await s3.getObject(params).promise();

    return data.Body.toString('utf-8');
  } catch (e) {
    throw new Error(`Could not retrieve file from S3: ${e.message}`)
  }
}

// To retrieve you need to use `await getObject()` or `getObject().then()`
getObject('my-bucket', 'path/to/the/object.txt').then(...);

5
getObject()末尾的.promise()是我的关键。我有时发现AWS开发工具包有点不直观。
安德鲁·哈里斯

我的回答是说“承诺{<pending>}”
jonask

1
@jonaskgetObject()是一个异步函数,您是否尝试用调用它await getObject(...)
阿里安·阿科斯塔

5

对于寻找NEST JS TYPESCRIPT上述版本的人:

    /**
     * to fetch a signed URL of a file
     * @param key key of the file to be fetched
     * @param bucket name of the bucket containing the file
     */
    public getFileUrl(key: string, bucket?: string): Promise<string> {
        var scopeBucket: string = bucket ? bucket : this.defaultBucket;
        var params: any = {
            Bucket: scopeBucket,
            Key: key,
            Expires: signatureTimeout  // const value: 30
        };
        return this.account.getSignedUrlPromise(getSignedUrlObject, params);
    }

    /**
     * to get the downloadable file buffer of the file
     * @param key key of the file to be fetched
     * @param bucket name of the bucket containing the file
     */
    public async getFileBuffer(key: string, bucket?: string): Promise<Buffer> {
        var scopeBucket: string = bucket ? bucket : this.defaultBucket;
        var params: GetObjectRequest = {
            Bucket: scopeBucket,
            Key: key
        };
        var fileObject: GetObjectOutput = await this.account.getObject(params).promise();
        return Buffer.from(fileObject.Body.toString());
    }

    /**
     * to upload a file stream onto AWS S3
     * @param stream file buffer to be uploaded
     * @param key key of the file to be uploaded
     * @param bucket name of the bucket 
     */
    public async saveFile(file: Buffer, key: string, bucket?: string): Promise<any> {
        var scopeBucket: string = bucket ? bucket : this.defaultBucket;
        var params: any = {
            Body: file,
            Bucket: scopeBucket,
            Key: key,
            ACL: 'private'
        };
        var uploaded: any = await this.account.upload(params).promise();
        if (uploaded && uploaded.Location && uploaded.Bucket === scopeBucket && uploaded.Key === key)
            return uploaded;
        else {
            throw new HttpException("Error occurred while uploading a file stream", HttpStatus.BAD_REQUEST);
        }
    }

4

或者,您可以使用minio-js客户端库 get-object.js

var Minio = require('minio')

var s3Client = new Minio({
  endPoint: 's3.amazonaws.com',
  accessKey: 'YOUR-ACCESSKEYID',
  secretKey: 'YOUR-SECRETACCESSKEY'
})

var size = 0
// Get a full object.
s3Client.getObject('my-bucketname', 'my-objectname', function(e, dataStream) {
  if (e) {
    return console.log(e)
  }
  dataStream.on('data', function(chunk) {
    size += chunk.length
  })
  dataStream.on('end', function() {
    console.log("End. Total size = " + size)
  })
  dataStream.on('error', function(e) {
    console.log(e)
  })
})

免责声明:我为Minio工作,它是开源的,与golang编写的S3兼容对象存储,并带有JavaPythonJsgolang中可用的客户端库。


尝试了mino,但是当我打印dataStream.Body其给出的“未定义”时,如何获取缓冲区数据。即console.log('datastream',dataStream.Body); // undefined
Dibish

3

乍一看,您似乎并没有做错任何事情,但是您并未显示所有代码。当我第一次检出S3和Node时,以下内容对我有用:

var AWS = require('aws-sdk');

if (typeof process.env.API_KEY == 'undefined') {
    var config = require('./config.json');
    for (var key in config) {
        if (config.hasOwnProperty(key)) process.env[key] = config[key];
    }
}

var s3 = new AWS.S3({accessKeyId: process.env.AWS_ID, secretAccessKey:process.env.AWS_KEY});
var objectPath = process.env.AWS_S3_FOLDER +'/test.xml';
s3.putObject({
    Bucket: process.env.AWS_S3_BUCKET, 
    Key: objectPath,
    Body: "<rss><data>hello Fred</data></rss>",
    ACL:'public-read'
}, function(err, data){
    if (err) console.log(err, err.stack); // an error occurred
    else {
        console.log(data);           // successful response
        s3.getObject({
            Bucket: process.env.AWS_S3_BUCKET, 
            Key: objectPath
        }, function(err, data){
            console.log(data.Body.toString());
        });
    }
});
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.