How to store files in Cassandra with NodeJs?

Let’s learn about how to store files in Cassandra using NodeJs. In Cassandra, a blob data type is capable of storing images, videos or other file formats (pdf, xlsx, etc.)

This POC covers the following:
Connectivity with Cassandra in NodeJs.
Part I : To insert and fetch an image using NodeJs and Cassandra?
Part II : [For large files]To insert an image in chunks, fetch those chunks and display the image.

Find full code on Github : https://github.com/crazyrich/NodeJS-Cassandra

Part I: To insert and fetch an image using NodeJs and Cassandra?

UploadFile is our request handler in which we first read the file using readFileSync with  base64 encoding and then wrap it into a Buffer. Buffer is equivalent to Blob in Cassandra. We execute the query and insert this buffer into a blob column in Cassandra table. Blob will store the content in some hexadecimal values (eg. 0x8678298439..). Create files folder at the root level to keep the images (which we will upload).

export const uploadFile = (req, res, next) => {

    const content = readFileSync(resolve(__dirname + '../../../files/' + 'img-to-upload.png'), 'base64');

    const objBuffer = new Buffer(content, 'base64')

    const query = 'INSERT INTO local_ksp.filestore (object_id,chunk_id,data) VALUES(?,?,?);';
    const params = ["100", "1", objBuffer];
    db.execute(query, params, (err, result) => {
        if (!err) {
            console.log('Uploaded;')
            res.json({ 'status': 'uploaded !' });
        } else {
            console.log(err)
            res.json({ 'status': 'Error uploading file !!' });
        }
    })
}

In GetFile, we hit the Select query to get the blob stored in Cassandra table and write it into a file. After execution, you may check the new image created at the root folder.

export const getFile = (req, res, next) => {
    const query = "SELECT object_id,chunk_id,data FROM local_ksp.filestore";
    db.execute(query, (error, result) => {
        if (!error) {
            if (result.rows.length > 0) {
                result.rows.forEach(file => {
                    writeFile('display-img.png', new Buffer(file.data, "base64"),
                        // writeFile(file.name + file.id + '.xlsx', new Buffer(file.object, "base64"),
                        (err) => {
                            if (!err) {
                                console.log('check restore folder!!')
                            }
                        });
                });
                res.json(result.rows);
            } else {
                res.json({ 'status': 'Result set empty !!' });
            }

        } else {
            console.error(error);
        }
    });
}

This approach is OK when your files are smaller in size(KBs).

Part II: Insert an image in chunks, fetch those chunks and display the image.

For images of varying size (in MBs), it’s advisable to first break the images in chunks and then store them in your table. So using createReadStream, we create a readstream for our file and check for the ‘data’ event. CreateReadStream has attributes Encoding type and HighWaterMark. HighWaterMark is used to define the chunk size. Here we have defined 128 *1024  = 131KB which means a chunk can have a max of 131KB size. As soon as a chunk is read, this event handler wraps it into a Buffer and inserts it into the table. This is done for all the other chunks created.

export const uploadFileChunks = (req, res, next) => {

    const stream = createReadStream(resolve(__dirname + '../../../files/' + 'DotNet-ebook.pdf'), {
        'encoding': 'base64',
        'highWaterMark': 128 * 1024
    });
    let counter = 0;
    stream.on('data', (data) => {
        const objBuffer = new Buffer(data, 'base64');
        const query = 'INSERT INTO local_ksp.filestore (object_id,chunk_id,data) 
        VALUES(?,?,?);';
        const params = ["100", counter++ + "", objBuffer];
        db.execute(query, params, (err, result) => {
            if (!err) {
                console.log('Uploaded chunk;')
            } else {
                console.log(err)
                res.json({ 'status': 'Error uploading file !!' });
            }
        });
    });


    stream.on('end', () => {
        console.log('end')
        res.json({ 'status': 'uploaded !' });
    })
}

In GetFileChunks, we select all the chunks against an object and merge them in order into 1 Buffer and write that Buffer into a file using writeFile (from ‘fs’ module). After execution, you may check that the file has been created at the root level.

export const getObjectChunks = (req, res, next) => {
    const query = "SELECT object_id,chunk_id,data FROM local_ksp.filestore where object_id ='100';";
    let mergeChunks = [];
    db.execute(query, (error, result) => {
        if (!error) {
            if (result.rows.length > 0) {
                result.rows.forEach(file => {
                    mergeChunks.push(file.data);
                });
                let imgBuffer = Buffer.concat(mergeChunks);

                writeFile('DotNet-ebook-chunk.pdf', imgBuffer,
                    (err) => {
                        if (!err) {
                            console.log('check restore folder!!')
                        }
                    });
                res.json(imgBuffer);
            } else {
                res.json({ 'status': 'Result set empty !!' });
            }
        } else {
            console.error(error);
        }
    });
}

This was all about storing files or store images/media in Cassandra with NodeJs.

Hope it was helpful. Thanks!