{"id":13507,"date":"2021-09-01T07:47:31","date_gmt":"2021-09-01T07:47:31","guid":{"rendered":"https:\/\/engineering.lusha.com\/?p=13507"},"modified":"2021-10-28T06:56:14","modified_gmt":"2021-10-28T06:56:14","slug":"upload-csv-from-large-data-table-to-s3-using-nodejs-stream","status":"publish","type":"post","link":"https:\/\/engineering.lusha.com\/blog\/upload-csv-from-large-data-table-to-s3-using-nodejs-stream\/","title":{"rendered":"Uploading CSV from Large Data Tables to S3 Using Node.js Stream"},"content":{"rendered":"

Lusha<\/a> is a Hypergrowth company having grown exponentially in the past year in number of employees, customers, scale, traffic, and more.<\/p>\n

In the engineering department, we often face challenges dealing with the faster paced growth in scale, traffic, and data that is growing all the time. One such example occurred one major feature that uploads large CSV files to Amazon S3, stopped working because of performance and memory issues we encounter.<\/p>\n

In this blog post, I will describe the challenge we had with this feature that directly impacted Lusha’s customers.<\/p>\n

The Problem<\/h2>\n

In Lusha dashboard UI<\/a>, customers can add contacts to their personal saved list. These lists can potentially contain a large number of contacts. In the UI of Lusha, there is a button to export a CSV containing all contacts.<\/p>\n

\"\"<\/p>\n

When the customer presses Export to CSV he is asked to type his email address and then receives an email with a download link to his CSV file containing all his contact information.<\/p>\n

Behind the scenes, we send RabbitMQ<\/a> message containing all the UI filters the customer selected. On the backend, this process is hooked up to a server that receives the message, pulls the data from our database (we use Cassandra as our main contact database), generates the CSV file to disk, uploads the file to Amazon S3, and sends an email with a download link to the customer.<\/p>\n

Here’s an example of a code snippet on the backend server:<\/p>\n

Pay attention to line 17<\/strong>, where we add all results from the Cassandra query to a global variable named contacts<\/strong>.<\/p>\n