How to create pdf from huge dataset of mongodb data about 10million data rows

Working with Data
, ,
1

I want to create pdf from huge dataset of mongodb. (about 10million rows) No specific format of data. you can assume employee database Iam using MEAN stack

Approches tried:

  1. Use nodejs lib. like pdfkit to convert mongodb result (arr of obj) to pdf by doing a for loop in result. (it causes heap out of storage issue + very slow)
  2. Create temporary collection → do a mongoexport to csv → csv to html using awk → html to pdf using wkhtmltopdf tool. (this still is very slow)

After i do mongo query → i cannot store this data in some variable because it will cause heap out of storage issue, so i cannot do any further processing on this data. I can query using limit and skip to get data in chunks and create html and then pdf from it. but it seems very slow process.

Possible approach i think could be to create small pdfs and then merge them together, or by using streams.

What is the most efficient way to create pdf from huge datasets in mongo. for csv mongoexport works great, is there some standard way for pdfs?

2

I would try to export into .csv.

Then load .csv into LibreOffice or Excel.

Print as .pdf

3

Hi @steevej thanks for answering, can we do this programmatically also via node/python/bash?

4

The only way I am aware is via

https://poi.apache.org/

and

1 Like
5

Hi all, I tried libreoffice soffice tool , it worked great, i was able to create pdf for 5Mil data rows. I havent tested for more than that, i think it will work.

1 Like
6

we can install a headless version for libreoffice for usage via a language