Hi there, the async insert_many() always froze at last few writes, used motor writing into mongoDB, total 65 tasks, each task 500 ducuments.
result = await coll.insert_many(batch, ordered = False)
if result.inserted_ids:
global box
box +=1
print(f"Box {box} ")
print(f"batch {ID} Completed")
return True
The first 60 tasks are very fast, but then the 61 took 5 minutes, and the program just froze there, some times need to waiting for 2 hours to complete or not complete at all, is that something normal?
async def Gather():
tasks = []
for ID in range(1, 65):
tasks.append(async_scraper(ID))
await asyncio.gather(*tasks)
loop = client.get_io_loop()
loop.run_until_complete(Gather())
The scraper scraping each batch and feed to mongoDB, my internet may has issues, so anyone knows what happened?
Right now it stopped at 61th task, and just froze no progress, total 65 tasks should be, no error throwing for scraping, everything is fine.