So, I need to loop over some 90-100k records in rails and create an entry in another other table that these records affect. Currently, I am processing these records serially, which does not seem to scale up. MySQL is repeatedly timing out. So I am thinking of following approach:
<ol>
<li>Spawn X sidekiq workers to process Y records each</li>
<li>Each of these Y records is executed in batches(say Y/20)</li>
</ol>
Does this approach seems good enough or is there a better way?
Also, I am thinking to build a dashboard for recording the progress(and failure) for the above processing. How do I go about it?

So, I need to loop over some 90-100k records in rails and create an entry in another other table that these records affect. Currently, I am processing these records serially, which does not seem to scale up. MySQL is repeatedly timing out. So I am thinking of following approach:

1. Spawn X sidekiq workers to process Y records each
2. Each of these Y records is executed in batches(say Y/20)

Does this approach seems good enough or is there a better way?

Also, I am thinking to build a dashboard for recording the progress(and failure) for the above processing. How do I go about it?

Best way to loop over large number of records in Rails?

Product

Explore

Company

Blogs

Support