Wondering if I'm ever going to run out of stack depth in this async recursion scenario

programming web dev
1

I have a pattern that I like to use for processing lists of data that I need to asynchronously process/operate on. Like let’s say I have 100k files I need to download and convert from JPG to PNG. So I get a list of the JPG URLs, and operate on them like so:

let bigArray = new Array(20000).fill('hi!')

let processThings = _ => {
  let things = bigArray.concat() // duplicate because maybe other parts of the program need to see the full list
  let results = []
  
  let processOneThing = _ => {
    let thing = things.pop()
    
    setTimeout(_ => {
      results.push(thing.toUpperCase())
      if(things.length){
        processOneThing()
      } else {
        console.log(`Done with ${ results.length } 1ms things.`)
      }
    }, 1)
  }
  
  processOneThing()
}

processThings()

Presumably, this doesn’t turn into a straight-up recursion scenario, because these functions all actually end up returning undefined and just kick something to the event loop.

Can someone confirm my thinking here? I’m pretty sure it makes sense, but I don’t want this to bite me later.

Tagging @senocular and @kirupa because you’ve probably thought about this.

2

As far as I know, you’re fine. The setTimeout takes you out of the stack and puts the callback in the queue which would be the top of the stack when its called later in the event loop. I can’t imagine any implementation doing any differently, as long as you enforce asynchrony. A case where that might be a problem might be if you do some synchronous error handling and catch something before going async and then continue with a recursive (sync) call. If everything errors doing that, then you’re out of luck

1 Like
3

Okay, good, thanks for looking at this. In reality, I’m using something like https.request instead of setTimeout, but it has the same implications for the call stack.

In my actual usage of this pattern, I think my error handling tends to be asynchronous too, so it shouldn’t ruin anything. But you’re right, depending on how it’s done it has the potential to blow things up if everything fails synchronously. (For me that’s fine because it’d just mean some 3rd-party service is down.)

The setTimeout takes you out of the stack and puts the callback in the queue which would be the top of the stack when its called later in the event loop. I can’t imagine any implementation doing any differently, as long as you enforce asynchrony.

In this case, I’m not even worried about the queue ordering, just the idea that if you’re using synchronous recursion, you’d hit a limit at like ~10k before the JavaScript engine killed your call stack because of its recursion limits. But the queue should be fine here, too, because I’m never starting more than one process step at a time.