Today, while working on a project at Insopha, we were parsing some HTML code using BeautifulSoup for screen scraping purposes when we faced a maximum recursion depth exceeded runtime error. At first we thought it was caused by some bug in the BeautifulSoup library, but then we realized that the Python interpreter was detecting a recursion problem because of the sequence of callbacks called in the BeautifulSoup SGML parser triggered by the HTML structure which contained a lot of nested <span> tags. But it wasn't really an infinite recursion (just too many recursions) so it wasn't actually a BeautifulSoup bug.
Python comes with a default value of 1000 for maximum recursion limit but it can be changed through sys.setrecursionlimit().
So the question is: how much is safe to raise that value?. Well, Python also comes with a script to test that. Running that script (in Python 2.5) on several OS and architectures we got the following results:
- Linux / 64 bits: 7400
- Linux / 32 bits: 5400
- OS X / ppc7450: 4800
- Windows XP: 2000