Proper way to free a variable?

[methodic]

Say you have a large list that takes up 5 megs of memory. What is the proper way to free up that memory from an already running script. Is it as simple as (set 'lst '()) or is there another function to use to free up memory from variables no longer used?



Thanks.

[Lutz]

Yes, just assign something else to it and the memory will be freed. It depends on the OS when it will come back into the OSs general memory pool. It may stay with the process for some time until the OS needs it, but newLISP frees it the same moment you de-reference it by assigning something else.



If the variable is a local in a user-defined function or a let, it will be nil'ed automatically when the function returns.



Lutz

[methodic]

What do you suggest for user-defined function recursion. Re-assigning any variables that you know take up a bunch of memory before it calls itself again? Seems to me if you don't a script could just tear up memory fast with recursion.



On a side note, do you recommend recursion or another means like (load) or (exec)? What is the least memory and cpu intensive?



Thanks.

[Lutz]

It all depends, there is now easy 'fits all' answer to it. Remember you always can pass big data by reference via a symbol or a context, see  http://newlisp.org/downloads/newlisp_manual.html#pass_big">http://newlisp.org/downloads/newlisp_ma ... l#pass_big">http://newlisp.org/downloads/newlisp_manual.html#pass_big .



Many recursive algorithms use up more time than memory, so before you are running out of memory, the whole things is taking to long anyway.



There is always a way to implement an iterative solution too. In my experience most recursive algorithms occuring in practice don't have either (memory or time) problems and run just fine. If you run into stack limitations you always can start up newlisp with more stack space see: http://newlisp.org/downloads/newlisp_manual.html#options">http://newlisp.org/downloads/newlisp_ma ... ml#options">http://newlisp.org/downloads/newlisp_manual.html#options



I remember having done this only ince, for an Ackerman algorithm benchmark.



Lutz

[methodic]

Interesting, I didn't know about the stack size for newlisp.... I think that might be one of the problems I was running into. I started up my script with a stack size of 10,000 to see if it will overflow again. What do you recommend a stack size/mem usage be for a list that can have roughly 95 million items, all integers. Is there a way to give newlisp an unlimited stack size?

[methodic]

It seems that for every 20,000 elements in the list, newlisp takes up around 300m of memory. Calculating it out, it appears I will need approx. 1.5G of ram to suppoer ~95 million integers in a list.

[cormullion]

It seems that for every 20,000 elements in the list, newlisp takes up around 300m of memory. Calculating it out, it appears I will need approx. 1.5G of ram to suppoer ~95 million integers in a list.

Wow. I can contain my curiosity no longer! What are these 95 million integers that must be stored in a list? ;-)

[methodic]

Working on a project that involves scraping social networking sites, MySpace being one of them and has ~95 mil users, all with their own unique integer.



I think most of the memory being taken up is due to the fact that when I scrape a webpage, I recurse into my function without cleaning up the variables I use to store the html and links and such.



My approach is clever but probably not the most optimal way. Although its more of a project than anything else.... so......



Is it recommended for any variables I use, right before I recurse into my function to assign them all nil values to free up memory? I can't think of any other way they would become free without doing it by hand.



Thanks.

[cormullion]

I wouldn't know how to handle data of that magnitude, but it sounds like an interesting project. Good luck!

[methodic]

A few updates... ever since I changed my newlisp script to not recurse out of the function that calls itself (instead I added another function to call the function in a while loop), my program hasn't crashed once and it uses WAY less memory... probably due to the fact all the data I use is free'd once the function ends, so that is good.



I am on my way to scouring the top 3 sites, and have a decent UI and a search bar. I'm hoping to launch the site in maybe a few weeks to a month... I just need to grab enough data, and move the site onto a fast box/connection and hopefully mysql won't crush in on itself.... I can really see this project just blowing up and having a ton of users going to it, so I want to make sure I have a decent plan if that happens.