Memory leak with large projects ?

[steptoe]

Following on from my post about there being a batch option but its all put into one folder so treated as a single batch project, I may have found a memory leak or memory issue with the video converter

As I'm in the process of ripping and doing an MKV remux to put DVDs/blu-ray episode disks and Disney films onto out mediaplayer, I set up a fairly large batch project that was at least 6-8 episode disks, so the entire filelist is at least 30-40 files

I left the conversion running and after checking it 3-4 hours later found that the current conversion process in the batch claimed it would take hours to convert each file instead of 1-2 minutes or less. The time counter was still rapidly climbing so I assume the conversion had stalled

I checked the task manager and found my free memory to be almost zero. Force closing the video converter task still didn't free up the memory so was forced to restart the computer. I ran another batch with the initial problem files at the start of the batch (again about 30-40 files in the batch). The problem files were converted perfectly in a very short time as I would have expected

Checking again a few hours later, the video converter had again stalled on the conversion telling me it would be hours before it completed the file it was processing, and again according to task manager my free memory was almost zero

Video converter is running with no CUDA/GPU assistance and processing two files at a time with all 4 cores available to the video converter on the lower task setting


I'm running win7 ultimate 64-bit with an Intel i5 2500k running at 4.5ghz (which it manages very happily even under 99% load for hours when stress testing it and has done for about 6 months now with NO blue screens or weird messages), with 16GB memory and pretty much nothing else running at the same time as I wasn't around when the batch was running so the computer was left to run the batch

EDIT :

Restarted the computer again, task manager states I have 12GB memory free and after running the video converter just under that. So its not taking much on initial runs before a project is run


Something that the video converter calls is taking all my memory. I've tried a small batch of 11 episodes from a single DVD rip and once the task completed according to the task manager my memory is almost zero. I've closed the video converter normally without force closing it via the task manager and the task manager says I've now got about 1GB of free memory. I should have around 11-12GB if memory is being freed once the video converter has finished a task or been quit

Any suggestion or pointers what may be going wrong and being a memory hog .....

[steptoe]

I've run a big batch this morning of about 50 files to remux to MKV from 7 full DVD rips of episodes (cartoon really so not very long episodes), increased the minimum memory in the video converter settings to 1024 (1GB) free and it seems to have worked

I still have almost zero memory available once a conversion project has been running for a few minutes which increases very slightly once its finished but not by much, but at least the project now doesn't stall and I am forced to restart the computer to carry on with the batch where it stalled

Also, memory still isn't being released once video converter is quit

[steptoe]

More updates on the issues with large batch projects ....

The project is converting PAL/NTSC video to a slightly smaller filesize, about 35% smaller with the same source resolutions and recoding the audio to 5.1 channel, 64bits per channel


If I use the GPU to process each file, and try to do process more than one file at the same time during the conversion, it appears that only one can use the GPU and the other task uses the CPU as one task runs about 400FPS the other task is drastically slower at about 45FPS

This is what I assume is the multiple GPU processes switch is for, but the video converter crashes if I use this which suggests my GPU doesn't support this (Nvidia GTX460)


Also, the first 3 or 4 conversions run about 350-410 FPS using the GPU, but then slow down considerably to about 130FPS the more conversions take place in the batch queue and holds fairly steady at this figure, this I'm guessing is related to the lack of free memory as explained in my previous posts above


Another fact I've found, if you set the switch for the GPU to decode AND encode the video it runs quite a bit slower than if you set the switch so the GPU encodes ONLY and let the CPU do the decoding. Again, I'm assuming this is the GPU can't handle more than one task at a time so flicks between decoding/encoding with each conversion

Setting the conversion task for one file at a time, letting the GPU encode ONLY so the CPU is in control of the decoding then it runs flat out as given above at an average of around 400FPS until my system starts running very low on memory


There also seems to be a minor issue with the minimum memory setting, if I change this to 1024 (1GB) to try and keep the encode running with stalling due to lack of free memory, this setting isn't saved with the profile or main settings so I have to remember to set this back to 1024 minimum memory on restarting the video converter software

[cedric]

Hello steptoe,

Thank you for ALL your feedbacks...
What version do you used for your tests?
I ask this because I fixed a memory leak in 1.1.0.10 version and I need to know if another memory leak is still there.

For your hardware encoder issue, I checked your graphic card spec and it should be able to convert more than one video using CUDA in same time.
Same question as before, what version do you used?
Can you provide a log file with failed conversion (with "Allow multiple instances of hardware optimization" enabled)?

For your hardware decoder issue, the problem is not in GPU multiprocessing (encoding/decoding switch ), but a lot of hardware decoders are limited to be able to decode a video in real time only (~30Fps)
That mean either your graphic card is able to decode at 400 fps, the decoding rate is limited to real time (or a little higher).
conclusion -> You're right, with recent computer hardware decoder should be turned off when hardware encoder should be turned on for best speed.

Regards,

[steptoe]

I'm using the very latest 1.1.1.10

I'll get back to you on seeing if the memory leak is still there, but according to the task manager the video converter is using about 600MB when doing an encode and the free memory gradually drops the more encodes are run in a project

But, free memory is not returned to anywhere near what my system had free before I started the encode project (around 12GB) and quit the video converter. Either by force closing it via the task manager or quitting it in the normal way via the video converter

I'll run a batch of about 10 episode files and get it to create a logfile and see what that comes back with, and also try the multiple GPU option with the same batch to see what the difference is in the logfiles

[steptoe]

Ran a batch of 13 episodes after restarting computer and turning multiple GPU encoding option on and a single encode operation, not a single issue. Batch completed very quickly, returning around 400FPS

Tried again after restarting computer with same batch and running 3 encode tasks with multiple GPU encoding turned on again, batch completed perfectly again. Less FPS but then its been split between the three encode operations, so still works out about 400FPS if using GPU encoding BUT leaving the CPU to perform the decoding



What I did notice is that its the cache memory that is filling, leaving almost no free memory, not as I reported that the actual system memory is being used. According to various sites, they are suggesting the computer is running faultlessly as its using all the free memory so has no issues. Free memory means unused memory, so it appears the video converter is pushing my system to its limits as much as possible as using the GPU as and when needed for what its needs


But, no matter what I try I can't persuade the system to release the cache memory once video converter has completed, and after I quit the converter. I've tried suggested methods on various sites to flush the cache memory, but none appear to work as suggested



Whatever else has changed in v1.1.1.0 it appears to have solved my encoding issues with multiple GPU encodes as that really the only thing that I changed from the last time I tried and the encodes failed before upgrading to v1.1.1.0 from v1.1.0.9

[steptoe]

Further to my other request, I did manage to find an option to allow each file to have its own bitrate set

But its not easy to find unless you click on the video file, then choose the video editor, then click the video track and only then do you get to see the 'quality advisor' and change the bitrate to set a rough filesize of the converted file


I was looking for a global option that allows you to set the each file in the project to a rough output filesize instead of having to change each one manually. But, at least its there and gives the user more control over whats happening

[cedric]

Hello steptoe,

I think I will add a new feature in future version that will fix all your requests.
I will add a new button/function in video editor allowing user to apply current title parameters to all titles in project.
That mean if you set a video bit rate to a title and use this new function, all titles in project will use this bit rate.

User will be asked to select the parameters he wants to apply to all titles:
- video settings (rotation, fit method, color adjustments...)
- video bit rate (your request)
- audio settings (audio compressor, language...)
- subtitle settings (font size/color/style, language, code page...)

Regards,

[cedric]

Are you still with us steptoe?

Regards,