When working with large canvas sizes, you want to adjust your undo buffer preferences appropriately.
They are in the general tab in the main studio artist preferences dialog.
You want to set your Undo Store preference to Disk.
And i would recommend reducing the number of Undo buffers. Since each one takes up memory, so they become bigger memory hogs as your canvas size increases.
Studio Artist is a 32 bit application.
So it is restricted by the operating system to a maximum of 4GB of virtual memory, end of story.
The fact that your computer has more than 4GB of physical ram memory doesn't make any difference, this is a limitation the operating system places on individual running application processes, and it has to do with the amount of memory that can be physically addressed in a 32 bit application.
Many people run multiple applications at the same time, and having that additional ram memory might be a big help for that, and for allowing the operating system to use it directly as opposed to spooling chunks of virtual memory back and forth from the hard disk to ram memory on your computer.
If Studio Artist asks for a chunk of memory to be allocated and the operating system says no, then you get that error message. Which sounds like it happened when studio artist tried to ask for memory to create an undo buffer.
Studio Artist is written differently than a program like Photoshop for example, which only works with small chunks of an image at any given time in memory. Studio Artist was written in a way where the complete image has to be in memory to be processed. This is less memory efficient when working with large image sizes than the other approach, but the trade off is that we can run special and unique image processing algorithms that you could not even use if you were only working with very small chunks of an image at any given time. So it was a design decision tradeoff, access to more interesting image processing , at the tradeoff of lower performance when working with really large image sizes.
Of course the definition of large image size has increased over the years, and what was considered very large when studio artist was first conceived might not be considered so big these days.
So the maximum canvas size you can work with for painting without running into problems with running out of memory is going to be around 10,000x10,000 pixels.
Some Studio Artist effects, like the vectorizer or some image operation effects, might be using higher precision image buffers internal to the effect. So while the canvas buffers are 8 bits per channel, the actual processing effect might be using 16 bits per channel or even higher, depending on the specific effect. So that could also constrain maximum canvas size depending on what kind of effect you are working with in Studio Artist.
Studio Artist does have vector file output capabilities. So, if you are working with something like the vectorizer, or vector paint effects, then an alternative approach to working is to use those vector effects, and record what you are doing into a Paint Action Sequence. You can then run one of the Action menu commands that output a PASeq to either an EPS or SVG vector file. You could then import something like an eps vector file into Photoshop, and rasterize it to any pixel resolution you want for printing.
We have plans to eventually move studio artist to a 64 bit application build This will resolve any memory limitations, since the operating system allows 64 bit apps to allocate a much larger amount of virtual memory. The issue is that we are very heavily dependent on the quicktime api right now, and using the quicktime api forces us to build the app 32 bits. Apple unfortunately is never going to provide a 64 bit quicktime api (even though they did promise developers they would at one point in time and we based our plans at that time on that promise), so we are having to be very clever and consider alternatives in our development plans to get around this issue.
We may do a special 64 bit build of studioartist V5 with some reduced movie functionality for 2D artists.
And studio artist V6 will be 64 bit from the ground up from the very beginning, but that's a much longer term project.
This is an issue we are aware of, and trying to address in the future.
Early versions of StudioArtist 4 did have a memory leak bug associated with undo of manual painting, which quickly became a huge memory consumer when doing manual painting on large canvas sizes. This was fixed in one of our updates, so if you are still using an earlier V4 build, let me know and i'll get you the latest 4.06 update.
Many Studio Artist users complete quite successful large format printing projects based on Studio Artist work.
So you are certainly able to use Studio Artist to create imagery for large format printing.
Here's an article about one of our users and his very successful large format printing projects, all based on Studio Artist work.