Best way to strore a huge number of images
-
100k at once?????
Can't this be done OFF hours. How is the app generating 100K images at once?
-
There is an application generating the files for mass printing out of our billing system once a week. Actually it is not 100K but 500K at once per week! While those printing files are created a copy of those invoices are also generated in PDF format.
I have to store those PDFs somewhere and retrieve them from SAP in an ad hoc bases (not more then 1000 consultation per day, so the retrieval performance is not an issue). Because the files are already there the easiest would be to store those files somewhere on the filesystem and register the path into the database. But I understand the integrity issue even if I have to keep the invoice images for 10 years!
Bye
Gabor -
500K, how long does it take to run that???
I agree with the integrity... but I wouldn't want to try to insert all those records in a batch!!!
-
You know, Rémi, this is exactly my issue why I was raising this topic!
This is a dilemma for me, because storing the data in a filesystem takes almost no ressource (beside disk space). As of today I have not tested how long it does take to store the images into the database. Basicaly the reason of my question is: How to store those file the most efficeint way into the database
Bye
Gabor -
From the top of my head : Insert OFF HOURS in small batches of 500/1000 reports at the time... Should be done before the next day
. If you have a 24/7 shop, maybe you can have a dedicated server for this task (one that would create the reports). Then it could perform the inserts off hours. -
I have been watching this thread with great interest because of my application. In my case I have been working on this application on and off for the past four years. When I started my current position I was told that a Document Archive was desired but that our main product would be my primary resposbility so I would spend a month on it and then be pulled off for three or four (at one point it sat in wait for a year.) Anyhow, I investigated using BLOBs four years ago and decided that it would add a level of complexity that didn't seem to have a large enough payoff to warrent. This brings me to my question:
If integrety is the issue why not implement a service that uses SDK calls to watch for change events in the directories where your files are. Then log these changes or even put the files back? Just index the filepath field on the DB and lookup where it belongs if it is moved. This is a thought that I have been toying with for a while but in my case I am the only person who Develops/Tests/implements/administers the system so I am very careful about those directories.
Tony
-
Yes, but you can always grant access to the application only to view and insert new file/directory, not to delete them. That way you minimize the risk of errors.
-
I agree that limiting the application rights will aid in the integrety of the system but I think the greater risk is an admin mistakenly dragging a file/folder into another folder without noticing. I say admin because no regular user should have access to this file system.
-
Would be nice if we could lock out the admins sometimes
. -
For sure, if only to protect the system from my own stupidity....
-
Of anyone's stupidity actually...
-
Is this topic still open? I'm in a similar situation. I don't need to post a large number of files at once, but I have a large number of files that will eventually be stored. I'd like to do it in a database because these files are associated with records in a database. Right now the records are in the db and the files are in one big folder. The problem I'm having is that accessing the files in the folder from the web or from an application takes FOREVER! Also I have a problem with the record being created, but the file not being completely uploaded so there is a broken link. I think storing the files in the db along with the record would eliminate my problems.
Am I right? If not, how can I make my situation better and more reliable?
-
I think you bring valid points... If you are unable to solve the other problems then this would be a viable alternative.
But since this post is very old, I'd suggest you make a new thread so that any member may give their 2 cents and not just the 4-5 members who subscribed to this thread.
Viewing 13 posts - 16 through 27 (of 27 total)
You must be logged in to reply to this topic. Login to reply