Home

Cardbox Talk

 

CardboxForumsCardbox Talk > "Loading Non Image files"

Loading Non Image files

Current user: [none]
Register / Log In · Help

Posted By Post

Juan

13-Sep-2009 11:25

On the 4th August I created a new file with an image field. I have been using it to store MPG4 and AVI files without problem. To do so I used the NON-IMAGE load facility. During the last few days however during this loading process files seemed to load correctly, although did take a long time. However when the file seems to have been saved and an image icon should have appeared, it doesn´t. Why not?

bert

13-Sep-2009 16:49

... does "view file" work?
... only the icon does not appear in Cardbox?
... does Win Explorer show the right icon?
... does same file outside Cardbox launch when you double click on it in Win Explorer?
regards
bert

Juan

13-Sep-2009 19:57

View file doesn´t work on the saved record. I can´t get that far because the image file is not actually saved to the record. It goes through the process of saving but when it finally does stop saving the image file icon is not there.
And yes I can view the file outside Cardbox.
This is just on a new, relatively small still, database I have created not on an existing database. No problem saving images on these other existing databases.
An interesting thing however and I do not know if it has anything to do with it, but I have begun to get a "Com Surrogate failure" message.

bert

13-Sep-2009 21:03

Although all this should not happen, I think a rebuild of your database perhaps could solve your problem.
First try a 'normal' database rebuild. If this give no result, try an extreme rebuild. Already tried?

Regards
Bert

Juan

15-Sep-2009 20:27

Thanks Bert I have tried the rebuild twice, not out of choice. While attempting to load the image file, the software left me without the add a record options so I had to rebuild. It made no difference to being able to add the images. I shall try and recreate the image files and see if the problem persists.

Charles Welling

17-Sep-2009 05:05

Juan, I just had a similar experience when I used a macro to read a file. Choosing more than one file at a time, even small ones, did not load the files.

But, I think you should end this discussion. Forums are NOT meant to discuss probable bugs. You should report this to Cardboss, as I will.

Charles

bert

17-Sep-2009 06:13

And I think it is good to communicate about such things, especially on this forum. It is not always clear that such problems are caused by a bug.
Last night I lost connection between the Cardbox Server and Client reading a non-image file (error message). Now I know others experience also problems loading non image files, I will not hesitate to write Cardboss, knowing this is not only a problem in my database.
Regards
Bert

Charles Welling

17-Sep-2009 13:12

I withdraw my previous posting. The error was in a macro that was used to pick the files.

And Juan: what filesystem does your disk use? If it's FAT32 the maximum filesize is 4 GB. If you've been loading a lot of music and video, your database may well have reached this maximum.

If you right-click a disk in Windows Explorer or My Computer, you will what filesystem is used. If it reports the filesystem as NTFS, a database larger than 4 GB is no problem at all.

Juan

22-Sep-2009 15:20

I haven´t been back to the Forum for a while. I came back today because I have discovered the reason behind the non loading of the non-image files. I have by chance now while catching up read Charles Welling post and he is upsolutely right. That is indeed the reason. The files I was loading on to Cardbox were too large. When I converted them from AVI to MPGEG4, which are much smaller the problem dissappeared. Thank you all for your help. Cardbox is just a brilliant piece of software, so are the people on this forum. What would I do without it.

Charles Welling

22-Sep-2009 18:54

I'm glad to hear that was indeed the reason. But that does not solve your problem. One day your database, or somebody else's, will reach 4 GB once more and the whole thing will again grind to a halt.

The real solution, instead of converting your files, is converting your disk from FAT32 to NTFS, or move your database to a disk already formatted as NTFS.

If your FAT32 disk only contains your own data and not Windows, it is quite a simple operation which involves making a decent backup, formatting your disk and restoring your files.

Juan

25-Sep-2009 15:05

I am not quite clear on the last post. I am confident you meant 4GB as the size of the non image file. Not the database. My database is much larger than that.
Regarding FAT32 and NTFS, it took me quite a while to learn the significance of the difference. So now whenever I buy new hard drives (external) if they are not already I re-format them to NTFS before using them.

Charles Welling

25-Sep-2009 18:48

Are you absolutely sure that the volume which contains your database is FAT32? I've never checked, but Microsoft say that the maximum file size is 4 GB, which should mean that your database could not be larger than that. Perhaps your file system is NTFS after all?

On the other hand: the size of your files could indeed be a problem. I'm sure Cardboss knows all the details, but if you have a multi-gigabyte file which is meant to be played byte by byte and now you process it as a whole, then it may well be that your computer simply runs out of memory.

Juan

14-Oct-2009 20:26

Sorry Charles , I have been away for a few days. The file is on NTFS. It is actually close to 100 GB and the PC is 2/3 full which is why I am considering soon if it becomes much bigger storing it on a new external hard drive. I am backing the file everytime I have to edit or add records to that hard drive already. A process which takes about two hours every time. I wish there was a way of just updating the copy instead of copying it as a whole every time. That would probably be too expensive for me to consider.

bert

14-Oct-2009 22:14

Mark your added / edited records in a special field. Use validator "default entry" for automatic marking of records.
A new record or an edited record is marked then in that special field. Select the marked records and you can backup only the new and/or edited records to a dmp file.
After writing the dmp, “clean up” those records with a macro (validation does not work when you use a macro for editing records).
Alternative is temporary switching off validation in “options” while using batch edit for cleaning up. This way saves a lot of time.

When a restore is needed, however, this way of working takes a lot of time. This because Cardbox does not owe special features reading dmp (or other) files.

It could be a great feature if you could read records, avoiding getting duplicates. A kind of build-in deduplicate, but then working while reading a dmp (or other) import files.

@Cardboss: an idea?

Btw, there is new build of Cardbox. This build is warning you when your disk is full...

Regards
Bert

Charles Welling

15-Oct-2009 05:09

If I'm correct, you have a database that contains many "heavy" files, along with some fields that give a description of those files. And your troubled by the fact that every time you edit the database you have to backup the whole thing.

If this is correct, then your trouble is caused by the fact that you merged static data and dynamic data. Static data being files that do not change after they've been created, and dynamic data being data that does change.

In such cases the better option is to keep that data seperated. Your database with the descriptions should not contain those large files, but only a link to those files, either with "file://" or "http://". Make a seperate backup of your large static files. You will only have to add the new files to that backup, instead of continuously backing up a huge database with files that didn't change.
Your database contains only the descriptions, and will be relatively small.

If, on the other hand, you have a real good reason to have those files inside a database, you could still seperate them. Have one database with descriptions and a unique number. Have a second database with that same unique number and the files. Set up a relational link between those files, using the unique (automatic) number as a key.

You could simply seperate your data by exporting the fields with your descriptions, make a copy of your FMT, and read them back. Then delete the descriptions from the original database.

Then make regular backups of the smaller textual database. If you keep the files that you added to the non-image-file database for, let's say, a month, you could make that backup once a month.

Juan

22-Oct-2009 12:34

Bert´s advice seems the ideal but it is too much for simple me. I much prefer Charles´ methodology so I am off to play. See what happens. Thank you all.

bert

22-Oct-2009 16:05

I like simple things in automation!

Btw: the problem of (many hours spending) backing up your large database is not solved!

Kind Regards
Bert

Charles Welling

22-Oct-2009 22:12

Bert, there are some very basic rules you must adhere to when you set up a database. One of those rules is not to mix incompatible data. When in earlier days librarians set up indexes to make it easier to find books, they did not store the books in the little drawers along with the index cards. Why not? More than one card (author, title) could point to a single book, and books were much larger. So, they didn't mix.

Juan has two kinds of files: text that has to be searched (the index as it were) and large static files that have to be found (the mpeg's, or "the books"). I can think of no reason why you should put the mpeg's into the database. It serves no purpose. When you keep them seperated from your text you don't need to back them up over and over again. Make some copies once, and add only the new files.

And if I read correctly: Juan does not believe that your solution is simple, but he believes that he himself is simple. I don't agree and I admire his perseverance in getting things right.

bert

22-Oct-2009 23:10

Charles,
Linking in a database to files "somewhere" I do not prefer - in most situations.
Databases mostly are used a long time, longer than most network architectures exists. Now the credo is "virtualised". New names for servers, Citrix, Remote Desktop, etc. You need to keep all your paths up to date! I think that is an Achilles' heel. Then: linked files can be deleted (even by accident) without deleting or changing relevant records. Records can be deleted/changed without deleting/changing relevant files. Then there is also the ever existing space to %20 translation in paths. It is a pity, Cardbox/Windows cannot make hot links between a db and a file!
That is why I am very enthusiastic for the new option in Cardbox "loading non-image-files."

Some two years ago I developed Cardbox db as "container” for images (photos). On that moment this was only possible using Base64 translation in Cardbox. You could save then any file into a text field. However, it was not perfect stable. Sometimes the db did some strange things.
A few months ago I changed it all to non-image files. I found out this is really stable (and more now in build 4283).
This file container is a complicated Cardbox application, but it works perfect (30 GB).
I admit, files > 40 MB or so, I think I will not store them into Cardbox. For storing large video files I think I will not use Cardbox.

Indeed, backup time is increased by such an application. That is why I searched for a time saving system. And I found.

B.t.w.: you can also write only the text fields in dmp file...

So you see, two situations, two approaches. Strong that Cardbox can be used in both situations!

Regards
Bert

Mary Doyle (DAF)

3-Nov-2009 16:53

Hi folks,

I have to say I am inclined to come down in favour of Bert’s system on this one, though I vary my decision on whether to include a file in a Cardbox record or link to an external file depending on circumstances. For example, in our Query Tracking database relevant files can be located in a variety of places – documents, email attachments etc. and we find it is much simpler to include all the information in one place. In this case, unlike several cards pointing to one book or file, one record can point to several files. Also we have had experience of servers changing and the problems that causes. We don’t always include the files in a database but our decision is not based on what we did when we were limited by the capabilities of printed catalogue cards. I, personally, am glad to be free of the tedium and limitations of non-electronic reference cards and to be able to take advantage of the flexibility of electronic databases. I am inclined to the view that it is horses for courses and that we should take full advantage of the capabilities electronic databases offer.

Best regards

Mary

Charles Welling

4-Nov-2009 06:51

Yes Mary, you are absolutely right and I also use a mixture of linked files and loaded files.
BUT...
My advice to Juan was just that: an advice to JUAN and not a general recommendation to use links in all circumstances. I suspect that Juan works on a single PC or in a limited network. He has single records and single, extremely large files. In that case loading the files has no advantages and a lot of disadvantages. And that's why I advised him to use links.

Please remember that this is a forum where people seek personal advice. It is not a place to argue on what's supposed to be right or wrong. It may be very useful to exchange views on how to use Cardbox in different circumstances, but in that case it's better to start a new topic.

Mary Doyle (DAF)

4-Nov-2009 10:38

Hello Charles

Re: "Please remember that this is a forum where people seek personal advice. It is not a place to argue on what's supposed to be right or wrong. It may be very useful to exchange views on how to use Cardbox in different circumstances, but in that case it's better to start a new topic."

I was merely responding to your earlier contribution to this present topic : "Bert, there are some very basic rules you MUST [caps are mine] adhere to when you set up a database. One of those rules is not to mix incompatible data. When in earlier days librarians set up indexes to make it easier to find books, they did not store the books in the little drawers along with the index cards. Why not? More than one card (author, title) could point to a single book, and books were much larger. So, they didn't mix."

I felt I had to balance what appeared to me as laying down a "definitive rule" by giving my opinion to readers of this topic (Juan and others who might read it)that there are not always hard and fast rules about was you must and must not do when setting up a database.

Charles Welling

4-Nov-2009 11:57

And again Mary, you are right to counterbalance any of my remarks that may seem to have been cut from granite. And again I'd like to stress that I was responding to Juan's problem. My analogy was based on my perception of Juan's database. In his case the data are more or less incompatible and they should not be mixed.
I wonder what Juan thinks of our discussion :-)

Quick Reply

Please log in or register before trying to post a reply.

 
© 2010 Cardbox Software Limited   Home