PDA

View Full Version : [Discussion] File Extension Types For New Downloads Section



Alex
04-16-2008, 12:46 PM
Hey Everyone,

I was hoping the community could give me a hand with this. I'm adding all of the file extensions that can be uploaded to the new downloads section.

So far I have:

zip rar tar gz pdf doc txt xls odt ods rtf 123 csv dll pps

and for images:
jpg jpeg jpg2 gif png bmp psd 3dm 3dmf


Obviously there are hundreds of file extensions out there and I don't want to add every single one of them, so what are all the file extensions that you use that are not listed above? This includes all of the CAD file extensions too.

thanks:D

Adrenalynn
04-16-2008, 01:06 PM
Just at a first-blush, for CAD/CAM/3D

3ds, asc, max, blend, c4d, dwf, dwg, dxf, iam, idw, sldasm, slddrw, sldprt, igs, ipj, ipt, lwo, lws, ma, mb, mc8, mc9, mcx, par, sab, sat, session, ops, part, pla, pln, prj, pxr, stl, srf, tpl, unv, xsi, xv0, xv2, xv3, z3d

Those are the extensions associated on my CAD/CAM machine right now, but I'm of course certain there are more. ;)

Alex
04-16-2008, 05:29 PM
Awesome! Thanks Adrenalynn:)

We're now ready to test the TRC Downloads Library out. We may run into a few issues, as with any new plugin, but hopefully nothing too major.

So, if anyone has some stuff they're working on that they'd like to try uploading and share:

http://forums.trossenrobotics.com/downloads.php

We'll do an official announcement once we see that all the kinks are worked out.

Let us know what you think! Also, I'm sure we didn't cover all the file extensions, so if there are any that anybody else uses regularly that you do not see yet in this thread, please let me know and I'll add them:D

Droid Works
04-16-2008, 05:31 PM
don't forget exe and rcb

Alex
04-16-2008, 05:35 PM
I don't think it's a good idea to put a straight up .exe extension on there. Sounds like a security risk to me. Feel free to discuss this though because I'm curious what everyone thinks.

LinuxGuy
04-16-2008, 05:58 PM
I was hoping the community could give me a hand with this. I'm adding all of the file extensions that can be uploaded to the new downloads section.

So far I have:

zip rar tar gz pdf doc txt xls odt ods rtf 123 csv dll pps
DLLs are usually a BAD idea to allow for uploads.

8-Dale

LinuxGuy
04-16-2008, 05:59 PM
I don't think it's a good idea to put a straight up .exe extension on there. Sounds like a security risk to me. Feel free to discuss this though because I'm curious what everyone thinks.
Yes, same as for DLLs, BAD idea.

8-Dale

LinuxGuy
04-16-2008, 06:04 PM
Just at a first-blush, for CAD/CAM/3D

3ds, asc, max, blend, c4d, dwf, dwg, dxf, iam, idw, sldasm, slddrw, sldprt, igs, ipj, ipt, lwo, lws, ma, mb, mc8, mc9, mcx, par, sab, sat, session, ops, part, pla, pln, prj, pxr, stl, srf, tpl, unv, xsi, xv0, xv2, xv3, z3d

Those are the extensions associated on my CAD/CAM machine right now, but I'm of course certain there are more. ;)
Not all of those are drawing or model files though, like PRJ (project), for instance and probably SESSION.

I'll add to that .ad_asm (Alibre Design assembly) and .ad_prt (Alibre Design part). Also .STEP (usually STEP 214), .IGES, and .STP (also somtimes STEP format).

8-Dale

DresnerRobotics
04-16-2008, 06:06 PM
Id imagine the majority will be in .zip or .rar anyway right? Or does the forum software actually scan the contents?

LinuxGuy
04-16-2008, 06:50 PM
Id imagine the majority will be in .zip or .rar anyway right? Or does the forum software actually scan the contents?
I dislike having to deal with multiple archive types per platform. ZIP seems to be available pretty much everywhere. I'd like to see us agree on ZIP for archives becuse they can be processed on all platforms that I am aware of. Unless a person has multiple files that need to be kept together, I don't see any reason why any approved extension (smaller list = better and more secure) couldn't be uploaded as single files.

8-Dale

Alex
04-16-2008, 07:22 PM
I dislike having to deal with multiple archive types per platform. ZIP seems to be available pretty much everywhere. I'd like to see us agree on ZIP for archives becuse they can be processed on all platforms that I am aware of.

I always use ZIP format, but I know way too many people who don't, so I'd have to say that it's unlikely we'd get people to agree to this.


Yes, same as for DLLs, BAD idea.

Why would this be a bad idea though? Most companies who would want to upload their library files would be in DLL format.

Alex
04-16-2008, 07:26 PM
does the forum software actually scan the contents

No, it doesn't scan the contents or anything. It just will only allow file extensions that I specify. Unfortunately there isn't a way to tell it to allow all files except ______:(

Allowing only compressed files might be the way to go anyway though to try and cut down on storage. I could see us maxing out our storage limit really quick if we don't compress the files. Thoughts?

Droid Works
04-16-2008, 07:36 PM
I just tried to upload some bs2 files and it wouldn't let me. So .bs2 would be nice...lol

Alex
04-16-2008, 07:46 PM
try it now droid

Alex
04-16-2008, 07:51 PM
ok, so just to clean up this thread a bit so more members can chime in without having to look through what was already mentioned, here is what I have in the allowed extensions:

zip rar tar gz pdf doc txt xls odt ods 3ds asc max blend c4d dwf dwg dxf iam idw sldasm slddrw sldprt igs ipj ipt lwo lws ma mb mc8 mc9 mcx par sab sat session ops part pla pln prj pxr stl srf tpl unv xsi xv0 xv2 xv3 z3d bs2 ad_asm ad_prt STEP IGES STP rcb

Are any of these extensions usually large files though? We have to be a little conservative with our storage;)

Matt
04-16-2008, 08:35 PM
My 2 cents. I think just asking people to post with zip or rar files is one way of keeping it simple. There are sooooooo many extensions out there. Think of your own sanity
Alex :)

Alex
04-16-2008, 08:43 PM
very true. But, there are many file extensions out there that are a total waste of time to compress such as motion files and I'm sure many others that I can't think of off hand. Obviously if it's a "project" like setup whereas a single upload will encompass many files, members will have to zip up. But I'm thinking that small single file uploads should be the types of extensions allowed (aside from compressed files).

LinuxGuy
04-16-2008, 09:06 PM
I always use ZIP format, but I know way too many people who don't, so I'd have to say that it's unlikely we'd get people to agree to this.
What other formats, besides RAR, are also common?


Why would this be a bad idea though? Most companies who would want to upload their library files would be in DLL format.
Does your server run on *NIX or Windows. If Windows, you need to be able to scan archives and .DLL and .EXE files for viri before posting them for the public. There should be a way for you to flag selected extensions for special handling. *NIX is not as prone to viri as Windows is.

8-Dale

Droid Works
04-16-2008, 09:18 PM
Just uploaded the bs2 file, worked fine:)

LinuxGuy
04-16-2008, 09:22 PM
No, it doesn't scan the contents or anything. It just will only allow file extensions that I specify. Unfortunately there isn't a way to tell it to allow all files except ______:(
Bummer. I hope you will look for a way to allow scanning for viri. This is pretty much mandatory for any system allowing a user file base to be built.


Allowing only compressed files might be the way to go anyway though to try and cut down on storage. I could see us maxing out our storage limit really quick if we don't compress the files. Thoughts?
I agree as long as there aren't a gazillion different archive formats. Compression will be our friend. :happy:

8-Dale

Matt
04-16-2008, 10:08 PM
very true. But, there are many file extensions out there that are a total waste of time to compress such as motion files and I'm sure many others that I can't think of off hand. Obviously if it's a "project" like setup whereas a single upload will encompass many files, members will have to zip up. But I'm thinking that small single file uploads should be the types of extensions allowed (aside from compressed files).

You are right, many files won't compress. That's not what I was getting at. I was thinking more about the simplification of have just a few extensions instead of an ever growing list. It keeps the doorway narrow and manageable. It also allows people to download files and scan them or peak at them on their own before unzipping them.

Alex
04-16-2008, 10:19 PM
What other formats, besides RAR, are also common?

RAR is the other major one that I see most people use:)

I can't remember the other one off hand though... tar? tar.gz? something like that


Yeah, it's a bummer that we can't specify what files we can't upload, but meh, whatever.

We are running a Linux based server (Apache) and our host is incredibly cool, so if there's a way to scan files when they're uploaded, they'll know about it and let me know how to set it up.

Adrenalynn
04-17-2008, 02:30 AM
.tar is tape archiver - it doesn't provide any compression per-se, it collects a mess of files into one handy place (archive). If you tarballed a bunch of text files, you could actually open a .tar in your favorite text editor and view the contents of the text files.

.tar.gz is gzip compressed tar
.tgz is also gzip compressed tar

These employ gnuzip - a very mature zip engine.

.bz/.bzip/.bz2 is another common compression in the unices. It's chronologically more mature than gnuzip, and frequently provides higher compression in my experience, but it's not as widely supported.


Under Linux, Viralator (http://viralator.sourceforge.net/) is an open-source proxy engine for virus scanning uploads/downloads. We use Viralator talking to both Bit Defender (http://www.bitdefender.com) and Sophos (http://www.sophos.com/downloads/products/index.cgi/next?GroupsID=5) ('cause one virus scanner is never enough)


Since we're talking about archives - how about .iso, .nrg, ... Disk images?

Adrenalynn
04-17-2008, 02:50 AM
Sorry - I opened the thread to the last page.

Even though motion files don't compress (and I don't see most of the motion extensions in that list, like BVH, right?), they can be stored in an archive.

Most source code coming from a unix-like system will be in .tar, .tar.gz or .tgz not zip. Zip still has issues associated with licensing, and although most distros include it, real unix hacks don't zip. We were tarballing stuff before your happy widdle winders boxes were even a glint in Billies eye.

Alex
04-17-2008, 09:29 AM
haha! Now my sanity is slowly vanishing:)

I'm going to remove the CAD file extensions for now and stick to only compression and motion file extensions.

So, which compression formats are widely used cross platform? I definitely don't want to allow extensions that are only supported in Windows or Linux or MAC, etc.

Does anyone have a list of all the motion file extensions?

Alex
04-17-2008, 09:32 AM
So, this is what I have at present:

zip rar tar gz pdf doc txt xls odt ods rtf 123 csv dll pps bs2


and for images:

jpg jpeg jpg2 gif png psd

Droid Works
04-17-2008, 12:39 PM
Don't forget rcb , they are much to small to zip. P.S. so far I only see files I posted. post some files ppl :)

Adrenalynn
04-17-2008, 01:46 PM
I think you missed tgz?

Winzip, and it's free cousin, 7zip, support tgz/gz/tar/bz/bz2/bzip under Windows. Winrar supports the t*/g* family, but not b* family, as memory serves

I think Winzip, 7zip, and Winrar must account for some massively high percentage of non-built-in decompressors under Windows. So you are at least safe with tar/gz/tgz, and probably something like 75% safe with bz/bz2/bzip.

The mocap files I most commonly use are BVH, BVA (biovision & poser), ASF/AMC (Acclaim), ASK/SDL (Alias), ASC (Ascension), C3D (C3D and Gait Analysis Studio), CSM (Max Character Studio), DAT (Polhemous), GTR/HTR/TRC/MOT/SKL (Acclaim Motion Analysis), HANIM, TVD (I don't remember why those two are in there)

There are several others, and they'll come to me. But if you're scrapping 3D formats, do you really want mocap formats? They can just be stored in an archive container like zip.

DresnerRobotics
04-17-2008, 02:15 PM
Is there any way to limit upload access to users with X amount of posts or something along those lines? Would be an easy way to avoid people randomly joining just to cause trouble, and if you were a legitimate newcomer and wanted to share before you hit X amount of posts, you could simply PM Alex.

Alex
04-17-2008, 02:26 PM
I'm sure I probably missed quite a few files adrenlynn. That's why I'm askin the community;) BTW, yes, we are scrapping 3D files. There are just too many to cover and it's just as easy (and probably save on storage) if members could compress them into an accepted format.

here are all the ones I have now:

zip rar tar gz tgz pdf doc txt xls odt ods rtf 123 csv dll pps bs2 rcb

jpg jpeg jpg2 gif png psd



Is there any way to limit upload access to users with X amount of posts or something along those lines

Good idea! It would make sense to have this setting but I can't seem to find it. I'll have to look a little deeper.

Alex
04-17-2008, 02:28 PM
BTW, as we get more and more files uploaded, I can create sub-categories to better organize everything:)

Adrenalynn
04-17-2008, 02:41 PM
BTW, yes, we are scrapping 3D files. There are just too many to cover and it's just as easy (and probably save on storage) if members could compress them into an accepted format.


Agreed, which is why I was suggesting treating motion files the same way. They don't necessarily compress well (neither do .max 3D files), but a few byte header added by the archiver really doesn't change anything, right?

BTW, most likely (and I can check on it in a moment) you have the gzip module turned on in Apache anyway, which means that any substantial browser traffic is going over a compressed link from the get-go.

Alex
04-17-2008, 03:27 PM
When you say "don't compress well", do you mean that the file doesn't decrease in size much, or that the file gets a little "flaky" after compression?

Droid Works
04-17-2008, 03:37 PM
Soon as I get some free time I will post more files. I have 1000's of robotics programs, files and docs.

Adrenalynn
04-17-2008, 03:39 PM
Sorry -

The compression ratio is very low for the binary formats in general. They're designed to be compressed well in the creation process, not post-saving. Some will get minutely larger if zipped. But they'll still come out bit-for-bit true. As memory serves (counting bytes in my head), the overhead for .zip standard should be about 30bytes + filename + extended comments field of variable length. Generally, around 64byte - 255bytes of overhead. As I mentioned - not worth even considering the storage downside for uncompressable data these days. Just compress everything and call it an "archive". *

All the major compressors should be doing a CRC on their data these days (all the way back to the late 80's). Anything going in should come out lossless, unless the archive is damaged after creation. (bad upload, for example, or an ascii upload instead of a binary or armored upload)

OT follows:
[From memory, so don't shoot me... If you're interested, the zip header looks something like this:

file header sig: 4bytes
required version: 2bytes
gp bit flag: 2
compression method: 2bytes (0-8 are defined in the standard, 0 being store-only, 8 being max compression)
last modified time: 2
last modified date: 2
crc32: 4
compressed size: 4
uncompressed size: 4
filename length: 2
extra description field length: 2
filename: variable, see above
extra description field: variable, see above

Now you know more about zip than you ever wanted to.

Adrenalynn
04-17-2008, 03:40 PM
Soon as I get some free time I will post more files. I have 1000's of robotics programs, files and docs.

You rock, Droid! Thanks for the effort!

Dave
04-17-2008, 04:06 PM
Sorry -

The compression ratio is very low for the binary formats in general. They're designed to be compressed well in the creation process, not post-saving. Some will get minutely larger if zipped. But they'll still come out bit-for-bit true. As memory serves (counting bytes in my head), the overhead for .zip standard should be about 30bytes + filename + extended comments field of variable length. Generally, around 64byte - 255bytes of overhead. As I mentioned - not worth even considering the storage downside for uncompressable data these days. Just compress everything and call it an "archive". *

All the major compressors should be doing a CRC on their data these days (all the way back to the late 80's). Anything going in should come out lossless, unless the archive is damaged after creation. (bad upload, for example, or an ascii upload instead of a binary or armored upload)

OT follows:
[From memory, so don't shoot me... If you're interested, the zip header looks something like this:

file header sig: 4bytes
required version: 2bytes
gp bit flag: 2
compression method: 2bytes (0-8 are defined in the standard, 0 being store-only, 8 being max compression)
last modified time: 2
last modified date: 2
crc32: 4
compressed size: 4
uncompressed size: 4
filename length: 2
extra description field length: 2
filename: variable, see above
extra description field: variable, see above

Now you know more about zip than you ever wanted to.

Permit me to translate this for those of you who don't speak engineer-ese: In some cases, compression neither hurts nor helps; but for our purposes we're going to use archives to reduce the number of file extensions we have to deal with. If your file format isn't on our list, zip it.