Online since 2002. Over 3300 puzzles, 2600 worldwide members, and 270,000 messages.

TwistyPuzzles.com Forum

It is currently Fri Apr 18, 2014 6:18 am

All times are UTC - 5 hours



Post new topic Reply to topic  [ 3 posts ] 
Author Message
 Post subject: Requesting help from fellow Linux Users
PostPosted: Sat Dec 29, 2012 8:18 am 
Offline

Joined: Sun Nov 23, 2008 2:18 am
Okay, I am having several issues I need some help with, and google searches are producing more frustration than useful information. Please note: Due to my vision, I am working strictly from the command-line, so please do not suggest any graphical applications.

1. I recently lost my collection of flac files ripped from CD in a drive format, but fortunately, I have recover the vast majority of the files thanks to photerec, and lltag has gone a long way to helping make-up for the fac that photorec cannot recover filenames. However, I have a few issues that leave me with about 200 flac files that are still a disorganized mess:

-Several albums worth of music with non-Latin characters in their tags, which translate to filenames that the command line cannot properly display and my screen reader cannot properly read. The culprits are mainly tracks from Japanese or Chinese artists. I would like to scan my entire music collection, romanizing the artist, album, title, etc tags of the Japanese and Chinese music in my collection as well as removing any accents from European music. Is there a command-line tool that can help with this?
-Several tracks, sometimes entire albums worth, that have valid Artist, Album, and Title tags, but are lacking a proper track number, preventing lltag from naming them in a manner that maintains the proper track ordering. Is there any command-line utility that can take the existing tag information, find the corresponding CDDB entry, and fill in the corrupted/missing tags?
-10 albums worth of files that have blank tags due to no CDDB results when the discs were originally ripped. Mostly classical, but at least 2 j-pop albums are mixed in with them. Is there anything that might be able to help give these files the proper tags?

For the record, most, if not all of the flac files were originally ripped via abcde and tagged automatically based on its CDDB query.

2. I have a large collection of files that needs organizing, and the first step towards that is converting the mixture of roman and Japanese text in their filenames to pure romaji so that I can properly manipulate the files via command line utilities. The second step is figuring out a way to mass extract the large number of archive files among this collection to individual folders so that I can compare their content to already organized files on my system. And speaking of comparing files, fdupes has proven quite useful for finding and removing duplicate files, and while the -N option is useful for cases that return large number of duplicate files, I have been unable to reliably determine which copy of a file will be automatically kept and which will be automatically deleted. Say for example, I have a folder of Sorted content that I know has no duplicates, and an unsorted folder that may have one or more duplicates, including duplicates of files in the sorted folder. Is there a way to force fdupes to keep everything in sorted, and delete any dupees it finds in unsorted without having to manually select which file to keep on a case-by-case basis.

3. I am running low on storage space and wish to compress a large collection of files I do not access very often. The files in question are well organized in a multi-tiered directory structure, and I would like to create an archive for each bottom level dierectory without losing the directory. Is their a way to automate this process, and what archive format would be best suited to the purpose. If it matters, the majority of the files are monochrome jpeg, though there are also png and other assorted file types in the mixed.

4. I need to burn some isos to cd. my understanding is that wodim is the tool for the job, but I can find a straight foward guide via google, the help page is longer than my screen and piping to nano failed to create a text document of the help page. I do not need anything fancy , just the command line to burn file.iso in the working directory to the blank CD-R in my only optical drive.

5. Since I have been relying on the command line for file management, I have noticed that when I use the mv command to to move entire directory structures, that if the source and destination have subdirectories with the same name, mv stops. Is there a way to make it add the contents from the source version of the directory to the into the destination copy?

Any help with any of these issues would be greatly appreciated.

_________________
Just so you know, I am blind.

I pledge allegiance to the whole of humanity, and to the world in which we live: one people under the heavens, indivisible, with Liberty and Equality for all.

My Shapeways Shop


Top
 Profile  
 
 Post subject: Re: Requesting help from fellow Linux Users
PostPosted: Sat Dec 29, 2012 11:14 am 
Offline
User avatar

Joined: Thu Dec 31, 2009 8:54 pm
Location: Bay Area, California
I'm going to be away / too busy for the next few days to provide a lengthy response.

I don't have a canned solution for your music tags / filenames issue since the devil is going to be in the details. It sure sounds like what you need is a custom script. One of the big issues you'll run into is how non-latin characters are encoded in the tags. You'll want to normalize to something, probably UTF8. Look into iconv. try 'man iconv' and 'man 3 iconv' to see the interface. Stripping accents is non-trivial. There are often multiple ways to store an accented character so you'll need to do some research.

I've never used fdupes so I can't comment much about its usage. I have written de-duplication script that used md5deep (or now sha256deep) to determine which files are the same. I'd suggest making a list of file hashes and then for each unsorted directory you can compare against the hash list and delete / link the duplicates.

For extracting archives, the devil is in the details. What archives? Zips? tar-{gz, bz2, xz, lzo, etc}? Rar?. A script to extract them all is trivial and would look something like

Code:
find dir/ | egrep -i '\.zip$' | while read LINE; do unzip $LINE; done


But you'll have to worry about the cwd and whether the zip has all of the files contained in one directory or not. You'll probably want a bit more smarts to the extraction.

For archiving your own files, if the files you intend to compress are JPEG and PNG files you're wasting your time. Those won't compress at all. The same goes for MP3 and FLAC and most other large media files. It sounds like you may want transparent filesystem compression instead. Then files that do compress automatically save you space. The best way to do this on Linux today is btrfs mounted with compress=gzip or compress=lzo. I'd suggest gzip if you want higher compression and lzo if you want fast writes. Benchmarks show btrfs with compression to outperform other filesystems, often significantly on nearly other operation other than writing.

If you want to do the archiving yourself, for the best compression ratio I'd suggest tar archive and XZ compression. If you have a ton of memory and CPU on your hands and want the best compression ratio possible use something like:
Code:
xz --stdout -z -vv --lzma2=dict=320MiB,lc=4,lp=0,pb=2,mode=normal,nice=273,mf=bt4,depth=1000 file > out.xz


For burning ISOs, I use cdrecord. I have a simple script for it:
Code:
brenrigh@lambda ~ $ cat scripts/burniso.sh
#!/bin/bash

/usr/bin/cdrecord -v gracetime=2 dev=/dev/sr0 -dao driveropts=burnfree -eject -data $1


If you want to make your own ISOs I use mkisofs. I don't have my script handy for this. The man page is huge and figuring out how iso9660 works is a pain. If you need to do this I'll dig up my script for examples.

You don't want to be using mv for what you're doing. mv is a rather thin wrapper around some VFS calls and doesn't have the smarts you need. Instead use rsync which can do local to local copies. It has all of the features you want and more for merging directories, resuming copying, etc. Once you've synced two locations then you can delete the source.

_________________
Prior to using my real name I posted under the account named bmenrigh.


Top
 Profile  
 
 Post subject: Re: Requesting help from fellow Linux Users
PostPosted: Thu Jan 03, 2013 9:28 am 
Offline

Joined: Sun Nov 23, 2008 2:18 am
To provide more details:

The archives in question are mostly zip and rar, with a few lzh archives, sorted into folders by extension and no further organization. Normally, I would just use
Code:
7z x '*.ext'


to extract all of the archives in a folder. However, many of the archives in question are ill-behaved, and doing so would scatter their contents all over the the current folder and cause problems if there are any clashing filenames.

Ideally, I would like a script that:
1. For each archive, creates a folder of the same name, minus the extension.
2. Extract each archive to its corresponding folder.
3. deletes each archive as it is extracted without error.

Also, transparent filesystem compression does sound like the more viable solution, though I have to ask if brtfs can applied to an existing ext3/4 volume without having to off-load its entire contents to an alternate storage location. Though, now that I think about it, the real problem is not lack of storage space, but rather the limitation of the partitioner used my my distribution's installer.

In response to my recent vision problems, I recently switched from vanilla Debian to Adriane Knoppix, which provides me with an excellent suite of text-based applications, a screenreader, and a wrapper for launching Firefox with the Orca screen reader without having to launch a full X-session. Unfortunately, there is a bit of instability in Orca at the moment with it either completely refusing to work, or working but rendering any firefox windows inaccessible. So far, the most reliable solution I have found for when this breakage occurs is to reinstall Knoppix, sometimes requiring a complete drive format. Unfortunately, manual partitioning within the installer used by knoppix is unintuitive to the point of being useless, and its guided mode wipes my entire internal drive and does not even include the option for a separate home partition. Between internal and external drives, I have plenty of storage, but with me feeling that my internal drive is no longer safe for data storage, available space is cut in half. setting btrfs on my external drive is probably a good idea to make the most of available storage, but I need to know a good partitioner I could use from the command line to partition my internal drive in a manner that makes it suitable both for a harddrive install of Adriane, and serving as additional storage space. I really like the partitioner used by the text-mode installer in Debian, but I do not know if it is available as a .deb from the repositories for use outside of the debian installer.

As for iso burning, I use it mainly for creating linux install discs, and I was going to use cdrecord, but apparently, it has been superceded by wodim and is no longer available from Debian's repositories. Fortunately, I was able to get help with wodim from one of the other sources I inquired at.

_________________
Just so you know, I am blind.

I pledge allegiance to the whole of humanity, and to the world in which we live: one people under the heavens, indivisible, with Liberty and Equality for all.

My Shapeways Shop


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 3 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Forum powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group