Breaking up files - Page 2

ewmayer · 2007-11-16, 16:56

Does anyone know if the "split" utility is able to do its work in-place, e.g. by simple feeding a suitably modified set of file start-end parameters to the OS? For really huge files, in-placeness [or not] would seem to be a key property.

[IIRC under *nix the relevant piece of the OS is the master boot record, but I could well be remembering that wrong- o mighty Unix gurus, hear my plea].

Edit (By popular request): By "in-place" I mean virtually breaking up the file, i.e. without actually moving any of the *contents* around on disk. To use a coding analogy, imagine you have a big C array, and instead of a single pointer to the head element, you provide multiple pointers to various chunks of the array, as well as a secondary datum associated with each pointer giving the size of the chunk the pointer points to. The analogy is illustrative but not exact, since files-on-disk may be fragmented into separate chunks. But contemplate a resizable array, where you want to add extra array storage without moving the existing contents around - the mechanism for this is essentially the same as used for file-management-on-disk. [And of course necessary there because disk storage is inherently slow and files are such dynamic entities.]

p.s.: Sorry for the pedantry, but you [well, just one of you] *did* ask...

Wacky · 2007-11-16, 19:50

[IIRC under *nix the relevant piece of the OS is the master boot record, but I could well be remembering that wrong- o mighty Unix gurus, hear my plea].

Nope. The Master Boot Record has to do with (bootstrapping code and) Hard Disk partitioning.

In almost all cases, there is some file system that is held within the partitions.
The partition is broken into logical blocks (originally disk sectors) and they are assigned to various files. In general, each file requires many logical blocks. Often, they are not consecutive. In some file systems, it is possible to break up a large file by manipulating the metadata that associates groups of logical blocks with a particular file. But, the particulars are very dependent on the type of file system.

Citrix · 2007-11-17, 06:50

Quote:

Originally Posted by roger

How would I do that?

.

Start-> my computer-> properties-> advanced (need admin rights)--> Performance--> Settings--> Advanced tab--> Virtual memory.

2007-11-16, 16:56	#12
ewmayer ∂²ω=0 Sep 2002 República de California 103×113 Posts	Does anyone know if the "split" utility is able to do its work in-place, e.g. by simple feeding a suitably modified set of file start-end parameters to the OS? For really huge files, in-placeness [or not] would seem to be a key property. [IIRC under nix the relevant piece of the OS is the master boot record, but I could well be remembering that wrong- o mighty Unix gurus, hear my plea]. Edit (By popular request): By "in-place" I mean virtually breaking up the file, i.e. without actually moving any of the contents* around on disk. To use a coding analogy, imagine you have a big C array, and instead of a single pointer to the head element, you provide multiple pointers to various chunks of the array, as well as a secondary datum associated with each pointer giving the size of the chunk the pointer points to. The analogy is illustrative but not exact, since files-on-disk may be fragmented into separate chunks. But contemplate a resizable array, where you want to add extra array storage without moving the existing contents around - the mechanism for this is essentially the same as used for file-management-on-disk. [And of course necessary there because disk storage is inherently slow and files are such dynamic entities.] p.s.: Sorry for the pedantry, but you [well, just one of you] did ask... Last fiddled with by ewmayer on 2007-11-16 at 18:12

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Posting log files or other text files	Xyzzy	Forum Feedback	3	2018-12-30 19:37
Grammar rule-breaking ftw	jasong	Lounge	71	2013-10-15 03:39
Breaking: US DOJ Spied for Months on AP Reporters	ewmayer	Soap Box	11	2013-06-06 06:15
Breaking a prime p into a^2 + 3* b^2	SPWorley	Math	3	2009-08-26 03:05
Beowolf cluster on the Cheap, breaking 100$/GFlop	jflin	Hardware	8	2007-09-06 08:25

2007-11-16, 19:50	#13
Wacky Jun 2003 The Texas Hill Country 1089₁₀ Posts	[IIRC under *nix the relevant piece of the OS is the master boot record, but I could well be remembering that wrong- o mighty Unix gurus, hear my plea]. Nope. The Master Boot Record has to do with (bootstrapping code and) Hard Disk partitioning. In almost all cases, there is some file system that is held within the partitions. The partition is broken into logical blocks (originally disk sectors) and they are assigned to various files. In general, each file requires many logical blocks. Often, they are not consecutive. In some file systems, it is possible to break up a large file by manipulating the metadata that associates groups of logical blocks with a particular file. But, the particulars are very dependent on the type of file system.