/commands/coreutils/split-large-file
split: split a large file into smaller files by lines or size
- tool
- GNU coreutils
- versions
- GNU coreutils 8.x, GNU coreutils 9.x
- os
- linux, macos, windows
split cuts one file into several smaller files: split -l 1000 big.log part_ writes 1000 lines per piece, split -b 100M big.iso part_ writes 100 MB per piece, and split -n 4 file makes four pieces of equal size. The pieces are named with the prefix you give (default x) followed by a suffix: aa, ab, ac, or 00, 01, 02 with -d. Join them again with cat part_* > file. The original file is never changed.
Split by lines, bytes or number of pieces
seq 1 10 > n.txt
split -l 4 n.txt part_
wc -l part_*
head -c 3000 /dev/zero > z.bin
split -b 1K -d z.bin z.bin.
ls z.bin.*
wc -c z.bin.*
cat z.bin.* > back.bin
cmp z.bin back.bin && echo "joined file is identical"Output
4 part_aa 4 part_ab 2 part_ac 10 total z.bin.00 z.bin.01 z.bin.02 1024 z.bin.00 1024 z.bin.01 952 z.bin.02 3000 total joined file is identical
-l 4 put lines 1-4, 5-8 and 9-10 in three files; the last piece holds what is left. -b 1K cut the 3000-byte file into two pieces of 1024 bytes and one of 952, and cat in glob order rebuilt an identical copy.
Options
| Option | What it does |
|---|---|
| -l N | N lines per piece (default 1000 lines when no size option is given) |
| -b SIZE | SIZE bytes per piece; suffixes K, M, G are powers of 1024, KB, MB, GB powers of 1000 |
| -C SIZE | At most SIZE bytes per piece without breaking lines (GNU) |
| -n N | Exactly N pieces of about the same size in bytes |
| -n l/N | N pieces, but only at line boundaries |
| -d | Numeric suffixes 00, 01, 02 instead of aa, ab, ac |
| -a N | Suffix length N (default 2) |
| --additional-suffix=.txt | Append an extension to every piece name (GNU) |
| --filter=CMD | Pipe each piece into CMD instead of writing it; $FILE holds the piece name (GNU) |
| - | Read standard input instead of a file |
Numbered pieces with an extension
seq 1 9 > n.txt
split -n l/3 -d --additional-suffix=.txt n.txt chunk_
for f in chunk_*; do echo "$f: $(paste -sd, "$f")"; doneOutput
chunk_00.txt: 1,2,3 chunk_01.txt: 4,5,6 chunk_02.txt: 7,8,9
-n l/3 made three pieces without cutting any line, and --additional-suffix gave each one a .txt extension, which is useful when another program decides how to open a file by its extension.
Pitfalls
- Suffixes run out. With the default two-letter suffix there are 676 names (
aatozz). Beyond that GNU split widens the suffix automatically when no-ais given; with a fixed-a 2it stops withsplit: output file suffixes exhausted. Set-a 3or more for very large splits. - Existing pieces are overwritten without a warning. Use a fresh prefix or an empty directory.
- macOS and BSD split support
-l,-b,-nand-a, and-don recent macOS releases, but not--additional-suffix,--filteror-C. Install GNU coreutils (gsplit) when a script relies on them. - Binary files must be split with
-bor-n, never with-l, and joined withcatin the same order.
Check that it worked
Count the pieces and their sizes with wc -l part_* or wc -c part_*, then rebuild the file and compare it with the original using cmp, as in the first example, or with sha256sum before and after a transfer.
FAQ
How do I join the pieces back together?
Concatenate them in name order with cat, for example cat part_* > original.txt. The default suffixes (aa, ab, ac, or 00, 01, 02 with -d) sort correctly with the shell glob, so the pieces come back in the right order. Compare the result with cmp or a checksum.
How do I keep the header line in every piece of a CSV?
split has no option for it. Split the body with tail -n +2 data.csv | split -l 1000 - body_, then prepend the header to each piece, or use split --filter with a small shell command that prints the header before the chunk.
Why does split -b cut a line in half?
-b counts bytes, not lines, so a piece can end in the middle of a line. Use -C SIZE (GNU) to put at most SIZE bytes of whole lines in each piece, or -n l/N to make N pieces without splitting lines.