What’s The Best Way to Handle Extremely Large (1GB+) Text Files?

Large files and big data are one of many inevitabilities that comes with today’s computing. If you’ve ever had the misfortune of encountering a file so freakishly large that your poor text editor crashes when trying to open it, we feel your pain. We’ve been there.

“Where is all this data coming from?”

1. Humans have created more data in the past two years than in the history of the human race combined
2. 7MB of new data will be created for every human on the planet each second by 2020
3. By 2025, all the data on Earth will grow from 4.4 zettabytes today to roughly 175 zettabytes. That’s 44 trillion gigabytes
4. For the average Fortune 1000 company, a 10% increase in data accessibility will get them more than $65 million in additional net income

“How does a programmer or IT professional access these large files and extract meaningful information?”

People ask how to open large text files over and over again on sites like Stack Overflow. Despite being a pretty common question, there are next to no satisfactory answers floating around. So, we wrote this blog to talk about the different options you have when you come across a large file and the pros and cons of each.

Solution 1: Download a Dedicated Large File Viewer

If all you need to do is read the large file, you can download a dedicated large file viewer such as the Large Text File Viewer. Such tools will open large text files with ease. There are even online tools that will let you upload a large text file to a web application that will open them online, such as http://www.readfileonline.com.

On Windows, there is a program that comes pre-installed and can open text files of any size. It’s called WordPad.

Pros:

• If all you need to do is view a large file, this is the most efficient method
• This is the quickest method if you rarely come across large files
• It’s completely free

Cons:

• You have to download a completely new program and you add an extra program to your workflow whenever you need to open a large file
• You can’t make any edits to the file

Note: UltraEdit is also a large file viewer, and you can make edits to the file—we’ll get into that shortly.

Solution 2: Split the Large File into Smaller Chunks

If your text editor can’t open a 2GB text file, you can split it into 20 200MB text files instead. Your computer will store the newly segmented pieces in a neat little folder where you can go through them one at a time and edit them or find the information you were looking for.

On Mac and Linux, there is a native command (split) that you can type into the terminal that will segment the files according to rules that you specify. In Windows, you’ll have to download a program to do it for you. Fortunately, most of them are free (such as HJ Split, the most popular tool for this purpose).

Pros:

• The split files can easily be put back together through the cat command (on Mac and Linux)
• You have full control over how the file gets split
• This method keeps you from having to download a new program that complicates your workflow (on Mac and Linux)
• It’s completely free

Cons:

• You have to download a new program to do one thing and it doesn’t really fit in your workflow (in Windows)
• Splitting the files makes it more difficult to find a specific piece of information
• If you need to ultimately put the information back together, this doesn’t solve the problem

Solution 3: Get a Text Editor that Natively Handles Large Files

If you’ve read those Stack Overflow threads we posted above, you’ll know that people inevitably answer the question with examples of text editors that do open those large files. Well, large files up to some threshold.

Of course, this blog isn’t about the reason certain programs handle large files better than others, but the fact is that UltraEdit opens files well in excess of +4GB and that there are just no other editors that do a better job of handling large files than UltraEdit. None. There are even settings specific to large files in UltraEdit that allow it to perform even better with large files.

Large files are part of today’s workflow, and file size is growing by the minute. Switching text editors for the sake of being able to open larger files disrupts workflow and can complicate file manipulation. Why not use an editor than can do everything any other editor can do, plus has no limitation of file size?

On one hand, if your current text editor meets all your large file needs, this isn’t exactly the most useful article for you. On the other hand, if large files are a more frequent part of your workflow, it is likely worth the switch. In the end, only you can decide. There are a handful of editors that open large files to some extent. Even these have conditions such as turning off line numbers, turning off UNDO/REDO, etc.

Pros:

• By switching to an editor that can handle large files, you’ve solved the problem. Forever.

Cons:

• There is a small learning curve when switching text editors (but UltraEdit offers a professional, in-house support team ready to answer any questions you may have)

No matter which method you decided was best for your file, we hope you learned something from this blog. We’re hopeful that UltraEdit might be able to provide a solution that will make your life easier.

If you have some large file-opening methods that we didn’t cover here, please let us know on Facebook! We look forward to hearing from you!

Try UltraEdit for free: Download UltraEdit to load and edit your large Files. Use UltraEdit for 30 days and see why it’s the world’s #1 ranked text editor.