Low end devices experience this error with files as small as 30Mb, whereas flagship models released since 2020 give this error around 75Mb large files and up. Android Emulators with an amped Heap size of 1Gb give this error starting from 150Mb, which is still relatively small concerning the use case of my app.
I'm developing an offline-first application for some 2 years now using Expo, which is technically concerned with storing rich content using Expo's FileSystem. The Filesystem is used in two key parts relevant to this critical issue:
I found that no one is experiencing a similar error on the internet, whereas the use-case seems to be very relevant in an offline-first scenario.
Is there a better way to load CSVs, JSONs, or other datasets for offline backups, or on-device analysis of data, in React-Native? Any clues, tips, or insights will be helpful.
Please avoid stating your preference for offloading personal user data to servers, and providing them just-in-time when users need them.
Run app in Expo Go, load a big JSON file after button click, might or might not cause an OOM error.
const load_file = async () => {
try {
data_obj = await DocumentPicker.getDocumentAsync({copyToCacheDirectory:true, type: '*/*'});
let pos_ii = 0, chunk_ii = 0, chunk_size = 1000000; // 1 Mb chunk
let data_string_array = [];
while (pos_ii < backup_obj.size) {
// load chunk
let _data_string_tmp = await FileSystem.readAsStringAsync(backup_obj.uri, {
encoding: FileSystem.EncodingType.UTF8,
position:pos_ii,
length:chunk_size}
// push chunk & update iters
data_string_array.push(_data_string_tmp);
pos_ii+=chunk_size
chunk_ii+=1;
// join chunks
const data_string = data_string_array.join();
console.log(`Loaded ${data_string.length} bytes`);
} catch (error) {
console.log(`Big error in importing: ${error}!`);
Using this approach, the aforementioned error occurs whilst reading the first chunk. The OOM exception states the same memory size allocation attempt as when reading the whole file at the same time.
Trying smaller chunk sizes (like 1 Kb) also triggers an OOM, hence my conclusion is that FileSystem.readAsStringAsync attempts to read the whole file anyhow.
Hi, I'm having almost exactly the same problem when I'm writing a backup file from my app using writeAsStringAsync and I get the Out Of Memory error on Expo Go on android (both on simulator and real device) when I try to write a file of about 30MB or bigger. I have two questions about the largeHeap flag:
Does anybody know if it's set to true in the ExpoGo app?
How to edit the android manifest to add android:largeHeap=true in an expo managed workflow?
@PhilemonBel the largeHeap flag isn't set in the ExpoGo app. You can set android:largeHeap=true (or any other flag) via EAS-builds and plugins.
Expo developers have provided a helpful guide here, which involves creating a plugin file, then mentioning your desired flags in app.js:
"expo": {
[
...]
"plugins": [
"./plugins/withAndroidMainActivityAttributes",
"android:largeHeap":
true,
"android:hardwareAccelerated":
true
The bad news is that I already tried this option, and I build my apps with forum-recommended flags enabled on both simulator and 3+ Real Android devices. The result is that OOM memories are still being played out.
The underlying issue might be how react-native is being build into android code. On forums I've seen native android developers (java) being able to load ridiculous amounts of data into working memory, yet react-native developers pump against JVMs limitation even with largeHeap=true, which might be linked to how JSC or Hermes is handling or paging large memory operations to metal.
I've also experienced a similar issue on expo web builds whilst loading 60Mb+ files using the above minimal code, the web app just CRASHES with no error. There too, I suspect, memory handling by the javascript engine misses the boat.
I'm currently reworking the whole data management architecture of my application to comply with this ridiculous 25Mbyte Heap memory size constraint, but it still remains an issue and a barking problem which I suspect is to be addressed at the engine level.
@adamblvck I am having the exact same issues as you as it relates to Out of Memory Exceptions. I am attempting to upload an mp4 directly to an s3 bucket. At which point everything works until the file size goes above 190mb then let data = fetch(localUri) fails with "Network Request Failed", this was odd since I simply wanted to blob the read Stream fetch returns, this is all done locally so 'Network Request Failed' is not applicable . When trying to figure out a work around to this I stumbled across the same error you have with FileSystem.readAsStringAsync() on the exact same files that fail to fetch due to size. Leading me to believe fetch fails due to this heap size limit but does not return helpful data on this error. XML also fails in the same way.
Ill start with something I discovered that may help you. In the FileSystem docs it states that {position: x, length:x} only work with FileSystem.EncodingType.Base64. Above another user asked if you tried base64 encoding and you said it made no difference. However, when I tested
base64 = await FileSystem.readAsStringAsync(file, {encoding:FileSystem.EncodingType.Base64, position :0 , length: 10000000});
it does in fact work on files upto 350 mb(max I tested). However, the same file and code with UTF8 it fails.
I noticed in your above response you typed. FileSystem.EncodingType.BASE64 when it is in fact FileSystem.EncodingType.Base64. When I type it in the way you may have BASE64 it does not throw an error however it does still encode it as utf8, and continues to the throw the memory exception error.
I am making a internal distribution app that is running on an s8 ultra with 8gb of ram. I find it extremely surprising that I cannot allocate more of that ram then 205mb, when I followed the tutorial for plugins you linked
and added a plugin that changed largeHeap to true. I got the exact same Failed to allocate error at the exact same memory level. How does one test what one's memoryHeap limit actually is and how does one confirm that "largeHeap": true is actually set?
ExponentFileSystem: Failed to allocate a 537133064 byte allocation with 25165824 free bytes and 205MB until OOM, target footprint 346481488, growth limit 536870912]
It seems like maybe the tutorial was incorrect and one may have to eject from a managed expo workflow. I found many cases on stack overflow in which largeHeap:true worked successfully on react native applications, but your link was the only description of how to do it in a managed workflow.
incomplete issue: missing or invalid repro
A minimal reproducible example is required for most issues
and removed
needs validation
Issue needs to be validated
labels
Feb 22, 2023
Hi there! It looks like your issue requires a minimal reproducible example, but it is invalid or absent. Please prepare such an example and share it in a new issue.
The best way to get attention to your issue is to provide a clean and easy way for a developer to reproduce the issue on their own machine. Please do not provide your entire project, or a project with more code than is necessary to reproduce the issue.
A side benefit of going through the process of narrowing down the minimal amount of code needed to reproduce the issue is that you may get lucky and discover that the bug is due to a mistake in your application code that you can quickly fix on your own.
Resources
"How to create a Minimal, Reproducible Example"
"How to narrow down the source of an error"
Common concerns
"I've only been able to reproduce it in private, proprietary code"
You may not have spent enough time narrowing down the root cause of the issue. Try out the techniques discussed in this manual debugging guide to learn how to isolate the problem from the rest of your codebase.
"I didn't have time to create one"
That's understandable, it can take some time to prepare. We ask that you hold off on filing an issue until you are able to fully complete the required fields in the issue template.
"You can reproduce it by yourself by creating a project and following these steps"
This is useful knowledge, but it's still valuable to have the resulting project that is produced from running the steps, where you have verified you can reproduce the issue.
@brentvatne I noticed that you flagged this issue as incomplete because there's no minimal reproducible example, whereas in fact there is a minimal reproducible example in the first post, and a few more as the thread evolved. Also as @JohnyClash mentioned, this is an active thread.
Please remove the tags which you've assigned otherwise these things won't get a chance to get fixed, nor addressed.
@JohnyClash I'll perhaps run some tests using Base64 instead of BASE64, which in the end won't truly fix the issue. On my dev devices I reach OOM between 30-120 MBytes, while being 8-16Gb devices. In a modern use case scenarios, like for example for utilizing local deep learning models (50-100Mb loaded in memory), or processing local node data for distributed data processing, this is a complete no-go.
As for the LargeHeap flag, today I tried opening my build apks (I use EAS Build) with ClassyShark to investigate the AndroidManifest, and surprisingly, the largeHeap hasn't been set to true.
So I believe there's more to excavate in this particular issue - namely - the correct setting of the largeHeap flag.
this was automatically closed when we ran an issue validation script across all open issues that had the "needs review" label - the reason it was closed is there wasn't a link to a reproducible example. it'd make this more quickly actionable for someone on the expo side if there was a clonable repro. i'll reopen nonetheless
incomplete issue: missing or invalid repro
A minimal reproducible example is required for most issues
labels
Feb 27, 2023
Thank you for filing this issue!
This comment acknowledges we believe this may be a bug and there’s enough information to investigate it.
However, we can’t promise any sort of timeline for resolution. We prioritize issues based on severity, breadth of impact, and alignment with our roadmap. If you’d like to help move it more quickly, you can continue to investigate it more deeply and/or you can open a pull request that fixes the cause.
@JohnyClash I revised the Expo Plugin, which you can now find in this gist. It correctly enables the android:largeHeap flag in AndroidManifest.
I've performed a few tests and have encountered no OOM errors thus far (up to 80 Mb files), which is fantastic news. I'll be testing more next week with heavier files. Please also try in your own use case!
@stefmedjo I don't know what you mean by that. Could you elaborate?
Sorry, I made a mistake (I was creating chunks and pasting it in a new file, didn't notice that when you are writing in a file, all previous content is erased).
So now I am reading a file with encoding base64 as parameter using position and chunk size and concatenating the result in a content variable.
I still get the OOM error.
@djaffer hey, thank you for pointing to the plugin and post. I believe I've mentioned it in this issue too, and sadly it didn't set the largeHeap flag correctly (after configuring app config) when built through EAS build.
The gist which I provided a few comments back contains a plugin which correctly sets the largeHeap flag to true. I've verified it using ClassyShark, and my app can also sustain more RAM now.
I'll be testing how far I can push it before hitting OOM.
