When you run the Spark program on windows OS, you often get the exception “Exception in thread “main” java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z”
Full-stack trace of the error
Exception in thread “main” java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
Solution:
In order to run the Spark program on windows, you would need Hadoop
winutils.exe
file as windows don’t support HDFS and winutils provides a wrapper.
If you don’t have winutils.exe installed, please download the
wintils.exe
and
hadoop.dll
files from
https://github.com/steveloughran/winutils
(select the Hadoop version you are using as winutils are specific to Hadoop versions)
Copy the downloaded files to a folder on your system, for example, let’s say you have copied these files to
c:/hadoop/bin
, set the below environment variables.
set HADOOP_HOME=c:/hadoop
set PATH=%PATH%;%HADOOP_HOME%/bin;
Close and reload the command line or terminal to initialize these variables. Sometimes you may also need to put
hadoop.dll
file into the
C:/Windows/System32
folder.
Now run your spark program and issue “
Windows.access0(Ljava/lang/String;I)Z
” should disappear.
However, if it still doesn’t work, try to restart your system as some of the above settings would get the effect with the restart.
Happy Learning !!
Related Articles
