vfs classpath could not replicate

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

vfs classpath could not replicate

Rob Verkuylen

Hi,


I'm trying to get vfs loading of my jar working in 1.7.2. On a test cluster of the same version this works fine and I see the jar being replicated in the cache in '/tmp/accumulo-vfs*/fstore-filters-1.3.2.jar', but when I do the exact same thing on prod, I get the stacktrace below. Any ideas? 


Setup I used:

config -s general.vfs.context.classpath.fstore=hdfs://nameservice1/libs/accumulo/fstore/.*jar
config -t test.fstore_index -s table.classpath.context=fstore

Stacktrace on tablet server:
Failed to load class
            java.lang.ClassNotFoundException: IO Error loading class org.apache.accumulo.tserver.compaction.DefaultCompactionStrategy
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:188)
                        at org.apache.accumulo.core.conf.Property.createInstance(Property.java:873)
                        at org.apache.accumulo.core.conf.Property.createTableInstanceFromPropertyName(Property.java:910)
                        at org.apache.accumulo.tserver.TabletServerResourceManager$TabletResourceManager.needsMajorCompaction(TabletServerResourceManager.java:647)
                        at org.apache.accumulo.tserver.tablet.Tablet.needsMajorCompaction(Tablet.java:1594)
                        at org.apache.accumulo.tserver.tablet.Tablet.initiateMajorCompaction(Tablet.java:1574)
                        at org.apache.accumulo.tserver.TabletServer$MajorCompactor.run(TabletServer.java:1875)
                        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
                        at java.lang.Thread.run(Thread.java:745)
            Caused by: org.apache.commons.vfs2.FileSystemException: Could not replicate "hdfs://nameservice1/libs/accumulo/fstore/fstore-filters-1.3.2.jar".
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:426)
                        at org.apache.commons.vfs2.provider.zip.ZipFileSystem.<init>(ZipFileSystem.java:66)
                        at org.apache.commons.vfs2.provider.jar.JarFileSystem.<init>(JarFileSystem.java:48)
                        at org.apache.commons.vfs2.provider.jar.JarFileProvider.doCreateFileSystem(JarFileProvider.java:80)
                        at org.apache.commons.vfs2.provider.AbstractLayeredFileProvider.createFileSystem(AbstractLayeredFileProvider.java:87)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1022)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1042)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.addFileObjects(VFSClassLoader.java:156)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.<init>(VFSClassLoader.java:119)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:147)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:162)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager$Context.getClassLoader(ContextManager.java:46)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.getClassLoader(ContextManager.java:174)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:186)
                        ... 8 more
            Caused by: org.apache.commons.vfs2.FileSystemException: Unknown message with code "No such file or directory".
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:68)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.doReplicateFile(AbstractFileSystem.java:473)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:422)
                        ... 21 more
            Caused by: java.io.IOException: No such file or directory
                        at java.io.UnixFileSystem.createFileExclusively(Native Method)
                        at java.io.File.createTempFile(File.java:2024)
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:60)
                       
... 23 more


 

Reply | Threaded
Open this post in threaded view
|

Re: vfs classpath could not replicate

Christopher Tubbs-2
It looks like the error is when vfs tries to create a temporary file in your tmpdir. I would check that your java.io.tmpdir points to a directory that exists and has the appropriate permissions for the user running the Accumulo process.

On Wed, Sep 26, 2018, 16:08 Rob Verkuylen <[hidden email]> wrote:

Hi,


I'm trying to get vfs loading of my jar working in 1.7.2. On a test cluster of the same version this works fine and I see the jar being replicated in the cache in '/tmp/accumulo-vfs*/fstore-filters-1.3.2.jar', but when I do the exact same thing on prod, I get the stacktrace below. Any ideas? 


Setup I used:

config -s general.vfs.context.classpath.fstore=hdfs://nameservice1/libs/accumulo/fstore/.*jar
config -t test.fstore_index -s table.classpath.context=fstore

Stacktrace on tablet server:
Failed to load class
            java.lang.ClassNotFoundException: IO Error loading class org.apache.accumulo.tserver.compaction.DefaultCompactionStrategy
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:188)
                        at org.apache.accumulo.core.conf.Property.createInstance(Property.java:873)
                        at org.apache.accumulo.core.conf.Property.createTableInstanceFromPropertyName(Property.java:910)
                        at org.apache.accumulo.tserver.TabletServerResourceManager$TabletResourceManager.needsMajorCompaction(TabletServerResourceManager.java:647)
                        at org.apache.accumulo.tserver.tablet.Tablet.needsMajorCompaction(Tablet.java:1594)
                        at org.apache.accumulo.tserver.tablet.Tablet.initiateMajorCompaction(Tablet.java:1574)
                        at org.apache.accumulo.tserver.TabletServer$MajorCompactor.run(TabletServer.java:1875)
                        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
                        at java.lang.Thread.run(Thread.java:745)
            Caused by: org.apache.commons.vfs2.FileSystemException: Could not replicate "hdfs://nameservice1/libs/accumulo/fstore/fstore-filters-1.3.2.jar".
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:426)
                        at org.apache.commons.vfs2.provider.zip.ZipFileSystem.<init>(ZipFileSystem.java:66)
                        at org.apache.commons.vfs2.provider.jar.JarFileSystem.<init>(JarFileSystem.java:48)
                        at org.apache.commons.vfs2.provider.jar.JarFileProvider.doCreateFileSystem(JarFileProvider.java:80)
                        at org.apache.commons.vfs2.provider.AbstractLayeredFileProvider.createFileSystem(AbstractLayeredFileProvider.java:87)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1022)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1042)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.addFileObjects(VFSClassLoader.java:156)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.<init>(VFSClassLoader.java:119)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:147)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:162)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager$Context.getClassLoader(ContextManager.java:46)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.getClassLoader(ContextManager.java:174)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:186)
                        ... 8 more
            Caused by: org.apache.commons.vfs2.FileSystemException: Unknown message with code "No such file or directory".
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:68)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.doReplicateFile(AbstractFileSystem.java:473)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:422)
                        ... 21 more
            Caused by: java.io.IOException: No such file or directory
                        at java.io.UnixFileSystem.createFileExclusively(Native Method)
                        at java.io.File.createTempFile(File.java:2024)
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:60)
                       
... 23 more


 

Reply | Threaded
Open this post in threaded view
|

Re: vfs classpath could not replicate

Rob Verkuylen

Hmm 'jinfo <tabletserver_pid> | grep tmp'  shows: 

java.io.tmpdir = /tmp


"ls -hal /" shows 777 on /tmp


Grepping for 'tmp' in tserver startup:

[zookeeper.ZooKeeper] INFO : Client environment:java.io.tmpdir=/tmp

[server.Accumulo] INFO : general.vfs.cache.dir = /tmp/accumulo-vfs-cache-accumulo

[server.Accumulo] INFO : tserver.dir.memdump = /tmp


I'm guessing the relevant code to get the dir would be inAccumuloVFSClassLoader.java

String cacheDirPath = AccumuloClassLoader.getAccumuloString(VFS_CACHE_DIR, System.getProperty("java.io.tmpdir"));  

with 

public static final String VFS_CACHE_DIR = "general.vfs.cache.dir";


which then would be ' /tmp/accumulo-vfs-cache-accumulo', but it seems this dir does not not need to exist and will be created. Could there be any other reason this would pop up? Can these settings be overridden somewhere? 




On Wed, Sep 26, 2018 at 10:17 PM Christopher <[hidden email]> wrote:
It looks like the error is when vfs tries to create a temporary file in your tmpdir. I would check that your java.io.tmpdir points to a directory that exists and has the appropriate permissions for the user running the Accumulo process.

On Wed, Sep 26, 2018, 16:08 Rob Verkuylen <[hidden email]> wrote:

Hi,


I'm trying to get vfs loading of my jar working in 1.7.2. On a test cluster of the same version this works fine and I see the jar being replicated in the cache in '/tmp/accumulo-vfs*/fstore-filters-1.3.2.jar', but when I do the exact same thing on prod, I get the stacktrace below. Any ideas? 


Setup I used:

config -s general.vfs.context.classpath.fstore=hdfs://nameservice1/libs/accumulo/fstore/.*jar
config -t test.fstore_index -s table.classpath.context=fstore

Stacktrace on tablet server:
Failed to load class
            java.lang.ClassNotFoundException: IO Error loading class org.apache.accumulo.tserver.compaction.DefaultCompactionStrategy
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:188)
                        at org.apache.accumulo.core.conf.Property.createInstance(Property.java:873)
                        at org.apache.accumulo.core.conf.Property.createTableInstanceFromPropertyName(Property.java:910)
                        at org.apache.accumulo.tserver.TabletServerResourceManager$TabletResourceManager.needsMajorCompaction(TabletServerResourceManager.java:647)
                        at org.apache.accumulo.tserver.tablet.Tablet.needsMajorCompaction(Tablet.java:1594)
                        at org.apache.accumulo.tserver.tablet.Tablet.initiateMajorCompaction(Tablet.java:1574)
                        at org.apache.accumulo.tserver.TabletServer$MajorCompactor.run(TabletServer.java:1875)
                        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
                        at java.lang.Thread.run(Thread.java:745)
            Caused by: org.apache.commons.vfs2.FileSystemException: Could not replicate "hdfs://nameservice1/libs/accumulo/fstore/fstore-filters-1.3.2.jar".
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:426)
                        at org.apache.commons.vfs2.provider.zip.ZipFileSystem.<init>(ZipFileSystem.java:66)
                        at org.apache.commons.vfs2.provider.jar.JarFileSystem.<init>(JarFileSystem.java:48)
                        at org.apache.commons.vfs2.provider.jar.JarFileProvider.doCreateFileSystem(JarFileProvider.java:80)
                        at org.apache.commons.vfs2.provider.AbstractLayeredFileProvider.createFileSystem(AbstractLayeredFileProvider.java:87)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1022)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1042)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.addFileObjects(VFSClassLoader.java:156)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.<init>(VFSClassLoader.java:119)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:147)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:162)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager$Context.getClassLoader(ContextManager.java:46)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.getClassLoader(ContextManager.java:174)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:186)
                        ... 8 more
            Caused by: org.apache.commons.vfs2.FileSystemException: Unknown message with code "No such file or directory".
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:68)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.doReplicateFile(AbstractFileSystem.java:473)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:422)
                        ... 21 more
            Caused by: java.io.IOException: No such file or directory
                        at java.io.UnixFileSystem.createFileExclusively(Native Method)
                        at java.io.File.createTempFile(File.java:2024)
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:60)
                       
... 23 more


 

Reply | Threaded
Open this post in threaded view
|

Re: vfs classpath could not replicate

Michael Wall
 java.lang.ClassNotFoundException: IO Error loading class org.apache.accumulo.tserver.compaction.DefaultCompactionStrategy

Do you have your accumulo jars in VFS?  I would recommend against that, put them in ACCUMULO_HOME/lib.  Those should not be reloaded between restarts and makes a lot of extra calls against HDFS than are necessary.

On Thu, Sep 27, 2018 at 2:36 PM Rob Verkuylen <[hidden email]> wrote:

Hmm 'jinfo <tabletserver_pid> | grep tmp'  shows: 

java.io.tmpdir = /tmp


"ls -hal /" shows 777 on /tmp


Grepping for 'tmp' in tserver startup:

[zookeeper.ZooKeeper] INFO : Client environment:java.io.tmpdir=/tmp

[server.Accumulo] INFO : general.vfs.cache.dir = /tmp/accumulo-vfs-cache-accumulo

[server.Accumulo] INFO : tserver.dir.memdump = /tmp


I'm guessing the relevant code to get the dir would be inAccumuloVFSClassLoader.java

String cacheDirPath = AccumuloClassLoader.getAccumuloString(VFS_CACHE_DIR, System.getProperty("java.io.tmpdir"));  

with 

public static final String VFS_CACHE_DIR = "general.vfs.cache.dir";


which then would be ' /tmp/accumulo-vfs-cache-accumulo', but it seems this dir does not not need to exist and will be created. Could there be any other reason this would pop up? Can these settings be overridden somewhere? 




On Wed, Sep 26, 2018 at 10:17 PM Christopher <[hidden email]> wrote:
It looks like the error is when vfs tries to create a temporary file in your tmpdir. I would check that your java.io.tmpdir points to a directory that exists and has the appropriate permissions for the user running the Accumulo process.

On Wed, Sep 26, 2018, 16:08 Rob Verkuylen <[hidden email]> wrote:

Hi,


I'm trying to get vfs loading of my jar working in 1.7.2. On a test cluster of the same version this works fine and I see the jar being replicated in the cache in '/tmp/accumulo-vfs*/fstore-filters-1.3.2.jar', but when I do the exact same thing on prod, I get the stacktrace below. Any ideas? 


Setup I used:

config -s general.vfs.context.classpath.fstore=hdfs://nameservice1/libs/accumulo/fstore/.*jar
config -t test.fstore_index -s table.classpath.context=fstore

Stacktrace on tablet server:
Failed to load class
            java.lang.ClassNotFoundException: IO Error loading class org.apache.accumulo.tserver.compaction.DefaultCompactionStrategy
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:188)
                        at org.apache.accumulo.core.conf.Property.createInstance(Property.java:873)
                        at org.apache.accumulo.core.conf.Property.createTableInstanceFromPropertyName(Property.java:910)
                        at org.apache.accumulo.tserver.TabletServerResourceManager$TabletResourceManager.needsMajorCompaction(TabletServerResourceManager.java:647)
                        at org.apache.accumulo.tserver.tablet.Tablet.needsMajorCompaction(Tablet.java:1594)
                        at org.apache.accumulo.tserver.tablet.Tablet.initiateMajorCompaction(Tablet.java:1574)
                        at org.apache.accumulo.tserver.TabletServer$MajorCompactor.run(TabletServer.java:1875)
                        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
                        at java.lang.Thread.run(Thread.java:745)
            Caused by: org.apache.commons.vfs2.FileSystemException: Could not replicate "hdfs://nameservice1/libs/accumulo/fstore/fstore-filters-1.3.2.jar".
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:426)
                        at org.apache.commons.vfs2.provider.zip.ZipFileSystem.<init>(ZipFileSystem.java:66)
                        at org.apache.commons.vfs2.provider.jar.JarFileSystem.<init>(JarFileSystem.java:48)
                        at org.apache.commons.vfs2.provider.jar.JarFileProvider.doCreateFileSystem(JarFileProvider.java:80)
                        at org.apache.commons.vfs2.provider.AbstractLayeredFileProvider.createFileSystem(AbstractLayeredFileProvider.java:87)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1022)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1042)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.addFileObjects(VFSClassLoader.java:156)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.<init>(VFSClassLoader.java:119)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:147)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:162)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager$Context.getClassLoader(ContextManager.java:46)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.getClassLoader(ContextManager.java:174)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:186)
                        ... 8 more
            Caused by: org.apache.commons.vfs2.FileSystemException: Unknown message with code "No such file or directory".
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:68)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.doReplicateFile(AbstractFileSystem.java:473)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:422)
                        ... 21 more
            Caused by: java.io.IOException: No such file or directory
                        at java.io.UnixFileSystem.createFileExclusively(Native Method)
                        at java.io.File.createTempFile(File.java:2024)
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:60)
                       
... 23 more


 

Reply | Threaded
Open this post in threaded view
|

Re: vfs classpath could not replicate

Rob Verkuylen
Nothing in VFS except my jar file.

On Thu, Sep 27, 2018 at 8:50 PM Michael Wall <[hidden email]> wrote:
 java.lang.ClassNotFoundException: IO Error loading class org.apache.accumulo.tserver.compaction.DefaultCompactionStrategy

Do you have your accumulo jars in VFS?  I would recommend against that, put them in ACCUMULO_HOME/lib.  Those should not be reloaded between restarts and makes a lot of extra calls against HDFS than are necessary.

On Thu, Sep 27, 2018 at 2:36 PM Rob Verkuylen <[hidden email]> wrote:

Hmm 'jinfo <tabletserver_pid> | grep tmp'  shows: 

java.io.tmpdir = /tmp


"ls -hal /" shows 777 on /tmp


Grepping for 'tmp' in tserver startup:

[zookeeper.ZooKeeper] INFO : Client environment:java.io.tmpdir=/tmp

[server.Accumulo] INFO : general.vfs.cache.dir = /tmp/accumulo-vfs-cache-accumulo

[server.Accumulo] INFO : tserver.dir.memdump = /tmp


I'm guessing the relevant code to get the dir would be inAccumuloVFSClassLoader.java

String cacheDirPath = AccumuloClassLoader.getAccumuloString(VFS_CACHE_DIR, System.getProperty("java.io.tmpdir"));  

with 

public static final String VFS_CACHE_DIR = "general.vfs.cache.dir";


which then would be ' /tmp/accumulo-vfs-cache-accumulo', but it seems this dir does not not need to exist and will be created. Could there be any other reason this would pop up? Can these settings be overridden somewhere? 




On Wed, Sep 26, 2018 at 10:17 PM Christopher <[hidden email]> wrote:
It looks like the error is when vfs tries to create a temporary file in your tmpdir. I would check that your java.io.tmpdir points to a directory that exists and has the appropriate permissions for the user running the Accumulo process.

On Wed, Sep 26, 2018, 16:08 Rob Verkuylen <[hidden email]> wrote:

Hi,


I'm trying to get vfs loading of my jar working in 1.7.2. On a test cluster of the same version this works fine and I see the jar being replicated in the cache in '/tmp/accumulo-vfs*/fstore-filters-1.3.2.jar', but when I do the exact same thing on prod, I get the stacktrace below. Any ideas? 


Setup I used:

config -s general.vfs.context.classpath.fstore=hdfs://nameservice1/libs/accumulo/fstore/.*jar
config -t test.fstore_index -s table.classpath.context=fstore

Stacktrace on tablet server:
Failed to load class
            java.lang.ClassNotFoundException: IO Error loading class org.apache.accumulo.tserver.compaction.DefaultCompactionStrategy
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:188)
                        at org.apache.accumulo.core.conf.Property.createInstance(Property.java:873)
                        at org.apache.accumulo.core.conf.Property.createTableInstanceFromPropertyName(Property.java:910)
                        at org.apache.accumulo.tserver.TabletServerResourceManager$TabletResourceManager.needsMajorCompaction(TabletServerResourceManager.java:647)
                        at org.apache.accumulo.tserver.tablet.Tablet.needsMajorCompaction(Tablet.java:1594)
                        at org.apache.accumulo.tserver.tablet.Tablet.initiateMajorCompaction(Tablet.java:1574)
                        at org.apache.accumulo.tserver.TabletServer$MajorCompactor.run(TabletServer.java:1875)
                        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
                        at java.lang.Thread.run(Thread.java:745)
            Caused by: org.apache.commons.vfs2.FileSystemException: Could not replicate "hdfs://nameservice1/libs/accumulo/fstore/fstore-filters-1.3.2.jar".
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:426)
                        at org.apache.commons.vfs2.provider.zip.ZipFileSystem.<init>(ZipFileSystem.java:66)
                        at org.apache.commons.vfs2.provider.jar.JarFileSystem.<init>(JarFileSystem.java:48)
                        at org.apache.commons.vfs2.provider.jar.JarFileProvider.doCreateFileSystem(JarFileProvider.java:80)
                        at org.apache.commons.vfs2.provider.AbstractLayeredFileProvider.createFileSystem(AbstractLayeredFileProvider.java:87)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1022)
                        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.createFileSystem(DefaultFileSystemManager.java:1042)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.addFileObjects(VFSClassLoader.java:156)
                        at org.apache.commons.vfs2.impl.VFSClassLoader.<init>(VFSClassLoader.java:119)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:147)
                        at org.apache.accumulo.start.classloader.vfs.AccumuloReloadingVFSClassLoader.<init>(AccumuloReloadingVFSClassLoader.java:162)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager$Context.getClassLoader(ContextManager.java:46)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.getClassLoader(ContextManager.java:174)
                        at org.apache.accumulo.start.classloader.vfs.ContextManager.loadClass(ContextManager.java:186)
                        ... 8 more
            Caused by: org.apache.commons.vfs2.FileSystemException: Unknown message with code "No such file or directory".
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:68)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.doReplicateFile(AbstractFileSystem.java:473)
                        at org.apache.commons.vfs2.provider.AbstractFileSystem.replicateFile(AbstractFileSystem.java:422)
                        ... 21 more
            Caused by: java.io.IOException: No such file or directory
                        at java.io.UnixFileSystem.createFileExclusively(Native Method)
                        at java.io.File.createTempFile(File.java:2024)
                        at org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator.replicateFile(UniqueFileReplicator.java:60)
                       
... 23 more